[PATCH v3 1/2] perf test: Workload test of metric and metricgroups

[PATCH v3 1/2] perf test: Workload test of metric and metricgroups

[Ian Rogers]

Test every metric and metricgroup with 'true' as a workload. For
metrics, check that we see the metric printed or get unsupported. If the
'true' workload executes too quickly retry with 'perf bench internals
synthesize'.

v3. Fix test condition (thanks to Paul A. Clarke <pc@us.ibm.com>). Add a
    fallback case of a larger workload so that we don't ignore "<not
    counted>".
v2. Switched the workload to something faster.

Signed-off-by: Ian Rogers <irogers@google.com>
---
 .../perf/tests/shell/stat_all_metricgroups.sh | 12 ++++++++++
 tools/perf/tests/shell/stat_all_metrics.sh    | 22 +++++++++++++++++++
 2 files changed, 34 insertions(+)
 create mode 100755 tools/perf/tests/shell/stat_all_metricgroups.sh
 create mode 100755 tools/perf/tests/shell/stat_all_metrics.sh

diff --git a/tools/perf/tests/shell/stat_all_metricgroups.sh b/tools/perf/tests/shell/stat_all_metricgroups.sh
new file mode 100755
index 000000000000..de24d374ce24
--- /dev/null
+++ b/tools/perf/tests/shell/stat_all_metricgroups.sh
@@ -0,0 +1,12 @@
+#!/bin/sh
+# perf all metricgroups test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+for m in $(perf list --raw-dump metricgroups); do
+  echo "Testing $m"
+  perf stat -M "$m" true
+done
+
+exit 0
diff --git a/tools/perf/tests/shell/stat_all_metrics.sh b/tools/perf/tests/shell/stat_all_metrics.sh
new file mode 100755
index 000000000000..7f4ba3cad632
--- /dev/null
+++ b/tools/perf/tests/shell/stat_all_metrics.sh
@@ -0,0 +1,22 @@
+#!/bin/sh
+# perf all metrics test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+for m in $(perf list --raw-dump metrics); do
+  echo "Testing $m"
+  result=$(perf stat -M "$m" true 2>&1)
+  if [[ ! "$result" =~ "$m" ]] && [[ ! "$result" =~ "<not supported>" ]]; then
+    # We failed to see the metric and the events are support. Possibly the
+    # workload was too small so retry with something longer.
+    result=$(perf stat -M "$m" perf bench internals synthesize 2>&1)
+    if [[ ! "$result" =~ "$m" ]]; then
+      echo "Metric '$m' not printed in:"
+      echo "$result"
+      exit 1
+    fi
+  fi
+done
+
+exit 0
-- 
2.33.0.464.g1972c5931b-goog

[PATCH v3 2/2] perf test: Workload test of all PMUs

[Ian Rogers]

Iterate over the list of PMUs and run the 'true' workload on them. If
the event isn't printed then run the large 'perf bench internals
synthesize' workload and check the event is counted.

On a Skylake this test takes 1m15s mainly running the 'true' workload.

Suggested-by: John Garry <john.garry@huawei.com>
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/shell/stat_all_pmu.sh | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100755 tools/perf/tests/shell/stat_all_pmu.sh

diff --git a/tools/perf/tests/shell/stat_all_pmu.sh b/tools/perf/tests/shell/stat_all_pmu.sh
new file mode 100755
index 000000000000..2de7fd0394fd
--- /dev/null
+++ b/tools/perf/tests/shell/stat_all_pmu.sh
@@ -0,0 +1,22 @@
+#!/bin/sh
+# perf all PMU test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+for p in $(perf list --raw-dump pmu); do
+  echo "Testing $p"
+  result=$(perf stat -e "$p" true 2>&1)
+  if [[ ! "$result" =~ "$p" ]] && [[ ! "$result" =~ "<not supported>" ]]; then
+    # We failed to see the event and it is supported. Possibly the workload was
+    # too small so retry with something longer.
+    result=$(perf stat -e "$p" perf bench internals synthesize 2>&1)
+    if [[ ! "$result" =~ "$p" ]]; then
+      echo "Event '$p' not printed in:"
+      echo "$result"
+      exit 1
+    fi
+  fi
+done
+
+exit 0
-- 
2.33.0.464.g1972c5931b-goog

Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups

[Arnaldo Carvalho de Melo]

Em Fri, Sep 17, 2021 at 11:42:39AM -0700, Ian Rogers escreveu:
> Test every metric and metricgroup with 'true' as a workload. For
> metrics, check that we see the metric printed or get unsupported. If the
> 'true' workload executes too quickly retry with 'perf bench internals
> synthesize'.
> 
> v3. Fix test condition (thanks to Paul A. Clarke <pc@us.ibm.com>). Add a
>     fallback case of a larger workload so that we don't ignore "<not
>     counted>".
> v2. Switched the workload to something faster.

Hi John, does your Reviewed-by stands for v3 too?

- Arnaldo
 
> Signed-off-by: Ian Rogers <irogers@google.com>
> ---
>  .../perf/tests/shell/stat_all_metricgroups.sh | 12 ++++++++++
>  tools/perf/tests/shell/stat_all_metrics.sh    | 22 +++++++++++++++++++
>  2 files changed, 34 insertions(+)
>  create mode 100755 tools/perf/tests/shell/stat_all_metricgroups.sh
>  create mode 100755 tools/perf/tests/shell/stat_all_metrics.sh
> 
> diff --git a/tools/perf/tests/shell/stat_all_metricgroups.sh b/tools/perf/tests/shell/stat_all_metricgroups.sh
> new file mode 100755
> index 000000000000..de24d374ce24
> --- /dev/null
> +++ b/tools/perf/tests/shell/stat_all_metricgroups.sh
> @@ -0,0 +1,12 @@
> +#!/bin/sh
> +# perf all metricgroups test
> +# SPDX-License-Identifier: GPL-2.0
> +
> +set -e
> +
> +for m in $(perf list --raw-dump metricgroups); do
> +  echo "Testing $m"
> +  perf stat -M "$m" true
> +done
> +
> +exit 0
> diff --git a/tools/perf/tests/shell/stat_all_metrics.sh b/tools/perf/tests/shell/stat_all_metrics.sh
> new file mode 100755
> index 000000000000..7f4ba3cad632
> --- /dev/null
> +++ b/tools/perf/tests/shell/stat_all_metrics.sh
> @@ -0,0 +1,22 @@
> +#!/bin/sh
> +# perf all metrics test
> +# SPDX-License-Identifier: GPL-2.0
> +
> +set -e
> +
> +for m in $(perf list --raw-dump metrics); do
> +  echo "Testing $m"
> +  result=$(perf stat -M "$m" true 2>&1)
> +  if [[ ! "$result" =~ "$m" ]] && [[ ! "$result" =~ "<not supported>" ]]; then
> +    # We failed to see the metric and the events are support. Possibly the
> +    # workload was too small so retry with something longer.
> +    result=$(perf stat -M "$m" perf bench internals synthesize 2>&1)
> +    if [[ ! "$result" =~ "$m" ]]; then
> +      echo "Metric '$m' not printed in:"
> +      echo "$result"
> +      exit 1
> +    fi
> +  fi
> +done
> +
> +exit 0
> -- 
> 2.33.0.464.g1972c5931b-goog

-- 

- Arnaldo

Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups

[John Garry]

On 24/09/2021 20:09, Arnaldo Carvalho de Melo wrote:
> Em Fri, Sep 17, 2021 at 11:42:39AM -0700, Ian Rogers escreveu:
>> Test every metric and metricgroup with 'true' as a workload. For
>> metrics, check that we see the metric printed or get unsupported. If the
>> 'true' workload executes too quickly retry with 'perf bench internals
>> synthesize'.
>>
>> v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
>>      fallback case of a larger workload so that we don't ignore "<not
>>      counted>".
>> v2. Switched the workload to something faster.
> Hi John, does your Reviewed-by stands for v3 too?
> 
> - Arnaldo
>   

Yeah,

Reviewed-by: John Garry <john.garry@huawei.com>


Thanks

Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups

[Arnaldo Carvalho de Melo]

Em Fri, Sep 24, 2021 at 08:39:18PM +0100, John Garry escreveu:
> On 24/09/2021 20:09, Arnaldo Carvalho de Melo wrote:
> > Em Fri, Sep 17, 2021 at 11:42:39AM -0700, Ian Rogers escreveu:
> > > Test every metric and metricgroup with 'true' as a workload. For
> > > metrics, check that we see the metric printed or get unsupported. If the
> > > 'true' workload executes too quickly retry with 'perf bench internals
> > > synthesize'.

> > > v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
> > >      fallback case of a larger workload so that we don't ignore "<not
> > >      counted>".
> > > v2. Switched the workload to something faster.
> > Hi John, does your Reviewed-by stands for v3 too?
 
> Yeah,
 
> Reviewed-by: John Garry <john.garry@huawei.com>

Thanks, applied.

- Arnaldo

Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups

[John Garry]

On 24/09/2021 20:09, Arnaldo Carvalho de Melo wrote:
> Em Fri, Sep 17, 2021 at 11:42:39AM -0700, Ian Rogers escreveu:
>> Test every metric and metricgroup with 'true' as a workload. For
>> metrics, check that we see the metric printed or get unsupported. If the
>> 'true' workload executes too quickly retry with 'perf bench internals
>> synthesize'.
>>
>> v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
>>      fallback case of a larger workload so that we don't ignore "<not
>>      counted>".
>> v2. Switched the workload to something faster.

Hi Ian,

I just noticed that this test fails on my broadwell machine.

I am using acme perf/core @ 09dd3c22daaf

metricgroup Memory_Bw fails, and it seems because of the "true" argument 
to "perf stat" (or any argument, like sleep 1):

john@localhost:~/kernel-dev9/tools/perf> sudo ./perf stat -M Memory_BW
^C
  Performance counter stats for 'system wide':

              2,184      arb/event=0x84,umask=0x1/ #     0.26 
DRAM_BW_Use
          2,954,938      arb/event=0x81,umask=0x1/ 

        736,368,852 ns   duration_time 

         58,202,980      l1d_pend_miss.pending_cycles #     2.34 MLP 
                  (80.11%)
        136,293,194      l1d_pend_miss.pending 
               (19.89%)
        736,368,852 ns   duration_time 

          1,065,656      longest_lat_cache.miss    #     0.09 
L3_Cache_Fill_BW         (39.71%)
        736,368,852 ns   duration_time 

          5,365,477      l2_lines_in.all           #     0.47 
L2_Cache_Fill_BW         (59.80%)
        736,368,852 ns   duration_time 

          3,557,362      l1d.replacement           #     0.31 
L1D_Cache_Fill_BW        (79.90%)
        736,368,852 ns   duration_time 


        0.736368852 seconds time elapsed


john@localhost:~/kernel-dev9/tools/perf> sudo ./perf stat -M Memory_BW true
Error:
The sys_perf_event_open() syscall returned with 22 (Invalid argument) 
for event (arb/event=0x84,umask=0x1/).
/bin/dmesg | grep -i perf may provide additional information.

john@localhost:~/kernel-dev9/tools/perf>

Anyone any idea on this before I start digging?

Thanks,
John

Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups

[Arnaldo Carvalho de Melo]

On January 12, 2022 9:24:29 AM GMT-03:00, John Garry <john.garry@huawei.com> wrote:
>On 24/09/2021 20:09, Arnaldo Carvalho de Melo wrote:
>> Em Fri, Sep 17, 2021 at 11:42:39AM -0700, Ian Rogers escreveu:
>>> Test every metric and metricgroup with 'true' as a workload. For
>>> metrics, check that we see the metric printed or get unsupported. If the
>>> 'true' workload executes too quickly retry with 'perf bench internals
>>> synthesize'.
>>>
>>> v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
>>>      fallback case of a larger workload so that we don't ignore "<not
>>>      counted>".
>>> v2. Switched the workload to something faster.
>
>Hi Ian,
>
>I just noticed that this test fails on my broadwell machine.
>
>I am using acme perf/core @ 09dd3c22daaf

Hi,

Can you try with tmp.perf/perf_cpu instead?

There's a patch there that maybe fixes this.

- Arnaldo
>
>metricgroup Memory_Bw fails, and it seems because of the "true" argument 
>to "perf stat" (or any argument, like sleep 1):
>
>john@localhost:~/kernel-dev9/tools/perf> sudo ./perf stat -M Memory_BW
>^C
>  Performance counter stats for 'system wide':
>
>              2,184      arb/event=0x84,umask=0x1/ #     0.26 
>DRAM_BW_Use
>          2,954,938      arb/event=0x81,umask=0x1/ 
>
>        736,368,852 ns   duration_time 
>
>         58,202,980      l1d_pend_miss.pending_cycles #     2.34 MLP 
>                  (80.11%)
>        136,293,194      l1d_pend_miss.pending 
>               (19.89%)
>        736,368,852 ns   duration_time 
>
>          1,065,656      longest_lat_cache.miss    #     0.09 
>L3_Cache_Fill_BW         (39.71%)
>        736,368,852 ns   duration_time 
>
>          5,365,477      l2_lines_in.all           #     0.47 
>L2_Cache_Fill_BW         (59.80%)
>        736,368,852 ns   duration_time 
>
>          3,557,362      l1d.replacement           #     0.31 
>L1D_Cache_Fill_BW        (79.90%)
>        736,368,852 ns   duration_time 
>
>
>        0.736368852 seconds time elapsed
>
>
>john@localhost:~/kernel-dev9/tools/perf> sudo ./perf stat -M Memory_BW true
>Error:
>The sys_perf_event_open() syscall returned with 22 (Invalid argument) 
>for event (arb/event=0x84,umask=0x1/).
>/bin/dmesg | grep -i perf may provide additional information.
>
>john@localhost:~/kernel-dev9/tools/perf>
>
>Anyone any idea on this before I start digging?
>
>Thanks,
>John

Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups

[John Garry]

On 12/01/2022 13:34, Arnaldo Carvalho de Melo wrote:
>>>> v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
>>>>       fallback case of a larger workload so that we don't ignore "<not
>>>>       counted>".
>>>> v2. Switched the workload to something faster.
>> Hi Ian,
>>
>> I just noticed that this test fails on my broadwell machine.
>>
>> I am using acme perf/core @ 09dd3c22daaf
> Hi,
> 
> Can you try with tmp.perf/perf_cpu instead?
> 
> There's a patch there that maybe fixes this.

Yeah, that (metricgroups) test passes on that branch. I assume it's the 
test script -a change...

Cheers,
john

Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups

[Arnaldo Carvalho de Melo]

Em Wed, Jan 12, 2022 at 05:32:56PM +0000, John Garry escreveu:
> On 12/01/2022 13:34, Arnaldo Carvalho de Melo wrote:
> > > > > v3. Fix test condition (thanks to Paul A. Clarke<pc@us.ibm.com>). Add a
> > > > >       fallback case of a larger workload so that we don't ignore "<not
> > > > >       counted>".
> > > > > v2. Switched the workload to something faster.
> > > Hi Ian,
> > > 
> > > I just noticed that this test fails on my broadwell machine.
> > > 
> > > I am using acme perf/core @ 09dd3c22daaf
> > Hi,
> > 
> > Can you try with tmp.perf/perf_cpu instead?
> > 
> > There's a patch there that maybe fixes this.
> 
> Yeah, that (metricgroups) test passes on that branch. I assume it's the test
> script -a change...

Can I get  that as a Tested-by: you tag?

- Arnaldo

Re: [PATCH v3 1/2] perf test: Workload test of metric and metricgroups

[John Garry]

On 12/01/2022 17:59, Arnaldo Carvalho de Melo wrote:
>>> There's a patch there that maybe fixes this.
>> Yeah, that (metricgroups) test passes on that branch. I assume it's the test
>> script -a change...
> Can I get  that as a Tested-by: you tag?

Tested-by: John Garry <john.garry@huawei.com>