Re: [PATCH v6 2/2] perf test: Json format checking

From: Arnaldo Carvalho de Melo <acme@kernel.org>
To: Ian Rogers <irogers@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@redhat.com>,
	Mark Rutland <mark.rutland@arm.com>,
	Alexander Shishkin <alexander.shishkin@linux.intel.com>,
	Jiri Olsa <jolsa@kernel.org>, Namhyung Kim <namhyung@kernel.org>,
	Kan Liang <kan.liang@linux.intel.com>,
	Zhengjun Xing <zhengjun.xing@linux.intel.com>,
	Sandipan Das <sandipan.das@amd.com>,
	Claire Jensen <cjense@google.com>, Alyssa Ross <hi@alyssa.is>,
	Like Xu <likexu@tencent.com>, James Clark <james.clark@arm.com>,
	Florian Fischer <florian.fischer@muhq.space>,
	linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org,
	Claire Jensen <clairej735@gmail.com>,
	Stephane Eranian <eranian@google.com>
Subject: Re: [PATCH v6 2/2] perf test: Json format checking
Date: Mon, 18 Jul 2022 12:49:25 -0300	[thread overview]
Message-ID: <YtWBBSHK2hAtykro@kernel.org> (raw)
In-Reply-To: <YtWAQqXFp98+N9ej@kernel.org>

Em Mon, Jul 18, 2022 at 12:46:10PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Mon, Jul 18, 2022 at 12:44:59PM -0300, Arnaldo Carvalho de Melo escreveu:
> > Em Thu, Jul 07, 2022 at 01:12:13PM -0700, Ian Rogers escreveu:
> > > From: Claire Jensen <cjense@google.com>
> > > 
> > > Add field checking tests for perf stat JSON output.
> > > Sanity checks the expected number of fields are present, that the
> > > expected keys are present and they have the correct values.
> > 
> > it isn't installing the lib:
> > 
> > [root@five ~]# perf test -v json
> >  91: perf stat JSON output linter                                    :
> > --- start ---
> > test child forked, pid 4086678
> > Checking json output: no args python3: can't open file '/var/home/acme/libexec/perf-core/tests/shell/lib/perf_json_output_lint.py': [Errno 2] No such file or directory
> > test child finished with -2
> > ---- end ----
> > perf stat JSON output linter: Skip
> > [root@five ~]#
> > 
> > I'm trying to fix, but please test it after installing...
> 
> 
> It should:
> 
> install-tests: all install-gtk
>         $(call QUIET_INSTALL, tests) \
>                 $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'; \
>                 $(INSTALL) tests/attr.py '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'; \
>                 $(INSTALL) tests/pe-file.exe* '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests'; \
>                 $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/attr'; \
>                 $(INSTALL) tests/attr/* '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/attr'; \
>                 $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell'; \
>                 $(INSTALL) tests/shell/*.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell'; \
>                 $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell/lib'; \
>                 $(INSTALL) tests/shell/lib/*.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell/lib'
> 
> /me checking...

The patch below is needed, but then should we mix .py and .sh in that
directory?

⬢[acme@toolbox perf]$ git diff

diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 8f0b1fb39984fb7b..65e5ba767fd6210e 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -1006,6 +1006,7 @@ install-tests: all install-gtk
                $(INSTALL) tests/shell/*.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell'; \
                $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell/lib'; \
                $(INSTALL) tests/shell/lib/*.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell/lib'
+               $(INSTALL) tests/shell/lib/*.py '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell/lib'

 install-bin: install-tools install-tests install-traceevent-plugins

⬢[acme@toolbox perf]$

And then, after that I'm getting:

[root@five ~]# perf test json
 91: perf stat JSON output linter                                    : FAILED!
[root@five ~]# perf test -v json |& tail -20
{"core" : "S0-D0-C15", "aggregate-number" : 2, "counter-value" : "2675.000000", "unit" : "", "event" : "stalled-cycles-backend", "event-runtime" : 2864158, "pcnt-running" : 100.00, "metric-value" : 0.341252, "metric-unit" : "backend cycles idle"}

{"core" : "S0-D0-C15", "aggregate-number" : 2, "counter-value" : "233533.000000", "unit" : "", "event" : "instructions", "event-runtime" : 2865528, "pcnt-running" : 100.00, "metric-value" : 0.297920, "metric-unit" : "insn per cycle"}

{"core" : "S0-D0-C15", "aggregate-number" : 2, "metric-value" : 0.192975, "metric-unit" : "stalled cycles per insn"}

{"core" : "S0-D0-C15", "aggregate-number" : 2, "counter-value" : "50214.000000", "unit" : "", "event" : "branches", "event-runtime" : 2865638, "pcnt-running" : 100.00, "metric-value" : 17.295742, "metric-unit" : "M/sec"}

{"core" : "S0-D0-C15", "aggregate-number" : 2, "counter-value" : "1513.000000", "unit" : "", "event" : "branch-misses", "event-runtime" : 1564989, "pcnt-running" : 54.00, "metric-value" : 3.013104, "metric-unit" : "of all branches"}

Traceback (most recent call last):
  File "/var/home/acme/libexec/perf-core/tests/shell/lib/perf_json_output_lint.py", line 92, in <module>
    check_json_output(expected_items)
  File "/var/home/acme/libexec/perf-core/tests/shell/lib/perf_json_output_lint.py", line 53, in check_json_output
    raise RuntimeError(f'wrong number of fields. counted {count} expected {expected_items}'
RuntimeError: wrong number of fields. counted 3 expected 8 in '{"core" : "S0-D0-C0", "aggregate-number" : 2, "metric-value" : 0.094345, "metric-unit" : "stalled cycles per insn"}
'
test child finished with -1
---- end ----
perf stat JSON output linter: FAILED!
[root@five ~]#

Can you please check and send a v7?

- Arnaldo
