Re: [PATCH 7/9] tests/performance: Add nightly tests

From: "Alex Bennée" <alex.bennee@linaro.org>
To: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
Cc: "ldoktor@redhat.com" <ldoktor@redhat.com>,
	"ehabkost@redhat.com" <ehabkost@redhat.com>,
	"philmd@redhat.com" <philmd@redhat.com>,
	"qemu-devel@nongnu.org" <qemu-devel@nongnu.org>,
	Ahmed Karaman <ahmedkhaledkaraman@gmail.com>,
	"jsnow@redhat.com" <jsnow@redhat.com>,
	"crosa@redhat.com" <crosa@redhat.com>,
	"rth@twiddle.net" <rth@twiddle.net>
Subject: Re: [PATCH 7/9] tests/performance: Add nightly tests
Date: Wed, 16 Sep 2020 09:31:31 +0100	[thread overview]
Message-ID: <87zh5qyvfg.fsf@linaro.org> (raw)
In-Reply-To: <CAHiYmc4_NkSxHJQSYDATPM5MnatO_PD3-MSGsLrQ2+HywsgAfw@mail.gmail.com>

Aleksandar Markovic <aleksandar.qemu.devel@gmail.com> writes:

> On Wednesday, September 2, 2020, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>>
>> Ahmed Karaman <ahmedkhaledkaraman@gmail.com> writes:
>>
>> > A nightly performance testing system to monitor any change in QEMU
>> > performance across seventeen different targets.
>> >
>> > The system includes eight different benchmarks to provide a variety
>> > of testing workloads.
>> >
>> > dijkstra_double:
>> > Find the shortest path between the source node and all other nodes
>> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn
>> > distances are double values. The value of n can be specified using
>> > the -n flag. The default value is 2000.
>> >
>> > dijkstra_int32:
>> > Find the shortest path between the source node and all other nodes
>> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn
>> > distances are int32 values. The value of n can be specified using
>> > the -n flag. The default value is 2000.
>> >
>> > matmult_double:
>> > Standard matrix multiplication of an n*n matrix of randomly generated
>> > double numbers from 0 to 100. The value of n is passed as an argument
>> > with the -n flag. The default value is 200.
>> >
>> > matmult_int32:
>> > Standard matrix multiplication of an n*n matrix of randomly generated
>> > integer numbers from 0 to 100. The value of n is passed as an
>> > argument with the -n flag. The default value is 200.
>> >
>> > qsort_double:
>> > Quick sort of an array of n randomly generated double numbers from 0
>> > to 1000. The value of n is passed as an argument with the -n flag.
>> > The default value is 300000.
>> >
>> > qsort_int32:
>> > Quick sort of an array of n randomly generated integer numbers from 0
>> > to 50000000. The value of n is passed as an argument with the -n
>> > flag.The default value is 300000.
>> >
>> > qsort_string:
>> > Quick sort of an array of 10000 randomly generated strings of size 8
>> > (including null terminating character). The sort process is repeated
>> > n number of times. The value of n is passed as an argument with the
>> > -n flag. The default value is 20.
>> >
>> > search_string:
>> > Search for the occurrence of a small string in a much larger random
>> > string (“needle in a hay”). The search process is repeated n number
>> > of times and each time, a different large random string (“hay”) is
>> > generated. The value of n can be specified using the -n flag. The
>> > default value is 20.
>> >
>> > Syntax:
>> >     nightly_tests_core.py [-h] [-r REF]
>> >     Optional arguments:
>> >         -h, --help            Show this help message and exit
>> >         -r REF, --reference REF
>> >                         Reference QEMU version - Default is v5.1.0
>> >     Example of usage:
>> >         nightly_tests_core.py -r v5.1.0 2>log.txt
>> >
>> > The following report includes detailed setup and execution details
>> > of the system:
>> > https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/
>> QEMU-Nightly-Performance-Tests/
>> >
>> > Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
>> > ---
>> >  tests/performance/nightly-tests/README.md     | 243 +++++
>> >  .../source/dijkstra_double/dijkstra_double.c  | 194 ++++
>> >  .../source/dijkstra_int32/dijkstra_int32.c    | 192 ++++
>> >  .../source/matmult_double/matmult_double.c    | 123 +++
>> >  .../source/matmult_int32/matmult_int32.c      | 121 +++
>> >  .../source/qsort_double/qsort_double.c        | 104 ++
>> >  .../source/qsort_int32/qsort_int32.c          | 103 ++
>> >  .../source/qsort_string/qsort_string.c        | 122 +++
>> >  .../source/search_string/search_string.c      | 110 +++
>> >  .../scripts/nightly_tests_core.py             | 920 ++++++++++++++++++
>> >  .../scripts/run_nightly_tests.py              | 135 +++
>> >  .../nightly-tests/scripts/send_email.py       |  56 ++
>> >  12 files changed, 2423 insertions(+)
>> >  create mode 100644 tests/performance/nightly-tests/README.md
>> >  create mode 100644 tests/performance/nightly-tests/benchmarks/source/
>> dijkstra_double/dijkstra_double.c
>> >  create mode 100644 tests/performance/nightly-tests/benchmarks/source/
>> dijkstra_int32/dijkstra_int32.c
>> >  create mode 100644 tests/performance/nightly-tests/benchmarks/source/
>> matmult_double/matmult_double.c
>> >  create mode 100644 tests/performance/nightly-tests/benchmarks/source/
>> matmult_int32/matmult_int32.c
>> >  create mode 100644 tests/performance/nightly-
>> tests/benchmarks/source/qsort_double/qsort_double.c
>> >  create mode 100644 tests/performance/nightly-
>> tests/benchmarks/source/qsort_int32/qsort_int32.c
>> >  create mode 100644 tests/performance/nightly-
>> tests/benchmarks/source/qsort_string/qsort_string.c
>> >  create mode 100644
>> > tests/performance/nightly-tests/benchmarks/source/
>> search_string/search_string.c
>>
>> Perhaps we could compress these paths down to:
>>
>>   tests/tcg/benchmarks/foo.c
>>   tests/tcg/benchmarks/bar.c
>>
>> and then we can also ensure they are built using the existing TCG tests
>> cross compile framework.
>>
>>
> Hi,
>
> May I just bring some alternative views on this topic of
> benchmarks/multiple cross-compiling/nightly tests?
>
> Having a known source code of any benchmark (and also license-compatible
> with QEMU) is, for sure, a good thing. However, for actual test execution,
> source code is not relevant, but the executables built for a diverse set of
> targets. Expectation that a test bed should be able to build let's say 17
> version for each benchmark is a tall order - and, in fact, unnecessary
> self-imposed limitation.
>
> My suggestion is that 17x8 executables needed for nightly tests in its
> current form should be prebuilt - and kept permanently in a separate
> repository (just not to burden QEMU repository). They are anyway built only
> once, there is no need to update them at all.
>
> This will allow nightly test execution on any system that can execute qemu,
> without any prerequisite for cross-compiling.
>
> Their source code could and should be integrated into QEMU tree, at the
> place of choice, but the test beds should not be subject to any burden of
> cross-compiling, simply because there is no need for that.

I have no problem with the test binaries being stored somewhere and
accessed when needed - as long as we have a source build-able version in
the tree. It would seem silly not to at least integrate the build for
the 19 different cross compilers we already support for the check-tcg
tests.

I'm not proposing we run them every test either, but at least have the
option to do:

  make build-tcg-benchmark
  make run-tcg-benchmark

so it's easy for a developer to get to a state when they can tinker and
debug.

>
> Yours,
> Aleksandar
>
>
>
>
>> >  create mode 100755 tests/performance/nightly-
>> tests/scripts/nightly_tests_core.py
>> >  create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_
>> tests.py
>> >  create mode 100644 tests/performance/nightly-
>> tests/scripts/send_email.py
>> >
>> > diff --git a/tests/performance/nightly-tests/README.md
>> b/tests/performance/nightly-tests/README.md
>> > new file mode 100644
>> > index 0000000000..6db3b351b3
>> > --- /dev/null
>> > +++ b/tests/performance/nightly-tests/README.md
>> > @@ -0,0 +1,243 @@
>> > +### QEMU Nightly Tests
>> > +
>> > +**Required settings:**
>> > +
>> > +Update the `GMAIL_USER` object in `send_email.py` with your credentials.
>> > +
>> > +For more details on how the system works, please check the [eighth
>> > report](https://ahmedkrmn.github.io/TCG-Continuous-
>> Benchmarking/QEMU-Nightly-Performance-Tests/)
>> > of the "TCG Continuos Benchmarking" series.
>>
>> As external URLs are potentially unstable I think we want to distil the
>> details into a rst do in docs/devel/
>>
>> <snip>
>>
>> --
>> Alex Bennée
>>

-- 
Alex Bennée