Re: [PATCH 8/8] Tests: add custom test jobs

["Philippe Mathieu-Daudé" <philmd@redhat.com>]

Hi Cleber,

On 4/15/21 11:51 PM, Cleber Rosa wrote:
> Different users (or even companies) have different interests, and
> may want to run a reduced set of tests during development, or a
> larger set of tests during QE.
> 
> To cover these use cases, some example (but functional) jobs are
> introduced here:
> 
> 1) acceptance-all-targets.py: runs all arch agnostic tests on all
>    built targets, unless there are conditions that make them not work
>    out of the box ATM, then run all tests that are specific to
>    predefined targets.
> 
> 2) acceptance-kvm-only.py: runs only tests that require KVM and are
>    specific to the host architecture.
> 
> 3) qtest-unit.py: runs a combination of qtest and unit tests (in
>    parallel).
> 
> 4) qtest-unit-acceptance.py: runs a combineation of qtest, unit tests

Typo "combination".

>    and acceptance tests (all of them in parallel)
> 
> To run the first two manually, follow the example bellow:
> 
>  $ cd build
>  $ make check-venv
>  $ ./tests/venv/bin/python3 tests/jobs/acceptance-all-targets.py
>  $ ./tests/venv/bin/python3 tests/jobs/acceptance-kvm-only.py
> 
> The third and fouth example depends on information coming from Meson,
> so the easiest way to run it is:
> 
>  $ cd build
>  $ make check-qtest-unit
>  $ make check-qtest-unit-acceptance
> 
> These are based on Avocado's Job API, a way to customize an Avocado
> job execution beyond the possibilities of command line arguments.
> For more Job API resources, please refer to:
> 
> a) Job API Examples:
>  - https://github.com/avocado-framework/avocado/tree/master/examples/jobs
> 
> b) Documentation about configurable features at the Job Level:
>  - https://avocado-framework.readthedocs.io/en/87.0/config/index.html
> 
> c) Documentation about the TestSuite class
>  - https://avocado-framework.readthedocs.io/en/87.0/api/core/avocado.core.html#avocado.core.suite.TestSuite
> 
> d) Documentation about the Job class
>  - https://avocado-framework.readthedocs.io/en/87.0/api/core/avocado.core.html#avocado.core.job.Job
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  configure                            |  2 +-
>  tests/Makefile.include               |  8 ++++
>  tests/jobs/acceptance-all-targets.py | 67 ++++++++++++++++++++++++++++
>  tests/jobs/acceptance-kvm-only.py    | 35 +++++++++++++++
>  tests/jobs/qtest-unit-acceptance.py  | 31 +++++++++++++
>  tests/jobs/qtest-unit.py             | 24 ++++++++++
>  tests/jobs/utils.py                  | 22 +++++++++
>  7 files changed, 188 insertions(+), 1 deletion(-)
>  create mode 100644 tests/jobs/acceptance-all-targets.py
>  create mode 100644 tests/jobs/acceptance-kvm-only.py
>  create mode 100644 tests/jobs/qtest-unit-acceptance.py
>  create mode 100644 tests/jobs/qtest-unit.py
>  create mode 100644 tests/jobs/utils.py

> +if __name__ == '__main__':
> +    sys.exit(main())
> diff --git a/tests/jobs/acceptance-kvm-only.py b/tests/jobs/acceptance-kvm-only.py
> new file mode 100644
> index 0000000000..acdcbbe087
> --- /dev/null
> +++ b/tests/jobs/acceptance-kvm-only.py
> @@ -0,0 +1,35 @@
> +#!/usr/bin/env python3
> +
> +import os
> +import sys
> +
> +# This comes from tests/acceptance/avocado_qemu/__init__.py and should
> +# not be duplicated here.  The solution is to have the "avocado_qemu"
> +# code and "python/qemu" available during build
> +BUILD_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
> +if os.path.islink(os.path.dirname(os.path.dirname(__file__))):
> +    # The link to the acceptance tests dir in the source code directory
> +    lnk = os.path.dirname(os.path.dirname(__file__))
> +    #: The QEMU root source directory
> +    SOURCE_DIR = os.path.dirname(os.path.dirname(os.readlink(lnk)))
> +else:
> +    SOURCE_DIR = BUILD_DIR
> +sys.path.append(os.path.join(SOURCE_DIR, 'python'))
> +
> +from avocado.core.job import Job
> +
> +from qemu.accel import kvm_available
> +
> +
> +def main():
> +    if not kvm_available():
> +        sys.exit(0)
> +
> +    config = {'run.references': ['tests/acceptance/'],
> +              'filter.by_tags.tags': ['accel:kvm,arch:%s' % os.uname()[4]]}

If we want forks to use their own set of tags, it would be better to
provide an uniform way, not adding new test entry point for each set
of fork tags. Could we consume a YAML config file instead? And provide
templates so forks could adapt to their needs?

Thanks,

Phil.