Re: [igt-dev] [PATCH i-g-t] tests: Add amdgpu test suite

["Martin Roukala (néé Peres)" <martin.roukala@mupuf.org>]

On 08/10/2021 22:02, Rodrigo Siqueira wrote:
> Add various test suites relevant for the amdgpu driver.
> 
> Cc: Harry Wentland <harry.wentland@amd.com>
> Cc: Nicholas Choi <Nicholas.Choi@amd.com>
> Cc: Rodrigo Siqueira <Rodrigo.Siqueira@amd.com>
> Cc: Sun peng (Leo) Li <sunpeng.li@amd.com>
> Cc: Alexander Deucher <alexander.deucher@amd.com>
> Cc: Martin Roukala <martin.roukala@mupuf.org>
> Cc: Hayden Goodfellow <hayden.goodfellow@amd.com>
> Cc: Simon Ser <contact@emersion.fr>
> Cc: Mark Yacoub <markyacoub@google.com>
> 
> Signed-off-by: Rodrigo Siqueira <Rodrigo.Siqueira@amd.com>
> ---
>   tests/amdgpu-ci/README                 |  31 +++++
>   tests/amdgpu-ci/fast-feedback.testlist | 135 +++++++++++++++++++
>   tests/amdgpu-ci/full-feedback.testlist | 173 +++++++++++++++++++++++++
>   3 files changed, 339 insertions(+)
>   create mode 100644 tests/amdgpu-ci/README
>   create mode 100644 tests/amdgpu-ci/fast-feedback.testlist
>   create mode 100644 tests/amdgpu-ci/full-feedback.testlist
> 
> diff --git a/tests/amdgpu-ci/README b/tests/amdgpu-ci/README
> new file mode 100644
> index 00000000..bd34245c
> --- /dev/null
> +++ b/tests/amdgpu-ci/README
> @@ -0,0 +1,31 @@
> +This directory contains test lists that are used by AMD's CI. The
> +files are passed to piglit with the --test-list parameter directly.
> +
> +The test lists are contained in the IGT repository for several
> +reasons:
> +
> +- The lists stay synchronized with the IGT codebase.
> +- Public availability. Kernel developers can see what tests are run,
> +  and can see what changes are done to the set, when, and why.

Thanks for doing this!

> +
> +Changing the test lists should only happen with approval from:
> +- Harry Wentland <harry.wentland@amd.com>
> +- Nicholas Choi <Nicholas.Choi@amd.com>
> +- Rodrigo Siqueira <Rodrigo.Siqueira@amd.com>
> +- Sun peng (Leo) Li <sunpeng.li@amd.com>
> +
> +======================
> +fast-feedback.testlist
> +======================
> +
> +Fast-feedback contains tests that roughly tests a wide coverage of features in
> +a short time. We use this list for presubmission validation. Results of a
> +fast-feedback test round should only be considered to mean that the kernel is
> +not obviously broken.

It would be good to have a target execution time for this list. ~10 
minutes and 12 minutes in the worst case has been working for Intel, but 
what would you like to use?

> +
> +======================
> +full-feedback.testlist
> +======================
> +
> +This is an extensive set of tests that takes a long time to complete in which
> +we usually use as post-submission.

 From experience, I would specify the full list as a opt-out, rather 
than opt-in. Otherwise, you will soon realize that it takes a lot of 
time and energy to maintain the list to always add new tests there.

By opting out of i915/gem/nouveau/udl/... tests, you will at least get 
every new KMS test by default.

Is that making sense?

Martin