[PATCH igt] intel-ci: CI is missing any wait tests in BAT

[PATCH igt] intel-ci: CI is missing any wait tests in BAT

[Chris Wilson]

Add the two basic gem_wait tests to the fast list, together they take a
total of 1s (when correctly functioning).

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
---
 tests/intel-ci/fast-feedback.testlist | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/intel-ci/fast-feedback.testlist b/tests/intel-ci/fast-feedback.testlist
index f59ec98..853b911 100644
--- a/tests/intel-ci/fast-feedback.testlist
+++ b/tests/intel-ci/fast-feedback.testlist
@@ -126,6 +126,8 @@ igt@gem_sync@basic-store-each
 igt@gem_tiled_blits@basic
 igt@gem_tiled_fence_blits@basic
 igt@gem_tiled_pread_basic
+igt@gem_wait@basic-busy-all
+igt@gem_wait@basic-wait-all
 igt@gem_workarounds@basic-read
 igt@gvt_basic@invalid-placeholder-test
 igt@kms_addfb_basic@addfb25-bad-modifier
-- 
2.9.3

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH igt] intel-ci: CI is missing any wait tests in BAT

[Petri Latvala]

On Thu, Oct 13, 2016 at 03:00:43PM +0100, Chris Wilson wrote:
> Add the two basic gem_wait tests to the fast list, together they take a
> total of 1s (when correctly functioning).
> 
> Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> ---
>  tests/intel-ci/fast-feedback.testlist | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/tests/intel-ci/fast-feedback.testlist b/tests/intel-ci/fast-feedback.testlist
> index f59ec98..853b911 100644
> --- a/tests/intel-ci/fast-feedback.testlist
> +++ b/tests/intel-ci/fast-feedback.testlist
> @@ -126,6 +126,8 @@ igt@gem_sync@basic-store-each
>  igt@gem_tiled_blits@basic
>  igt@gem_tiled_fence_blits@basic
>  igt@gem_tiled_pread_basic
> +igt@gem_wait@basic-busy-all
> +igt@gem_wait@basic-wait-all
>  igt@gem_workarounds@basic-read
>  igt@gvt_basic@invalid-placeholder-test
>  igt@kms_addfb_basic@addfb25-bad-modifier
> -- 
> 2.9.3


In the future we will test new tests in the CI system. Unfortunately
the technology isn't there yet (tm) so this had to be tested
manually. Sorry for the delay.

For the record, the testing requirements for incoming new tests is
converging to:

- Tests must be stable. 1000 repeated runs with the new tests alone
  must give the same results. 10 repeated runs of modified BAT set
  must give the same results (excluding known issues in other tests).
- Tests must be fast. Full modified BAT run must not exceed the time
  budget.
- Tests must pass on current kernels, on some platform.

This will be documented when we can get CI do the testing for us on
multiple platforms and properly know what kind of requirements can
actually be tested.

For these tests, manual testing was done on IVB, HSW, and BYT. Stable,
fast and passing on all.


Acked-by: Petri Latvala <petri.latvala@intel.com>

Please push.



--
Petri Latvala
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH igt] intel-ci: CI is missing any wait tests in BAT

[Chris Wilson]

On Fri, Oct 28, 2016 at 01:46:51PM +0300, Petri Latvala wrote:
> On Thu, Oct 13, 2016 at 03:00:43PM +0100, Chris Wilson wrote:
> > Add the two basic gem_wait tests to the fast list, together they take a
> > total of 1s (when correctly functioning).
> > 
> > Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
> > ---
> >  tests/intel-ci/fast-feedback.testlist | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/tests/intel-ci/fast-feedback.testlist b/tests/intel-ci/fast-feedback.testlist
> > index f59ec98..853b911 100644
> > --- a/tests/intel-ci/fast-feedback.testlist
> > +++ b/tests/intel-ci/fast-feedback.testlist
> > @@ -126,6 +126,8 @@ igt@gem_sync@basic-store-each
> >  igt@gem_tiled_blits@basic
> >  igt@gem_tiled_fence_blits@basic
> >  igt@gem_tiled_pread_basic
> > +igt@gem_wait@basic-busy-all
> > +igt@gem_wait@basic-wait-all
> >  igt@gem_workarounds@basic-read
> >  igt@gvt_basic@invalid-placeholder-test
> >  igt@kms_addfb_basic@addfb25-bad-modifier
> > -- 
> > 2.9.3
> 
> 
> In the future we will test new tests in the CI system. Unfortunately
> the technology isn't there yet (tm) so this had to be tested
> manually. Sorry for the delay.
> 
> For the record, the testing requirements for incoming new tests is
> converging to:
> 
> - Tests must be stable. 1000 repeated runs with the new tests alone
>   must give the same results. 10 repeated runs of modified BAT set
>   must give the same results (excluding known issues in other tests).
> - Tests must be fast. Full modified BAT run must not exceed the time
>   budget.
> - Tests must pass on current kernels, on some platform.

An idea I had in passing... Remove any test from BAT that has not failed
in the last 180days of trybot and replace it with a current failing
test. BAT is useless if it is not detecting the right class of bugs,
those that we typically make mistakes in (i.e. we should optimise BAT
for coverage of coding errors). Plus we also look at BAT results pretty
frequency and failing tests get fixed faster due to the greater nagging
and faster response time of the automated system. (If by the end of the
week we haven't fixed the new failing cases, we are slacking!) Some
caveats apply, we definitely want to ensure coverage of ABI and the
tests must still be selected to fit within a time budget (I hopeful that
an expert system backed up by coverage analysis would be extremely
invaluable in refining our tests sets). Filtering of trybot results
would have to be done, probably with a voting system: select the most
frequently failed test cases, and then discount failures in other tests
in the same runs (based on the assumption that the same bug was seem by
the different tests, given a large enough sample size should ensure good
coverage).

Just an idea based around tests that always pass are boring (and a waste
of time), but tests we occasionally fail are more sensitive to our errors.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx