Meeting Minutes for 2018-08-20

Meeting Minutes for 2018-08-20

[Dan Rue]

- Attendees: Ana, Guillaume, Matt, Milosz, Rafael, Dan, Mark

- Guillaume
  - Dealing with test results:
    - We currently have test suites/sets/cases, but test sets aren’t
      used and their implementation is broken.  They are mostly a side
      effect of parsing LAVA callback data, they are not meaningful from
      a user point of view.
    - Suggestion: Remove test sets to simplify things.  If we need a
      better backend solution, we should consider SQUAD and rethink this
      from a global point of view.
    - If some test suites need to be broken down, then we can have
      several smaller test plans.  We could also consider running
      several test plans in a row with a single LAVA job. Example:
      https://lava.collabora.co.uk/results/1240317
      - [drue/Mark] - In LKFT we run LTP across several test runs, and
        they end up getting called “suite” LTP-subsetA, LTP-subsetB,
        etc, because squad doesn’t support sets.
        - [broonie] The goal here is to both get faster runs when
          there’s multiple boards and to improve robustness against test
          runs crashing and losing a lot of results on larger
          testsuites.
      - [ana] I would like to remove it because what we currently have
        doesn’t work and we don’t use it
      - [matt] this was all designed a while ago. A test set is supposed
        to be a part of a bigger test suite. If it’s not useful now, we
        can scrap it. There are performance issues too that you can see
        when viewing the test data. +1 remove
    - Decision: No objections to removing test sets
  - Reshaping test configurations:
    - YAML-based description now being tested
    - In the process of adding all the devices and test plans to achieve
      same test coverage as we currently have with the “device_map”
    - Removing dead LAVA v1 code
    - Aiming at eventually merging kernelci-build and lava-ci.
    - Looking for feedback on
      https://github.com/kernelci/lava-ci-staging/pull/96
  - Multiple bisections run for single regression: investigating...

- Ana
  - Reporting test suite results:
    - Sent v2 of mail reports last week
      https://github.com/kernelci/kernelci-backend/pull/70 Comments
      welcome. This is a first step to start getting the results of the
      test suites by mail.
    - Sent small PR fixing a problem with the template for sending
      duplicate mails
  - Start reporting on regressions: this is blocked by the point
    mentioned above about test suites/sets/cases. I would like to send a
    PR to the backend for tracking and reporting test suites regressions
    in the test suites but If we want to use test sets, some code needs
    to be added to fix its usage in the backend.
  - Documentations update
    - Added 2 new pages documenting the test suites (on-going) and the
      rootfs images used by the test suites.
    - http://wiki.kernelci.org now tell readers about the github wiki.
    - Help welcome savaging the useful parts from wiki.kci.o
    - Diagrams and other graphics can be added in the github wiki via
      git : git@github.com:kernelci/kernelci-doc.wiki.git
    - [drue] it looks like you can’t host a github wiki at its own url,
      can you? I looked.
      - [ana] no, but it’s stored in markdown in a git repo and we could
        use something else to render and host it in the future.
    - Github permissions - right now I can’t ask people to review
      kernelci related pull requests
      - [matt] https://github.com/orgs/kernelci/teams exists but needs
        to be made better

- Milosz
  - Kernelci testing data presentation in SQUAD:
    https://staging-qa-reports.linaro.org/kernelci/
  - [ana] can results be filtered by date?
    - [milosz] you can use the API to get that. In the UI it’s sorted by
      date, but you can’t filter by date
  - [matt] it looks better than kernelci’s views. Some of the sorting
    and mapping can be improved
    - https://staging-qa-reports.linaro.org/kernelci/ shows it by
      branch, but what if someone wants to see it by board or by
      architecture/defconfig
    - [milosz] i’m working on filtering by environment (architecture,
      board), so that users can view results by board, or by config
      - [milosz] I’m currently working on
        https://github.com/Linaro/squad/issues/245
    - [guill] custom views would be really nice so that an individual
      could have a search for what they’re interested in that could be
      saved in a link, and shared that way
      - [milosz] this is a nice idea
      - [matt] here’s an example of a multi-field search in kernelci
        https://kernelci.org/boot/?apq8016-sbc%20lab-mhart%20master%20next
        - lava/squad doesn’t do that because of django’s default filter
          implementation

Re: [kernelci] Meeting Minutes for 2018-08-20

[Kevin Hilman]

Dan, thanks again for the minutes!

"Dan Rue" <dan.rue@linaro.org> writes:

> - Attendees: Ana, Guillaume, Matt, Milosz, Rafael, Dan, Mark
>
> - Guillaume
>   - Dealing with test results:
>     - We currently have test suites/sets/cases, but test sets aren’t
>       used and their implementation is broken.

What is broken in the implementation?  LAVA? backend? frontend?

>        They are mostly a side effect of parsing LAVA callback data,

Not sure I'd call it a side effect.  LAVA supports the notion of suite,
set and case, and we need to deal with that if we're handling LAVA data
(even if there are no current kernelCI tests using it.)

>       they are not meaningful from a user point of view.

I disagree.  They are extrememly useful for large test suites (LTP,
kselftest, Android CTP, etc.)

If you mean our *current* ways of visualizing them are not meaningful, I
agree, but that's different from whether they are useful at all.

>     - Suggestion: Remove test sets to simplify things.  If we need a
>       better backend solution, we should consider SQUAD and rethink this
>       from a global point of view.

Not sure how squad helps here.  Seems it was designed with this same
limitation.  IMO we need the raw data to be coherent, and cleanly broken
down, and the frontend/UI used to view needs to be independent from
that.

>     - If some test suites need to be broken down, then we can have
>       several smaller test plans.  We could also consider running
>       several test plans in a row with a single LAVA job. Example:
>       https://lava.collabora.co.uk/results/1240317
>       - [drue/Mark] - In LKFT we run LTP across several test runs, and
>         they end up getting called “suite” LTP-subsetA, LTP-subsetB,
>         etc, because squad doesn’t support sets.

IMO, acknowleding that some suites need to be broken up is acknowleging
that he notion of test sets is useful.  Just concatening the suite name
and set name is a hack, not a solution.

>         - [broonie] The goal here is to both get faster runs when
>           there’s multiple boards and to improve robustness against test
>           runs crashing and losing a lot of results on larger
>           testsuites.

So in LKFT, the seprate LTP sub-sets are run on separate boots?  Seems
like an LKFT implementation decision, not somthing you would want to
force on test writers, nor assume in the data model.

>       - [ana] I would like to remove it because what we currently have
>         doesn’t work and we don’t use it

By "we" you mean current kernelCI users and kernelCI tests.  IMO, that's
not a very good sample size, as there's currently only a few of us, and
we don't have very many tests, and the tests that we have are pretty
small.

>       - [matt] this was all designed a while ago. A test set is supposed
>         to be a part of a bigger test suite. If it’s not useful now, we
>         can scrap it. There are performance issues too that you can see
>         when viewing the test data. +1 remove

Are you saying LAVA is dropping the notion of test sets?

>     - Decision: No objections to removing test sets

I object.  (Can we please be careful about decision based on "no
objections" when not everyone is able to participate in the call.)

What happens when trying to support LAVA test jobs that actually use
suite, set and case?  We're trying to expand kernelCI users and test
cases and there are lots of LAVA users out there with large test suites.
They may not be sharing results (yet), but it would be a shame to have
to have users restructure their test jobs because kCI doesn't support
test sets.

Also, what about kselftest?  It has several suites (e.g. each subdir of
tools/testing/selftest) and within each suite can have sets
(e.g. net/forwarding, ftrace/test.d/*)

I'd really like kernelCI to scale to multiple forms of frontend/UI etc.,
and for that the data model needs to be generic, flexible and especially
scalable.

To do that, I think it's very premature to eliminate one level of test
hierarchy, for the primary reason that it's convenient in the short
term.

>   - Reshaping test configurations:
>     - YAML-based description now being tested
>     - In the process of adding all the devices and test plans to achieve
>       same test coverage as we currently have with the “device_map”
>     - Removing dead LAVA v1 code
>     - Aiming at eventually merging kernelci-build and lava-ci.
>     - Looking for feedback on
>       https://github.com/kernelci/lava-ci-staging/pull/96
>   - Multiple bisections run for single regression: investigating...
>
> - Ana
>   - Reporting test suite results:
>     - Sent v2 of mail reports last week
>       https://github.com/kernelci/kernelci-backend/pull/70 Comments
>       welcome. This is a first step to start getting the results of the
>       test suites by mail.
>     - Sent small PR fixing a problem with the template for sending
>       duplicate mails
>   - Start reporting on regressions: this is blocked by the point
>     mentioned above about test suites/sets/cases. I would like to send a
>     PR to the backend for tracking and reporting test suites regressions
>     in the test suites but If we want to use test sets, some code needs
>     to be added to fix its usage in the backend.

I would much prefer fixing the backend than removing test sets all
together.

>   - Documentations update
>     - Added 2 new pages documenting the test suites (on-going) and the
>       rootfs images used by the test suites.
>     - http://wiki.kernelci.org now tell readers about the github wiki.
>     - Help welcome savaging the useful parts from wiki.kci.o

IMO, there is a lot useful there.

>     - Diagrams and other graphics can be added in the github wiki via
>       git : git@github.com:kernelci/kernelci-doc.wiki.git
>     - [drue] it looks like you can’t host a github wiki at its own url,
>       can you? I looked.
>       - [ana] no, but it’s stored in markdown in a git repo and we could
>         use something else to render and host it in the future.
>     - Github permissions - right now I can’t ask people to review
>       kernelci related pull requests
>       - [matt] https://github.com/orgs/kernelci/teams exists but needs
>         to be made better

Kevin

> - Milosz
>   - Kernelci testing data presentation in SQUAD:
>     https://staging-qa-reports.linaro.org/kernelci/
>   - [ana] can results be filtered by date?
>     - [milosz] you can use theAPI to get that. In the UI it’s sorted by
>       date, but you can’t filter by date
>   - [matt] it looks better than kernelci’s views. Some of the sorting
>     and mapping can be improved
>     - https://staging-qa-reports.linaro.org/kernelci/ shows it by
>       branch, but what if someone wants to see it by board or by
>       architecture/defconfig
>     - [milosz] i’m working on filtering by environment (architecture,
>       board), so that users can view results by board, or by config
>       - [milosz] I’m currently working on
>         https://github.com/Linaro/squad/issues/245
>     - [guill] custom views would be really nice so that an individual
>       could have a search for what they’re interested in that could be
>       saved in a link, and shared that way
>       - [milosz] this is a nice idea
>       - [matt] here’s an example of a multi-field search in kernelci
>         https://kernelci.org/boot/?apq8016-sbc%20lab-mhart%20master%20next
>         - lava/squad doesn’t do that because of django’s default filter
>           implementation

Configurable, custom views is really the killer feature, but also the
hardest part.  That's why, IMO, we need to get the data model right, and
start using more generic analytic stacks (e.g. Elastic stack[1], or
Apache Spark, etc.)

Kevin

[1] https://www.elastic.co/webinars/introduction-elk-stack

Re: [kernelci] Meeting Minutes for 2018-08-20

[Guillaume Tucker]

On Mon, Aug 20, 2018 at 7:45 PM Kevin Hilman <khilman@baylibre.com> wrote:
>
> Dan, thanks again for the minutes!
>
> "Dan Rue" <dan.rue@linaro.org> writes:
>
> > - Attendees: Ana, Guillaume, Matt, Milosz, Rafael, Dan, Mark
> >
> > - Guillaume
> >   - Dealing with test results:
> >     - We currently have test suites/sets/cases, but test sets aren’t
> >       used and their implementation is broken.
>
> What is broken in the implementation?  LAVA? backend? frontend?

The way we currently deal with test sets in the backend does not
add any value with the short test plans we currently have.  As I
explained in my previous email thread, both the test suite and
test set entries have a list of test cases, making test sets
effectively redundant.

> >        They are mostly a side effect of parsing LAVA callback data,
>
> Not sure I'd call it a side effect.  LAVA supports the notion of suite,
> set and case, and we need to deal with that if we're handling LAVA data
> (even if there are no current kernelCI tests using it.)

Yes we need to be able to deal with the data sent by LAVA, but it
doesn't mean we need to follow the way the data is formatted in
the LAVA callbacks.

> >       they are not meaningful from a user point of view.
>
> I disagree.  They are extrememly useful for large test suites (LTP,
> kselftest, Android CTP, etc.)
>
> If you mean our *current* ways of visualizing them are not meaningful, I
> agree, but that's different from whether they are useful at all.

Right now, it's not clear what test sets are all about from our
backend point of view.  There also seems to be some confusion
around what they're supposed to be from a LAVA point of view, so
here's some clarification from the docs:

A test case is an individual test in a LAVA job, I think we all
know that that is (run a command, get a single result).

A test suite is a group of test cases run in a job:

  https://validation.linaro.org/static/docs/v2/results-intro.html#test-suite

A test set is a subdivision of a test suite in a given LAVA job:

  https://validation.linaro.org/static/docs/v2/results-intro.html#test-set-results

LAVA jobs can contain multiple test suites, each of them contain
some test cases which can be optionally grouped as sets.

So if we want to split big test suites across several jobs, test
sets are not what we're after.

Using test sets only make sense when running big jobs with a
large test suite that needs to be split into smaller chunks.
We're not using these at the moment, and it goes against what we
discussed today (i.e. having smaller tests with parts of a larger
test suite).

Regarding the "broken" part, we're currently storing test cases
twice: as a "default" test set as well as the actual test suite.
So we have the same list of test cases in one test set entry and
one test suite entry - which is why the test set part appears to
be redundant.

> >     - Suggestion: Remove test sets to simplify things.  If we need a
> >       better backend solution, we should consider SQUAD and rethink this
> >       from a global point of view.
>
> Not sure how squad helps here.  Seems it was designed with this same
> limitation.  IMO we need the raw data to be coherent, and cleanly broken
> down, and the frontend/UI used to view needs to be independent from
> that.

It's important to consider what SQUAD can bring to KernelCI -
with what it can do now or what can be done with some changes.
It may turn out to not be a good fit, but we can't ignore the
fact that it was designed for a very similar use-case: storing
results from multiple platforms running tests, and presenting
these results.  Maybe SQUAD can provide a coherent data model
that works perfectly well for KernelCI, maybe we need to mix it
or merge it with our current backend solution somehow.

We now have the KernelCI folks learning about SQUAD and the SQUAD
folks learning about KernelCI, we'll see where that leads us.

> >     - If some test suites need to be broken down, then we can have
> >       several smaller test plans.  We could also consider running
> >       several test plans in a row with a single LAVA job. Example:
> >       https://lava.collabora.co.uk/results/1240317
> >       - [drue/Mark] - In LKFT we run LTP across several test runs, and
> >         they end up getting called “suite” LTP-subsetA, LTP-subsetB,
> >         etc, because squad doesn’t support sets.
>
> IMO, acknowleding that some suites need to be broken up is acknowleging
> that he notion of test sets is useful.  Just concatening the suite name
> and set name is a hack, not a solution.

As I explained above, test sets were not designed to be a group
of test suites to run across multiple jobs, but parts of a test
suite to run inside a job.  Indeed, using naming convention is
fragile.  What we could do however is have some reliable way of
telling that some test jobs are meant to be grouped together,
either with meta-data or maybe with something new in LAVA.  They
could just be using the same test suite, with different test
cases each.

Whether this should be visible at all from a user point of view,
I don't think so.  Say, if we split a test suite arbitrarily to
better spread the load, get more reliable test runs and run tests
in parallel, would this need to be shown to the user?  I would
have thought that we just need to present the list of all the
test cases for that suite.

> >         - [broonie] The goal here is to both get faster runs when
> >           there’s multiple boards and to improve robustness against test
> >           runs crashing and losing a lot of results on larger
> >           testsuites.
>
> So in LKFT, the seprate LTP sub-sets are run on separate boots?  Seems
> like an LKFT implementation decision, not somthing you would want to
> force on test writers, nor assume in the data model.
>
> >       - [ana] I would like to remove it because what we currently have
> >         doesn’t work and we don’t use it
>
> By "we" you mean current kernelCI users and kernelCI tests.  IMO, that's
> not a very good sample size, as there's currently only a few of us, and
> we don't have very many tests, and the tests that we have are pretty
> small.
>
> >       - [matt] this was all designed a while ago. A test set is supposed
> >         to be a part of a bigger test suite. If it’s not useful now, we
> >         can scrap it. There are performance issues too that you can see
> >         when viewing the test data. +1 remove
>
> Are you saying LAVA is dropping the notion of test sets?
>
> >     - Decision: No objections to removing test sets
>
> I object.  (Can we please be careful about decision based on "no
> objections" when not everyone is able to participate in the call.)

You have the minutes (thanks Dan) to reply to here, and PR
comments (if anything concrete lands on Github).  Maybe we can
have another call in your timezone to discuss this further?

> What happens when trying to support LAVA test jobs that actually use
> suite, set and case?  We're trying to expand kernelCI users and test
> cases and there are lots of LAVA users out there with large test suites.
> They may not be sharing results (yet), but it would be a shame to have
> to have users restructure their test jobs because kCI doesn't support
> test sets.

We don't have to impose that on users, but we need to be clear on
how we deal with the LAVA test results: how we store them in the
backend, how we report them.  The backend is not a LAVA UI proxy,
we don't have to stick to the LAVA data structure.  If we do
really think that having another layer of grouping of the test
cases is useful, then we can make such a conscious decision -
knowing the pros and cons, and the fact that it tends to mean
running longer jobs.

> Also, what about kselftest?  It has several suites (e.g. each subdir of
> tools/testing/selftest) and within each suite can have sets
> (e.g. net/forwarding, ftrace/test.d/*)
>
> I'd really like kernelCI to scale to multiple forms of frontend/UI etc.,
> and for that the data model needs to be generic, flexible and especially
> scalable.
> To do that, I think it's very premature to eliminate one level of test
> hierarchy, for the primary reason that it's convenient in the short
> term.

It's important to consider the whole picture, in particular
whether SQUAD would be a better solution to store and report test
results.  It's also important to start sending email reports with
test results.  I guess the simplest thing we could do is to
ignore test sets for the time being (on the UI and emails).  Once
we've actually thought through how we want to be dealing with
test results in the longer term, for more complex test suites,
then we can decide whether we should fix or implement new things
in the kernelci-backend or move to an alternative system.

How does that sound?

Guillaume

> >   - Reshaping test configurations:
> >     - YAML-based description now being tested
> >     - In the process of adding all the devices and test plans to achieve
> >       same test coverage as we currently have with the “device_map”
> >     - Removing dead LAVA v1 code
> >     - Aiming at eventually merging kernelci-build and lava-ci.
> >     - Looking for feedback on
> >       https://github.com/kernelci/lava-ci-staging/pull/96
> >   - Multiple bisections run for single regression: investigating...
> >
> > - Ana
> >   - Reporting test suite results:
> >     - Sent v2 of mail reports last week
> >       https://github.com/kernelci/kernelci-backend/pull/70 Comments
> >       welcome. This is a first step to start getting the results of the
> >       test suites by mail.
> >     - Sent small PR fixing a problem with the template for sending
> >       duplicate mails
> >   - Start reporting on regressions: this is blocked by the point
> >     mentioned above about test suites/sets/cases. I would like to send a
> >     PR to the backend for tracking and reporting test suites regressions
> >     in the test suites but If we want to use test sets, some code needs
> >     to be added to fix its usage in the backend.
>
> I would much prefer fixing the backend than removing test sets all
> together.
>
> >   - Documentations update
> >     - Added 2 new pages documenting the test suites (on-going) and the
> >       rootfs images used by the test suites.
> >     - http://wiki.kernelci.org now tell readers about the github wiki.
> >     - Help welcome savaging the useful parts from wiki.kci.o
>
> IMO, there is a lot useful there.
>
> >     - Diagrams and other graphics can be added in the github wiki via
> >       git : git@github.com:kernelci/kernelci-doc.wiki.git
> >     - [drue] it looks like you can’t host a github wiki at its own url,
> >       can you? I looked.
> >       - [ana] no, but it’s stored in markdown in a git repo and we could
> >         use something else to render and host it in the future.
> >     - Github permissions - right now I can’t ask people to review
> >       kernelci related pull requests
> >       - [matt] https://github.com/orgs/kernelci/teams exists but needs
> >         to be made better
>
> Kevin
>
> > - Milosz
> >   - Kernelci testing data presentation in SQUAD:
> >     https://staging-qa-reports.linaro.org/kernelci/
> >   - [ana] can results be filtered by date?
> >     - [milosz] you can use theAPI to get that. In the UI it’s sorted by
> >       date, but you can’t filter by date
> >   - [matt] it looks better than kernelci’s views. Some of the sorting
> >     and mapping can be improved
> >     - https://staging-qa-reports.linaro.org/kernelci/ shows it by
> >       branch, but what if someone wants to see it by board or by
> >       architecture/defconfig
> >     - [milosz] i’m working on filtering by environment (architecture,
> >       board), so that users can view results by board, or by config
> >       - [milosz] I’m currently working on
> >         https://github.com/Linaro/squad/issues/245
> >     - [guill] custom views would be really nice so that an individual
> >       could have a search for what they’re interested in that could be
> >       saved in a link, and shared that way
> >       - [milosz] this is a nice idea
> >       - [matt] here’s an example of a multi-field search in kernelci
> >         https://kernelci.org/boot/?apq8016-sbc%20lab-mhart%20master%20next
> >         - lava/squad doesn’t do that because of django’s default filter
> >           implementation
>
> Configurable, custom views is really the killer feature, but also the
> hardest part.  That's why, IMO, we need to get the data model right, and
> start using more generic analytic stacks (e.g. Elastic stack[1], or
> Apache Spark, etc.)
>
> Kevin
>
> [1] https://www.elastic.co/webinars/introduction-elk-stack
>
>
> 
>

Re: [kernelci] Meeting Minutes for 2018-08-20

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 3249 bytes --]

On Mon, 20 Aug 2018 at 21:21 Guillaume Tucker <guillaume.tucker@gmail.com>
wrote:

> On Mon, Aug 20, 2018 at 7:45 PM Kevin Hilman <khilman@baylibre.com> wrote:
>
> > >        They are mostly a side effect of parsing LAVA callback data,
> >
> > Not sure I'd call it a side effect.  LAVA supports the notion of suite,
> > set and case, and we need to deal with that if we're handling LAVA data
> > (even if there are no current kernelCI tests using it.)
>
> Yes we need to be able to deal with the data sent by LAVA, but it
> doesn't mean we need to follow the way the data is formatted in
> the LAVA callbacks.
>

Like I said in the meeting it's not just a side effect of what LAVA's
doing, it's also a practical thing for large testsuites especially when
there's some risk of the job failing.


> So if we want to split big test suites across several jobs, test
> sets are not what we're after.
>

According to what the LAVA docs say but this isn't the LAVA data model,
it's the kernelci one! To be honest I'm not sure the LAVA documentation
reflects what users are actually doing with the feature anyway.


> Whether this should be visible at all from a user point of view,
> I don't think so.  Say, if we split a test suite arbitrarily to
> better spread the load, get more reliable test runs and run tests
> in parallel, would this need to be shown to the user?  I would
> have thought that we just need to present the list of all the
> test cases for that suite.
>

I tend to agree that it should be somewhat hidden from the user if it's
there. I don't think we can completely hide it as at some point users will
want to see things like the full job log.


>
> > >         - [broonie] The goal here is to both get faster runs when
> > >           there’s multiple boards and to improve robustness against
> test
> > >           runs crashing and losing a lot of results on larger
> > >           testsuites.
> >
> > So in LKFT, the seprate LTP sub-sets are run on separate boots?  Seems
> > like an LKFT implementation decision, not somthing you would want to
> > force on test writers, nor assume in the data model.
>

I'd say it's definitely a useful feature for test writers to have available
to them - it's solving a real problem that some testsuites encounter.


> > >     - Decision: No objections to removing test sets
> >
> > I object.  (Can we please be careful about decision based on "no
> > objections" when not everyone is able to participate in the call.)
>
> You have the minutes (thanks Dan) to reply to here, and PR
> comments (if anything concrete lands on Github).  Maybe we can
> have another call in your timezone to discuss this further?
>

BTW, one thing I meant to bring up with moving things more to the list:
instead of putting things in the document for discussion it'd probably help
if we were to instead put them in mails to the list (I'll try to do this
the next time I have some topic).


> > Also, what about kselftest?  It has several suites (e.g. each subdir of
> > tools/testing/selftest) and within each suite can have sets
> > (e.g. net/forwarding, ftrace/test.d/*)
>

That seems like a good candidate for splitting up FWIW.

[-- Attachment #2: Type: text/html, Size: 4573 bytes --]

Re: [kernelci] Meeting Minutes for 2018-08-20

[Kevin Hilman]

"Guillaume Tucker" <guillaume.tucker@gmail.com> writes:

> On Mon, Aug 20, 2018 at 7:45 PM Kevin Hilman <khilman@baylibre.com> wrote:
>>
>> Dan, thanks again for the minutes!
>>
>> "Dan Rue" <dan.rue@linaro.org> writes:
>>
>> > - Attendees: Ana, Guillaume, Matt, Milosz, Rafael, Dan, Mark
>> >
>> > - Guillaume
>> >   - Dealing with test results:
>> >     - We currently have test suites/sets/cases, but test sets aren’t
>> >       used and their implementation is broken.
>>
>> What is broken in the implementation?  LAVA? backend? frontend?
>
> The way we currently deal with test sets in the backend does not
> add any value with the short test plans we currently have.  As I
> explained in my previous email thread, both the test suite and
> test set entries have a list of test cases, making test sets
> effectively redundant.

For the test plans we have, I see how that is redundant, but for test
plans that actually include test sets, is it still redunant?  

My understanding was that test suites contain test sets (at least one
"default" if no others are defined), and that test sets contain test
cases.

>> >        They are mostly a side effect of parsing LAVA callback data,
>>
>> Not sure I'd call it a side effect.  LAVA supports the notion of suite,
>> set and case, and we need to deal with that if we're handling LAVA data
>> (even if there are no current kernelCI tests using it.)
>
> Yes we need to be able to deal with the data sent by LAVA, but it
> doesn't mean we need to follow the way the data is formatted in
> the LAVA callbacks.

Fully agree.  But as the examples that have already been presented (LTP,
kselftest, IGT, etc.) the 3-level hierarchy seems quite useful, not just
for LAVA.  I also suspect that when we start looking at test report
emails with multiple test suites, and lots of cases, having test sets
will be useful to make sense out of all the tests.

>> >       they are not meaningful from a user point of view.
>>
>> I disagree.  They are extrememly useful for large test suites (LTP,
>> kselftest, Android CTP, etc.)
>>
>> If you mean our *current* ways of visualizing them are not meaningful, I
>> agree, but that's different from whether they are useful at all.
>
> Right now, it's not clear what test sets are all about from our
> backend point of view.  There also seems to be some confusion
> around what they're supposed to be from a LAVA point of view, so
> here's some clarification from the docs:
>
> A test case is an individual test in a LAVA job, I think we all
> know that that is (run a command, get a single result).
>
> A test suite is a group of test cases run in a job:
>
>   https://validation.linaro.org/static/docs/v2/results-intro.html#test-suite
>
> A test set is a subdivision of a test suite in a given LAVA job:
>
>   https://validation.linaro.org/static/docs/v2/results-intro.html#test-set-results
>
> LAVA jobs can contain multiple test suites, each of them contain
> some test cases which can be optionally grouped as sets.
>
> So if we want to split big test suites across several jobs, test
> sets are not what we're after.

Agreed, I'm not talking about splittng across jobs. That just came up
because it seems to be what LKFT does, but I think that's a completely
different subject.

> Using test sets only make sense when running big jobs with a
> large test suite that needs to be split into smaller chunks.

Yes.

> We're not using these at the moment, and it goes against what we
> discussed today (i.e. having smaller tests with parts of a larger
> test suite).

I guess this is part of what I don't understand in what was discussed.
I don't understand how "smaller tests with parts of larger test suite"
implies that you only need 2 levels (suite, case.)  Even if you have 20
test cases in a test suite, it makes sense to me that you'd what to
logically group them into (sub)sets even if just for better reporting
purpose.

A simple example is this CAN bus test suite recently merged into
AGL[1]. There's only ~35 test cases, but we still grouped them into 5 sets.
One suite with ~35 tests is a lot to swallow from a
reporting/visualization PoV.

> Regarding the "broken" part, we're currently storing test cases
> twice: as a "default" test set as well as the actual test suite.
> So we have the same list of test cases in one test set entry and
> one test suite entry - which is why the test set part appears to
> be redundant.

In that case, seems like a simple fix to just not store them as part of
the test suite.

>> >     - Suggestion: Remove test sets to simplify things.  If we need a
>> >       better backend solution, we should consider SQUAD and rethink this
>> >       from a global point of view.
>>
>> Not sure how squad helps here.  Seems it was designed with this same
>> limitation.  IMO we need the raw data to be coherent, and cleanly broken
>> down, and the frontend/UI used to view needs to be independent from
>> that.
>
> It's important to consider what SQUAD can bring to KernelCI -
> with what it can do now or what can be done with some changes.
> It may turn out to not be a good fit, but we can't ignore the
> fact that it was designed for a very similar use-case: storing
> results from multiple platforms running tests, and presenting
> these results.  Maybe SQUAD can provide a coherent data model
> that works perfectly well for KernelCI, maybe we need to mix it
> or merge it with our current backend solution somehow.
>
> We now have the KernelCI folks learning about SQUAD and the SQUAD
> folks learning about KernelCI, we'll see where that leads us.

I agree that we should be considering SQUAD.  As I've said a few times
now, I think we need more experimentation in the UI/visualization front.
I'm not opposed to that at all, and would love to see more.

Just in the context of the debate over test sets, I didn't understand
what SQUAD has to offer since it was a design decision to not use them,
but instead just create a new virtual suite which concatenates the suite
name and the set name (yuck.)

>> >     - If some test suites need to be broken down, then we can have
>> >       several smaller test plans.  We could also consider running
>> >       several test plans in a row with a single LAVA job. Example:
>> >       https://lava.collabora.co.uk/results/1240317
>> >       - [drue/Mark] - In LKFT we run LTP across several test runs, and
>> >         they end up getting called “suite” LTP-subsetA, LTP-subsetB,
>> >         etc, because squad doesn’t support sets.
>>
>> IMO, acknowleding that some suites need to be broken up is acknowleging
>> that he notion of test sets is useful.  Just concatening the suite name
>> and set name is a hack, not a solution.
>
> As I explained above, test sets were not designed to be a group
> of test suites to run across multiple jobs, but parts of a test
> suite to run inside a job.

Sorry for the confusion, I'm not sure how I communicated that, but I
didn't mean to suggest that.  The hierarchy is suites contain sets
contain cases.

> Indeed, using naming convention is
> fragile.  What we could do however is have some reliable way of
> telling that some test jobs are meant to be grouped together,
> either with meta-data or maybe with something new in LAVA.  They
> could just be using the same test suite, with different test
> cases each.
>
> Whether this should be visible at all from a user point of view,
> I don't think so.  Say, if we split a test suite arbitrarily to
> better spread the load, get more reliable test runs and run tests
> in parallel, would this need to be shown to the user?  I would
> have thought that we just need to present the list of all the
> test cases for that suite.

IMO breaking up tests across multiple jobs is a totally different
subject.  First we need to resolve the basic hierarchy that the data
model should support within a single job.

>
>> >         - [broonie] The goal here is to both get faster runs when
>> >           there’s multiple boards and to improve robustness against test
>> >           runs crashing and losing a lot of results on larger
>> >           testsuites.
>>
>> So in LKFT, the seprate LTP sub-sets are run on separate boots?  Seems
>> like an LKFT implementation decision, not somthing you would want to
>> force on test writers, nor assume in the data model.
>>
>> >       - [ana] I would like to remove it because what we currently have
>> >         doesn’t work and we don’t use it
>>
>> By "we" you mean current kernelCI users and kernelCI tests.  IMO, that's
>> not a very good sample size, as there's currently only a few of us, and
>> we don't have very many tests, and the tests that we have are pretty
>> small.
>>
>> >       - [matt] this was all designed a while ago. A test set is supposed
>> >         to be a part of a bigger test suite. If it’s not useful now, we
>> >         can scrap it. There are performance issues too that you can see
>> >         when viewing the test data. +1 remove
>>
>> Are you saying LAVA is dropping the notion of test sets?
>>
>> >     - Decision: No objections to removing test sets
>>
>> I object.  (Can we please be careful about decision based on "no
>> objections" when not everyone is able to participate in the call.)
>
> You have the minutes (thanks Dan) to reply to here, 

Yes, which is what I've done. :)

The minutes read as though a decision was made, which should not happen
(IMO) without all the interested partiese, especially when the topic
been raised earlier, and I had objected then as well[2].

I should've probably just interpreted that "Decision" line as "Proposal"
and then I would not have gotten worked up about it.

> and PR comments (if anything concrete lands on Github).
> Maybe we can have another call in your timezone to discuss this
> further?
>
>> What happens when trying to support LAVA test jobs that actually use
>> suite, set and case?  We're trying to expand kernelCI users and test
>> cases and there are lots of LAVA users out there with large test suites.
>> They may not be sharing results (yet), but it would be a shame to have
>> to have users restructure their test jobs because kCI doesn't support
>> test sets.
>
> We don't have to impose that on users, but we need to be clear on
> how we deal with the LAVA test results: how we store them in the
> backend, how we report them.  The backend is not a LAVA UI proxy,
> we don't have to stick to the LAVA data structure.

Agreed, we don't have to follow LAVA, but we have to deal with LAVA
data, and following it is one (easy) choice, especially when it's proven
to be useful for large test suites.

> If we do
> really think that having another layer of grouping of the test
> cases is useful, then we can make such a conscious decision -
> knowing the pros and cons, and the fact that it tends to mean
> running longer jobs.

Why does it mean longer running jobs?   A few extra shell commands to
run "lava-test-set start <name>" and "lava-test-set stop" are not going
to (noticably) increase the runtime.

>> Also, what about kselftest?  It has several suites (e.g. each subdir of
>> tools/testing/selftest) and within each suite can have sets
>> (e.g. net/forwarding, ftrace/test.d/*)
>>
>> I'd really like kernelCI to scale to multiple forms of frontend/UI etc.,
>> and for that the data model needs to be generic, flexible and especially
>> scalable.
>> To do that, I think it's very premature to eliminate one level of test
>> hierarchy, for the primary reason that it's convenient in the short
>> term.
>
> It's important to consider the whole picture, in particular
> whether SQUAD would be a better solution to store and report test
> results.

No objections to more variety in test reporting.  There will be lots of
ways to slice, dice and view the test data.

> It's also important to start sending email reports with
> test results.

Agreed, I think we should merge Ana's PR and start getting emails and
then start iterating on it.

Even looking at the gist examples[3] from that PR, the IGT one is
already massive, and IMO, difficult to process (1 test-suite: 185 test
cases !!).

Similar to the boot report emails, I suspect they will evolve to be a
quick summary, and just have links to the UI for all the details, rather
than a full list of all the test cases.

e.g. in the current email examples, it's missing the high-level
summary:  How many test suites ran? how many passed/failed?  on how many
boards. (Note that I dont' think this shoudl prevent us from merging
Ana's PR, but we should continue to iterate/improve.  I'll comment on
the PR directly)

> I guess the simplest thing we could do is to
> ignore test sets for the time being (on the UI and emails).  Once
> we've actually thought through how we want to be dealing with
> test results in the longer term, for more complex test suites,
> then we can decide whether we should fix or implement new things
> in the kernelci-backend or move to an alternative system.
>
> How does that sound?

Sounds OK, but IMO we're already running complex enough test suites that
are begging for test sets (e.g. IGT, kselftest)

We could also quite simply add sets to the "simple" test plan so that
our emails and UI can evolve accordingly with actual data in the
backend.  The emails/UIs can still choose to ignore (or concatenate) the
sets, but at least there will be a choice to be able to to finer-graned
grouping/reporting/visualization, and the data model can support it.

Kevin

[1] https://git.automotivelinux.org/src/qa-testdefinitions/tree/test-suites/short-smoke/test_can.yaml
[2] https://groups.io/g/kernelci/message/36
[3] https://gist.github.com/ana/af7f153384e95588132c588dac983215
[4] https://github.com/kernelci/lava-ci-staging/commit/b62b2823787cf67f96f82da2248969c0feef3847

Re: [kernelci] Meeting Minutes for 2018-08-20

[Milosz Wasilewski]

On 20 August 2018 at 21:20, Guillaume Tucker <guillaume.tucker@gmail.com> wrote:

>> >     - Suggestion: Remove test sets to simplify things.  If we need a
>> >       better backend solution, we should consider SQUAD and rethink this
>> >       from a global point of view.
>>
>> Not sure how squad helps here.  Seems it was designed with this same
>> limitation.  IMO we need the raw data to be coherent, and cleanly broken
>> down, and the frontend/UI used to view needs to be independent from
>> that.
>
> It's important to consider what SQUAD can bring to KernelCI -
> with what it can do now or what can be done with some changes.
> It may turn out to not be a good fit, but we can't ignore the
> fact that it was designed for a very similar use-case: storing
> results from multiple platforms running tests, and presenting
> these results.  Maybe SQUAD can provide a coherent data model
> that works perfectly well for KernelCI, maybe we need to mix it
> or merge it with our current backend solution somehow.
>
> We now have the KernelCI folks learning about SQUAD and the SQUAD
> folks learning about KernelCI, we'll see where that leads us.
>

Guillaume,
Maybe we can speed up this learning process a bit by arranging a f2f
meeting some time this/next week in Cambridge? I'm free all day on
Thursday and I can visit you or we can meet at Makespace.

milosz

Re: [kernelci] Meeting Minutes for 2018-08-20

[Tomeu Vizoso]

Hi Kevin,

On 08/21/2018 01:52 AM, Kevin Hilman wrote:
> "Guillaume Tucker" <guillaume.tucker@gmail.com> writes:
> 
>> On Mon, Aug 20, 2018 at 7:45 PM Kevin Hilman <khilman@baylibre.com> wrote:
>>>
>>> Dan, thanks again for the minutes!
>>>
>>> "Dan Rue" <dan.rue@linaro.org> writes:
>>>
>>>> - Attendees: Ana, Guillaume, Matt, Milosz, Rafael, Dan, Mark
>>>>
>>>> - Guillaume
>>>>    - Dealing with test results:
>>>>      - We currently have test suites/sets/cases, but test sets aren’t
>>>>        used and their implementation is broken.
>>>
>>> What is broken in the implementation?  LAVA? backend? frontend?
>>
>> The way we currently deal with test sets in the backend does not
>> add any value with the short test plans we currently have.  As I
>> explained in my previous email thread, both the test suite and
>> test set entries have a list of test cases, making test sets
>> effectively redundant.
> 
> For the test plans we have, I see how that is redundant, but for test
> plans that actually include test sets, is it still redunant?
> 
> My understanding was that test suites contain test sets (at least one
> "default" if no others are defined), and that test sets contain test
> cases.

Can you explain what do you think KernelCI should do with the hierarchy 
information? That would allow me to understand the need for storing tests 
within a multiple level hierarchy.

For the use cases I currently care about, it's enough to have test plans 
(corresponding to a kernel subsystem, so we know where to send regression 
reports) containing tests.

Note that for big test suites, it's normal to have 6 levels or more, so 
I'm a bit concerned that if we decide to go now with 4 levels, we may 
need we need more in the future. But right now I'm having trouble 
figuring out what kernelci would do with the hierarchy information.

Thanks,

Tomeu

Re: [kernelci] Meeting Minutes for 2018-08-20

[Guillaume Tucker]

Hi Milosz,

On Tue, Aug 21, 2018 at 11:20 AM Milosz Wasilewski
<milosz.wasilewski@linaro.org> wrote:
>
> On 20 August 2018 at 21:20, Guillaume Tucker <guillaume.tucker@gmail.com> wrote:
>
> >> >     - Suggestion: Remove test sets to simplify things.  If we need a
> >> >       better backend solution, we should consider SQUAD and rethink this
> >> >       from a global point of view.
> >>
> >> Not sure how squad helps here.  Seems it was designed with this same
> >> limitation.  IMO we need the raw data to be coherent, and cleanly broken
> >> down, and the frontend/UI used to view needs to be independent from
> >> that.
> >
> > It's important to consider what SQUAD can bring to KernelCI -
> > with what it can do now or what can be done with some changes.
> > It may turn out to not be a good fit, but we can't ignore the
> > fact that it was designed for a very similar use-case: storing
> > results from multiple platforms running tests, and presenting
> > these results.  Maybe SQUAD can provide a coherent data model
> > that works perfectly well for KernelCI, maybe we need to mix it
> > or merge it with our current backend solution somehow.
> >
> > We now have the KernelCI folks learning about SQUAD and the SQUAD
> > folks learning about KernelCI, we'll see where that leads us.
> >
>
> Guillaume,
> Maybe we can speed up this learning process a bit by arranging a f2f
> meeting some time this/next week in Cambridge? I'm free all day on
> Thursday and I can visit you or we can meet at Makespace.

Thanks for suggesting this.  My only worry with meeting up is
that it's only going to directly benefit you and I, but we have
many other people involved with this work.  I think it's better
that we all learn, maybe slower, but at the same pace.

Guillaume

> 
>

Re: [kernelci] Meeting Minutes for 2018-08-20

[Dan Rue]

On Wed, Aug 22, 2018 at 12:11:06PM +0200, Tomeu Vizoso wrote:
> Hi Kevin,
> 
> On 08/21/2018 01:52 AM, Kevin Hilman wrote:
> > "Guillaume Tucker" <guillaume.tucker@gmail.com> writes:
> > 
> > > On Mon, Aug 20, 2018 at 7:45 PM Kevin Hilman <khilman@baylibre.com> wrote:
> > > > 
> > > > Dan, thanks again for the minutes!
> > > > 
> > > > "Dan Rue" <dan.rue@linaro.org> writes:
> > > > 
> > > > > - Attendees: Ana, Guillaume, Matt, Milosz, Rafael, Dan, Mark
> > > > > 
> > > > > - Guillaume
> > > > >    - Dealing with test results:
> > > > >      - We currently have test suites/sets/cases, but test sets aren’t
> > > > >        used and their implementation is broken.
> > > > 
> > > > What is broken in the implementation?  LAVA? backend? frontend?
> > > 
> > > The way we currently deal with test sets in the backend does not
> > > add any value with the short test plans we currently have.  As I
> > > explained in my previous email thread, both the test suite and
> > > test set entries have a list of test cases, making test sets
> > > effectively redundant.
> > 
> > For the test plans we have, I see how that is redundant, but for test
> > plans that actually include test sets, is it still redunant?
> > 
> > My understanding was that test suites contain test sets (at least one
> > "default" if no others are defined), and that test sets contain test
> > cases.
> 
> Can you explain what do you think KernelCI should do with the hierarchy
> information? That would allow me to understand the need for storing tests
> within a multiple level hierarchy.
> 
> For the use cases I currently care about, it's enough to have test plans
> (corresponding to a kernel subsystem, so we know where to send regression
> reports) containing tests.
> 
> Note that for big test suites, it's normal to have 6 levels or more, so I'm
> a bit concerned that if we decide to go now with 4 levels, we may need we
> need more in the future. But right now I'm having trouble figuring out what
> kernelci would do with the hierarchy information.

I can give an example that we have run into in LKFT that would benefit
from my understanding of what test sets are. Since SQUAD doesn't have
test sets, I guess this is like a preview of what it might look like
without them.

Browse to
https://qa-reports.linaro.org/lkft/linux-mainline-oe/build/v4.18-11219-gad1d69735878/

This is the test results of a recent mainline revision. We run LTP, and
have it broken up into 20 separate test sets, which each correspond to a
runfile in LTP, and each set is run in a separate lava job.

We want SQUAD to also show this distinction, since there are a lot of
tests. Now, click on the "+ Display more" link under the "Metadata" box.
You'll notice an explosion of metadata which is largely duplicate - the
version of LTP for each test set which culminates into this beauty:

    suite_versions:
        {'kselftest': '4.18+gitAUTOINC+ad1d697358'}
        {'libhugetlbfs': '2.20+gitAUTOINC+02df38e93e+next'}
        {'ltp-cap_bounds-tests': '20180515'}
        {'ltp-containers-tests': '20180515'}
        {'ltp-cve-tests': '20180515'}
        {'ltp-fcntl-locktests-tests': '20180515'}
        {'ltp-filecaps-tests': '20180515'}
        {'ltp-fs-tests': '20180515'}
        {'ltp-fs_bind-tests': '20180515'}
        {'ltp-fs_perms_simple-tests': '20180515'}
        {'ltp-fsx-tests': '20180515'}
        {'ltp-hugetlb-tests': '20180515'}
        {'ltp-io-tests': '20180515'}
        {'ltp-ipc-tests': '20180515'}
        {'ltp-math-tests': '20180515'}
        {'ltp-nptl-tests': '20180515'}
        {'ltp-open-posix-tests': '20180515'}
        {'ltp-pty-tests': '20180515'}
        {'ltp-sched-tests': '20180515'}
        {'ltp-securebits-tests': '20180515'}
        {'ltp-syscalls-tests': '20180515'}
        {'ltp-timers-tests': '20180515'}

Since SQUAD does not understand test sets, in order to tell squad what
version of each test 'suite' is running, we have to submit it for each
test 'set'. So we concatenate the suite name with the set name during
submission.

So now we know what version each test set is running, but it creates a
bit of a mess which is hard to unroll in reporting :(

All this discussion leads me to believe that the prudent course for the
time being seems to be to 'do nothing', and revisit this at a later date
once things are more clear.

Dan

Re: [kernelci] Meeting Minutes for 2018-08-20

[Kevin Hilman]

"Tomeu Vizoso" <tomeu.vizoso@collabora.com> writes:

> Hi Kevin,
>
> On 08/21/2018 01:52 AM, Kevin Hilman wrote:
>> "Guillaume Tucker" <guillaume.tucker@gmail.com> writes:
>>
>>> On Mon, Aug 20, 2018 at 7:45 PM Kevin Hilman <khilman@baylibre.com> wrote:
>>>>
>>>> Dan, thanks again for the minutes!
>>>>
>>>> "Dan Rue" <dan.rue@linaro.org> writes:
>>>>
>>>>> - Attendees: Ana, Guillaume, Matt, Milosz, Rafael, Dan, Mark
>>>>>
>>>>> - Guillaume
>>>>>    - Dealing with test results:
>>>>>      - We currently have test suites/sets/cases, but test sets aren’t
>>>>>        used and their implementation is broken.
>>>>
>>>> What is broken in the implementation?  LAVA? backend? frontend?
>>>
>>> The way we currently deal with test sets in the backend does not
>>> add any value with the short test plans we currently have.  As I
>>> explained in my previous email thread, both the test suite and
>>> test set entries have a list of test cases, making test sets
>>> effectively redundant.
>>
>> For the test plans we have, I see how that is redundant, but for test
>> plans that actually include test sets, is it still redunant?
>>
>> My understanding was that test suites contain test sets (at least one
>> "default" if no others are defined), and that test sets contain test
>> cases.
>
> Can you explain what do you think KernelCI should do with the
> hierarchy information? That would allow me to understand the need for
> storing tests within a multiple level hierarchy.

For me, it's just about reporting an visualization (something we are not
currently doing great at).  IMO, even a test suite with > 20 test cases
really needs to be broken up a bit for ease of making reports that are
easy to understand.

> For the use cases I currently care about, it's enough to have test
> plans (corresponding to a kernel subsystem, so we know where to send
> regression reports) containing tests.

For low-level kernel focused testing that's a minimum.  But I hope that
kernelCI can grow to handle lots of differnet kinds of test suites
(power, performance, benchmarks, long-running stability, etc.) as well
as distro/OS focused testing (things like Android CTS.)  

> Note that for big test suites, it's normal to have 6 levels or more,
> so I'm a bit concerned that if we decide to go now with 4 levels, we
> may need we need more in the future. But right now I'm having trouble
> figuring out what kernelci would do with the hierarchy information.

I agree that large test suites might need more levels, and for that, it
might be as simple as allowing test sets to contain test sets as well as
test cases.

Kevin

Re: [kernelci] Meeting Minutes for 2018-08-20

[Kevin Hilman]

"Dan Rue" <dan.rue@linaro.org> writes:

> On Wed, Aug 22, 2018 at 12:11:06PM +0200, Tomeu Vizoso wrote:
>> Hi Kevin,
>> 
>> On 08/21/2018 01:52 AM, Kevin Hilman wrote:
>> > "Guillaume Tucker" <guillaume.tucker@gmail.com> writes:
>> > 
>> > > On Mon, Aug 20, 2018 at 7:45 PM Kevin Hilman <khilman@baylibre.com> wrote:
>> > > > 
>> > > > Dan, thanks again for the minutes!
>> > > > 
>> > > > "Dan Rue" <dan.rue@linaro.org> writes:
>> > > > 
>> > > > > - Attendees: Ana, Guillaume, Matt, Milosz, Rafael, Dan, Mark
>> > > > > 
>> > > > > - Guillaume
>> > > > >    - Dealing with test results:
>> > > > >      - We currently have test suites/sets/cases, but test sets aren’t
>> > > > >        used and their implementation is broken.
>> > > > 
>> > > > What is broken in the implementation?  LAVA? backend? frontend?
>> > > 
>> > > The way we currently deal with test sets in the backend does not
>> > > add any value with the short test plans we currently have.  As I
>> > > explained in my previous email thread, both the test suite and
>> > > test set entries have a list of test cases, making test sets
>> > > effectively redundant.
>> > 
>> > For the test plans we have, I see how that is redundant, but for test
>> > plans that actually include test sets, is it still redunant?
>> > 
>> > My understanding was that test suites contain test sets (at least one
>> > "default" if no others are defined), and that test sets contain test
>> > cases.
>> 
>> Can you explain what do you think KernelCI should do with the hierarchy
>> information? That would allow me to understand the need for storing tests
>> within a multiple level hierarchy.
>> 
>> For the use cases I currently care about, it's enough to have test plans
>> (corresponding to a kernel subsystem, so we know where to send regression
>> reports) containing tests.
>> 
>> Note that for big test suites, it's normal to have 6 levels or more, so I'm
>> a bit concerned that if we decide to go now with 4 levels, we may need we
>> need more in the future. But right now I'm having trouble figuring out what
>> kernelci would do with the hierarchy information.
>
> I can give an example that we have run into in LKFT that would benefit
> from my understanding of what test sets are. Since SQUAD doesn't have
> test sets, I guess this is like a preview of what it might look like
> without them.
>
> Browse to
> https://qa-reports.linaro.org/lkft/linux-mainline-oe/build/v4.18-11219-gad1d69735878/
>
> This is the test results of a recent mainline revision. We run LTP, and
> have it broken up into 20 separate test sets, which each correspond to a
> runfile in LTP, and each set is run in a separate lava job.
>
> We want SQUAD to also show this distinction, since there are a lot of
> tests. Now, click on the "+ Display more" link under the "Metadata" box.
> You'll notice an explosion of metadata which is largely duplicate - the
> version of LTP for each test set which culminates into this beauty:
>
>     suite_versions:
>         {'kselftest': '4.18+gitAUTOINC+ad1d697358'}
>         {'libhugetlbfs': '2.20+gitAUTOINC+02df38e93e+next'}
>         {'ltp-cap_bounds-tests': '20180515'}
>         {'ltp-containers-tests': '20180515'}
>         {'ltp-cve-tests': '20180515'}
>         {'ltp-fcntl-locktests-tests': '20180515'}
>         {'ltp-filecaps-tests': '20180515'}
>         {'ltp-fs-tests': '20180515'}
>         {'ltp-fs_bind-tests': '20180515'}
>         {'ltp-fs_perms_simple-tests': '20180515'}
>         {'ltp-fsx-tests': '20180515'}
>         {'ltp-hugetlb-tests': '20180515'}
>         {'ltp-io-tests': '20180515'}
>         {'ltp-ipc-tests': '20180515'}
>         {'ltp-math-tests': '20180515'}
>         {'ltp-nptl-tests': '20180515'}
>         {'ltp-open-posix-tests': '20180515'}
>         {'ltp-pty-tests': '20180515'}
>         {'ltp-sched-tests': '20180515'}
>         {'ltp-securebits-tests': '20180515'}
>         {'ltp-syscalls-tests': '20180515'}
>         {'ltp-timers-tests': '20180515'}
>
> Since SQUAD does not understand test sets, in order to tell squad what
> version of each test 'suite' is running, we have to submit it for each
> test 'set'. So we concatenate the suite name with the set name during
> submission.
>
> So now we know what version each test set is running, but it creates a
> bit of a mess which is hard to unroll in reporting :(

Exactly.  Reporting is the key here.

If we want to get to a point of concise reporting, having a hierarchy is
crucial.

> All this discussion leads me to believe that the prudent course for the
> time being seems to be to 'do nothing', and revisit this at a later date
> once things are more clear.

I disagree.  We're talking about the levels of hierarchy precisely
because we're trying to put in place some better reporting.

IMO, we need to get the hierarchy in the data model sorted out before
we're going to have reporting and visualization that can scale to even
medium sized test suites.

Kevin

Re: [kernelci] Meeting Minutes for 2018-08-20

[Milosz Wasilewski]

On 23 August 2018 at 04:40, Kevin Hilman <khilman@baylibre.com> wrote:
> "Tomeu Vizoso" <tomeu.vizoso@collabora.com> writes:
>
>> Hi Kevin,
>>
>> On 08/21/2018 01:52 AM, Kevin Hilman wrote:
>>> "Guillaume Tucker" <guillaume.tucker@gmail.com> writes:
>>>
>>>> On Mon, Aug 20, 2018 at 7:45 PM Kevin Hilman <khilman@baylibre.com> wrote:
>>>>>
>>>>> Dan, thanks again for the minutes!
>>>>>
>>>>> "Dan Rue" <dan.rue@linaro.org> writes:
>>>>>
>>>>>> - Attendees: Ana, Guillaume, Matt, Milosz, Rafael, Dan, Mark
>>>>>>
>>>>>> - Guillaume
>>>>>>    - Dealing with test results:
>>>>>>      - We currently have test suites/sets/cases, but test sets aren’t
>>>>>>        used and their implementation is broken.
>>>>>
>>>>> What is broken in the implementation?  LAVA? backend? frontend?
>>>>
>>>> The way we currently deal with test sets in the backend does not
>>>> add any value with the short test plans we currently have.  As I
>>>> explained in my previous email thread, both the test suite and
>>>> test set entries have a list of test cases, making test sets
>>>> effectively redundant.
>>>
>>> For the test plans we have, I see how that is redundant, but for test
>>> plans that actually include test sets, is it still redunant?
>>>
>>> My understanding was that test suites contain test sets (at least one
>>> "default" if no others are defined), and that test sets contain test
>>> cases.
>>
>> Can you explain what do you think KernelCI should do with the
>> hierarchy information? That would allow me to understand the need for
>> storing tests within a multiple level hierarchy.
>
> For me, it's just about reporting an visualization (something we are not
> currently doing great at).  IMO, even a test suite with > 20 test cases
> really needs to be broken up a bit for ease of making reports that are
> easy to understand.

I'm not sure you realize what you're asking for. Let's take the
example of LTP syscalls. It contains ~1000 test cases. If you break
them down to contain fewer than 20 you'll get 50+ groups (sets). This
doesn't solve any problem, just moves it one level up. Test suites
will contain a lot of tests. This is a fact of life that just needs to
be accepted.

milosz

Re: [kernelci] Meeting Minutes for 2018-08-20

[Guillaume Tucker]

On Thu, Aug 23, 2018 at 4:50 AM Kevin Hilman <khilman@baylibre.com> wrote:
>
> "Dan Rue" <dan.rue@linaro.org> writes:
>
> > On Wed, Aug 22, 2018 at 12:11:06PM +0200, Tomeu Vizoso wrote:
> >> Hi Kevin,
> >>
> >> On 08/21/2018 01:52 AM, Kevin Hilman wrote:
> >> > "Guillaume Tucker" <guillaume.tucker@gmail.com> writes:
> >> >
> >> > > On Mon, Aug 20, 2018 at 7:45 PM Kevin Hilman <khilman@baylibre.com> wrote:
> >> > > >
> >> > > > Dan, thanks again for the minutes!
> >> > > >
> >> > > > "Dan Rue" <dan.rue@linaro.org> writes:
> >> > > >
> >> > > > > - Attendees: Ana, Guillaume, Matt, Milosz, Rafael, Dan, Mark
> >> > > > >
> >> > > > > - Guillaume
> >> > > > >    - Dealing with test results:
> >> > > > >      - We currently have test suites/sets/cases, but test sets aren’t
> >> > > > >        used and their implementation is broken.
> >> > > >
> >> > > > What is broken in the implementation?  LAVA? backend? frontend?
> >> > >
> >> > > The way we currently deal with test sets in the backend does not
> >> > > add any value with the short test plans we currently have.  As I
> >> > > explained in my previous email thread, both the test suite and
> >> > > test set entries have a list of test cases, making test sets
> >> > > effectively redundant.
> >> >
> >> > For the test plans we have, I see how that is redundant, but for test
> >> > plans that actually include test sets, is it still redunant?
> >> >
> >> > My understanding was that test suites contain test sets (at least one
> >> > "default" if no others are defined), and that test sets contain test
> >> > cases.
> >>
> >> Can you explain what do you think KernelCI should do with the hierarchy
> >> information? That would allow me to understand the need for storing tests
> >> within a multiple level hierarchy.
> >>
> >> For the use cases I currently care about, it's enough to have test plans
> >> (corresponding to a kernel subsystem, so we know where to send regression
> >> reports) containing tests.
> >>
> >> Note that for big test suites, it's normal to have 6 levels or more, so I'm
> >> a bit concerned that if we decide to go now with 4 levels, we may need we
> >> need more in the future. But right now I'm having trouble figuring out what
> >> kernelci would do with the hierarchy information.
> >
> > I can give an example that we have run into in LKFT that would benefit
> > from my understanding of what test sets are. Since SQUAD doesn't have
> > test sets, I guess this is like a preview of what it might look like
> > without them.
> >
> > Browse to
> > https://qa-reports.linaro.org/lkft/linux-mainline-oe/build/v4.18-11219-gad1d69735878/
> >
> > This is the test results of a recent mainline revision. We run LTP, and
> > have it broken up into 20 separate test sets, which each correspond to a
> > runfile in LTP, and each set is run in a separate lava job.
> >
> > We want SQUAD to also show this distinction, since there are a lot of
> > tests. Now, click on the "+ Display more" link under the "Metadata" box.
> > You'll notice an explosion of metadata which is largely duplicate - the
> > version of LTP for each test set which culminates into this beauty:
> >
> >     suite_versions:
> >         {'kselftest': '4.18+gitAUTOINC+ad1d697358'}
> >         {'libhugetlbfs': '2.20+gitAUTOINC+02df38e93e+next'}
> >         {'ltp-cap_bounds-tests': '20180515'}
> >         {'ltp-containers-tests': '20180515'}
> >         {'ltp-cve-tests': '20180515'}
> >         {'ltp-fcntl-locktests-tests': '20180515'}
> >         {'ltp-filecaps-tests': '20180515'}
> >         {'ltp-fs-tests': '20180515'}
> >         {'ltp-fs_bind-tests': '20180515'}
> >         {'ltp-fs_perms_simple-tests': '20180515'}
> >         {'ltp-fsx-tests': '20180515'}
> >         {'ltp-hugetlb-tests': '20180515'}
> >         {'ltp-io-tests': '20180515'}
> >         {'ltp-ipc-tests': '20180515'}
> >         {'ltp-math-tests': '20180515'}
> >         {'ltp-nptl-tests': '20180515'}
> >         {'ltp-open-posix-tests': '20180515'}
> >         {'ltp-pty-tests': '20180515'}
> >         {'ltp-sched-tests': '20180515'}
> >         {'ltp-securebits-tests': '20180515'}
> >         {'ltp-syscalls-tests': '20180515'}
> >         {'ltp-timers-tests': '20180515'}
> >
> > Since SQUAD does not understand test sets, in order to tell squad what
> > version of each test 'suite' is running, we have to submit it for each
> > test 'set'. So we concatenate the suite name with the set name during
> > submission.
> >
> > So now we know what version each test set is running, but it creates a
> > bit of a mess which is hard to unroll in reporting :(
>
> Exactly.  Reporting is the key here.
>
> If we want to get to a point of concise reporting, having a hierarchy is
> crucial.

Essentially, when running test suites manually, you can typically
choose to only see the results you care about one way or another.
Either by choosing to run only some cases, or by filtering the
output.  We need to be able to reproduce this use-case via
KernelCI and not make it stand in the way between test suites and
results visible to the user.

So I belive the first step is to capture the output of the test
suites in a generic way that contains all the original
information.  It should then be possible to view the results in
any format, even maybe turning them back into a plain text log to
let users filter it themselves with grep...  I'm not saying
that's an amazing feature and user experience or even something
we should actually do, but if the system makes it impossible to
implement that then I think we have a problem.

> > All this discussion leads me to believe that the prudent course for the
> > time being seems to be to 'do nothing', and revisit this at a later date
> > once things are more clear.
>
> I disagree.  We're talking about the levels of hierarchy precisely
> because we're trying to put in place some better reporting.
>
> IMO, we need to get the hierarchy in the data model sorted out before
> we're going to have reporting and visualization that can scale to even
> medium sized test suites.

Following the principle that we need to capture all the data, we
can't just keep the implementation we have right now or ignore
anything beyond a single level of hierarchy.  I agree with Dan
that we still have a few things to clarify, so we should probably
do only the minimal required to store consistent data.  I don't
think we know exactly what we want to do with the data (how to
search, which flavours of email reports...), and if we want to
split tests between jobs we'll need a robust way of piecing them
back in the backend.

I like Kevin's suggestion of having sets in sets, in fact that's
something we were discussing yesterday with Ana.  Here's a
pseudo-schema I think would work well:

TestCase:
  * name
  * result
  * meta-data

TestGroup:
  * name
  * meta-data
  * list of test cases
  * list of test groups

Think of it a bit like a file system, with groups being
directories and cases files.  You can have groups within
groups...  This can scale both ways, to have sets within suites,
and also suites within a collection of suites or whatever we want
to do with this.  Then we can have a way of calling them with
some kind of path like suite/set/case, or suite.set.case, or just
suite.case, or super-plan.suite.case or whatever it takes to
store the test suite results.

I believe the changes should be relatively simple, and should
result in a simpler database schema than what we have now with
suites & sets mixed up.

Guillaume

Re: [kernelci] Meeting Minutes for 2018-08-20

[Kevin Hilman]

Guillaume Tucker <guillaume.tucker@gmail.com> writes:

[...]

> I like Kevin's suggestion of having sets in sets, in fact that's
> something we were discussing yesterday with Ana.  Here's a
> pseudo-schema I think would work well:
>
> TestCase:
>   * name
>   * result
>   * meta-data
>
> TestGroup:
>   * name
>   * meta-data
>   * list of test cases
>   * list of test groups
>
> Think of it a bit like a file system, with groups being
> directories and cases files.  You can have groups within
> groups...  This can scale both ways, to have sets within suites,
> and also suites within a collection of suites or whatever we want
> to do with this.  Then we can have a way of calling them with
> some kind of path like suite/set/case, or suite.set.case, or just
> suite.case, or super-plan.suite.case or whatever it takes to
> store the test suite results.
>
> I believe the changes should be relatively simple, and should
> result in a simpler database schema than what we have now with
> suites & sets mixed up.

I like it!  Can support relatively flat (like we have today) or
arbitrary scaling.  

Kevin