RFC: dashboards, visualization and analytics for test results

RFC: dashboards, visualization and analytics for test results

[Kevin Hilman]

Hello folks interested in kernel testing/automation,

The KernelCI project is starting to look at what's next for dashboards,
visualization and analytics for the various Linux focused testing
projects.

At Linux Plumbers, we launched some discussions[1] around common ways to
collect test results, logs and metadata into a public, shared dataset,
and we've already started collecting data from several different
sources.

So the next question is... how do we best use all of this data?

We're beginning to brainstorm how to visualize, analyze and learn from
this data in useful ways.

To that end, we're starting to collect a set of user stories to help us
brainstorm a new design for web based dashboard and analytics, and we'd
like to hear from you.

Below is the start of a list of user stories[2], but we want to grow
this list with your ideas, so please share them on this thread. 

We're also very interested in talking with any big data people and data
scientists who might be willing to look at this growing set of data and
help us better plan for the future full of lots of test data.

We appreciate your ideas and feedback,

Kevin (on behalf of the KernelCI team)


[1] c.f. Unifying Test Reporting with KernelCI from the testing/fuzzing
    micro-conference:
    https://linuxplumbersconf.org/event/7/sessions/80/#20200826



[2] Example user stories

A kernel developer has sent a patch which caused a regression
   - Find the details, how to reproduce, check when it’s fixed

A maintainer is getting a branch ready for the next merge window
  - Compare results against mainline, ensure all tests were run correctly

An OEM or SoC vendor needs to upgrade their kernel or move to upstream
  - See all results for a particular platform on various stable releases

Regular visitors who want to know how the kernel is doing
  - Highlight new regressions, show trends for pass/fail and performance
  - Does kernel X work on my hardware? 

Distro vendor wants to know
  - Is the latest kernel stable enough for my distro?
  - Why does this test fail on my distro kernel but not stable kernel?

Stable kernel maintainer wants to know
  - Is this kernel stable enough to release?

Kernel / subsystem maintainer wants to know
  - Is my code/subsystem being tested?  How much coverage?
  - Is the kernel size growing?  Why?  Which subsystems
  - Is boot time / test execution time increasing/decreasing?  What are the historical trends?

[cip-dev] FW: [Automated-testing] RFC: dashboards, visualization and analytics for test results

[Chris Paterson]

[-- Attachment #1: Type: text/plain, Size: 3149 bytes --]

FYI

Please feel free to add any use cases or comments to the mail thread below from KernelCI.

Kind regards, Chris

> From: automated-testing@lists.yoctoproject.org <automated-
> testing@lists.yoctoproject.org> On Behalf Of Kevin Hilman via
> lists.yoctoproject.org
> Sent: 07 October 2020 18:48
> To: automated-testing@lists.yoctoproject.org; kernelci@groups.io
> Cc: tools@linux.kernel.org
> Subject: [Automated-testing] RFC: dashboards, visualization and analytics for
> test results
> 
> Hello folks interested in kernel testing/automation,
> 
> The KernelCI project is starting to look at what's next for dashboards,
> visualization and analytics for the various Linux focused testing
> projects.
> 
> At Linux Plumbers, we launched some discussions[1] around common ways
> to
> collect test results, logs and metadata into a public, shared dataset,
> and we've already started collecting data from several different
> sources.
> 
> So the next question is... how do we best use all of this data?
> 
> We're beginning to brainstorm how to visualize, analyze and learn from
> this data in useful ways.
> 
> To that end, we're starting to collect a set of user stories to help us
> brainstorm a new design for web based dashboard and analytics, and we'd
> like to hear from you.
> 
> Below is the start of a list of user stories[2], but we want to grow
> this list with your ideas, so please share them on this thread.
> 
> We're also very interested in talking with any big data people and data
> scientists who might be willing to look at this growing set of data and
> help us better plan for the future full of lots of test data.
> 
> We appreciate your ideas and feedback,
> 
> Kevin (on behalf of the KernelCI team)
> 
> 
> [1] c.f. Unifying Test Reporting with KernelCI from the testing/fuzzing
>     micro-conference:
>     https://linuxplumbersconf.org/event/7/sessions/80/#20200826
> 
> 
> 
> [2] Example user stories
> 
> A kernel developer has sent a patch which caused a regression
>    - Find the details, how to reproduce, check when it’s fixed
> 
> A maintainer is getting a branch ready for the next merge window
>   - Compare results against mainline, ensure all tests were run correctly
> 
> An OEM or SoC vendor needs to upgrade their kernel or move to upstream
>   - See all results for a particular platform on various stable releases
> 
> Regular visitors who want to know how the kernel is doing
>   - Highlight new regressions, show trends for pass/fail and performance
>   - Does kernel X work on my hardware?
> 
> Distro vendor wants to know
>   - Is the latest kernel stable enough for my distro?
>   - Why does this test fail on my distro kernel but not stable kernel?
> 
> Stable kernel maintainer wants to know
>   - Is this kernel stable enough to release?
> 
> Kernel / subsystem maintainer wants to know
>   - Is my code/subsystem being tested?  How much coverage?
>   - Is the kernel size growing?  Why?  Which subsystems
>   - Is boot time / test execution time increasing/decreasing?  What are the
> historical trends?
> 


[-- Attachment #2: Type: text/plain, Size: 420 bytes --]


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#5571): https://lists.cip-project.org/g/cip-dev/message/5571
Mute This Topic: https://lists.cip-project.org/mt/77483847/4520388
Group Owner: cip-dev+owner@lists.cip-project.org
Unsubscribe: https://lists.cip-project.org/g/cip-dev/leave/8129055/727948398/xyzzy [cip-dev@archiver.kernel.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: RFC: dashboards, visualization and analytics for test results

[Nick Desaulniers]

On Wed, Oct 7, 2020 at 10:47 AM Kevin Hilman <khilman@baylibre.com> wrote:
>
> Hello folks interested in kernel testing/automation,
>
> The KernelCI project is starting to look at what's next for dashboards,
> visualization and analytics for the various Linux focused testing
> projects.
>
> At Linux Plumbers, we launched some discussions[1] around common ways to
> collect test results, logs and metadata into a public, shared dataset,
> and we've already started collecting data from several different
> sources.
>
> So the next question is... how do we best use all of this data?
>
> We're beginning to brainstorm how to visualize, analyze and learn from
> this data in useful ways.
>
> To that end, we're starting to collect a set of user stories to help us
> brainstorm a new design for web based dashboard and analytics, and we'd
> like to hear from you.
>
> Below is the start of a list of user stories[2], but we want to grow
> this list with your ideas, so please share them on this thread.
>
> We're also very interested in talking with any big data people and data
> scientists who might be willing to look at this growing set of data and
> help us better plan for the future full of lots of test data.
>
> We appreciate your ideas and feedback,
>
> Kevin (on behalf of the KernelCI team)
>
>
> [1] c.f. Unifying Test Reporting with KernelCI from the testing/fuzzing
>     micro-conference:
>     https://linuxplumbersconf.org/event/7/sessions/80/#20200826
>
>
>
> [2] Example user stories

additional user story:

A developer wants to slice and dice warnings by toolchain and
toolchain version to see what errors or warnings are compiler
specific/need to be fixed.

>
> A kernel developer has sent a patch which caused a regression
>    - Find the details, how to reproduce, check when it’s fixed
>
> A maintainer is getting a branch ready for the next merge window
>   - Compare results against mainline, ensure all tests were run correctly
>
> An OEM or SoC vendor needs to upgrade their kernel or move to upstream
>   - See all results for a particular platform on various stable releases
>
> Regular visitors who want to know how the kernel is doing
>   - Highlight new regressions, show trends for pass/fail and performance
>   - Does kernel X work on my hardware?
>
> Distro vendor wants to know
>   - Is the latest kernel stable enough for my distro?
>   - Why does this test fail on my distro kernel but not stable kernel?
>
> Stable kernel maintainer wants to know
>   - Is this kernel stable enough to release?
>
> Kernel / subsystem maintainer wants to know
>   - Is my code/subsystem being tested?  How much coverage?
>   - Is the kernel size growing?  Why?  Which subsystems
>   - Is boot time / test execution time increasing/decreasing?  What are the historical trends?
>
>
>
>
> 
>
>


-- 
Thanks,
~Nick Desaulniers

Re: RFC: dashboards, visualization and analytics for test results

[Kevin Hilman]

Nick Desaulniers <ndesaulniers@google.com> writes:

> On Wed, Oct 7, 2020 at 10:47 AM Kevin Hilman <khilman@baylibre.com> wrote:
>>
>> Hello folks interested in kernel testing/automation,
>>
>> The KernelCI project is starting to look at what's next for dashboards,
>> visualization and analytics for the various Linux focused testing
>> projects.
>>
>> At Linux Plumbers, we launched some discussions[1] around common ways to
>> collect test results, logs and metadata into a public, shared dataset,
>> and we've already started collecting data from several different
>> sources.
>>
>> So the next question is... how do we best use all of this data?
>>
>> We're beginning to brainstorm how to visualize, analyze and learn from
>> this data in useful ways.
>>
>> To that end, we're starting to collect a set of user stories to help us
>> brainstorm a new design for web based dashboard and analytics, and we'd
>> like to hear from you.
>>
>> Below is the start of a list of user stories[2], but we want to grow
>> this list with your ideas, so please share them on this thread.
>>
>> We're also very interested in talking with any big data people and data
>> scientists who might be willing to look at this growing set of data and
>> help us better plan for the future full of lots of test data.
>>
>> We appreciate your ideas and feedback,
>>
>> Kevin (on behalf of the KernelCI team)
>>
>>
>> [1] c.f. Unifying Test Reporting with KernelCI from the testing/fuzzing
>>     micro-conference:
>>     https://linuxplumbersconf.org/event/7/sessions/80/#20200826
>>
>>
>>
>> [2] Example user stories
>
> additional user story:
>
> A developer wants to slice and dice warnings by toolchain and
> toolchain version to see what errors or warnings are compiler
> specific/need to be fixed.

Excellent, thanks!

Kevin

Re: RFC: dashboards, visualization and analytics for test results

[LABBE Corentin]

On Wed, Oct 07, 2020 at 10:47:32AM -0700, Kevin Hilman wrote:
> Hello folks interested in kernel testing/automation,
> 
> The KernelCI project is starting to look at what's next for dashboards,
> visualization and analytics for the various Linux focused testing
> projects.
> 
> At Linux Plumbers, we launched some discussions[1] around common ways to
> collect test results, logs and metadata into a public, shared dataset,
> and we've already started collecting data from several different
> sources.
> 
> So the next question is... how do we best use all of this data?
> 
> We're beginning to brainstorm how to visualize, analyze and learn from
> this data in useful ways.
> 
> To that end, we're starting to collect a set of user stories to help us
> brainstorm a new design for web based dashboard and analytics, and we'd
> like to hear from you.
> 
> Below is the start of a list of user stories[2], but we want to grow
> this list with your ideas, so please share them on this thread. 
> 
> We're also very interested in talking with any big data people and data
> scientists who might be willing to look at this growing set of data and
> help us better plan for the future full of lots of test data.
> 
> We appreciate your ideas and feedback,
> 
> Kevin (on behalf of the KernelCI team)
> 
> 
> [1] c.f. Unifying Test Reporting with KernelCI from the testing/fuzzing
>     micro-conference:
>     https://linuxplumbersconf.org/event/7/sessions/80/#20200826
> 
> 
> 
> [2] Example user stories
> 
> A kernel developer has sent a patch which caused a regression
>    - Find the details, how to reproduce, check when it’s fixed
> 
> A maintainer is getting a branch ready for the next merge window
>   - Compare results against mainline, ensure all tests were run correctly
> 
> An OEM or SoC vendor needs to upgrade their kernel or move to upstream
>   - See all results for a particular platform on various stable releases
> 
> Regular visitors who want to know how the kernel is doing
>   - Highlight new regressions, show trends for pass/fail and performance
>   - Does kernel X work on my hardware? 
> 
> Distro vendor wants to know
>   - Is the latest kernel stable enough for my distro?
>   - Why does this test fail on my distro kernel but not stable kernel?
> 
> Stable kernel maintainer wants to know
>   - Is this kernel stable enough to release?
> 
> Kernel / subsystem maintainer wants to know
>   - Is my code/subsystem being tested?  How much coverage?
>   - Is the kernel size growing?  Why?  Which subsystems
>   - Is boot time / test execution time increasing/decreasing?  What are the historical trends?
> 

Hello

As a maintainer, viewing all boot including a specific hardware/driver.
Does my xxxx driver works everywhere ?

Recent example:
- I have added support for PRNG/hash on sun8i-ce, I want to check all boots with this driver.
More complex recent example:
- since commit bbc4d71d6354 ("net: phy: realtek: fix rtl8211e rx/tx delay config") many board could loose network, I want to see all boots including this patch with this PHY.

Regards

Re: RFC: dashboards, visualization and analytics for test results

[Corentin Labbe]

On Wed, Oct 07, 2020 at 10:47:32AM -0700, Kevin Hilman wrote:
> Hello folks interested in kernel testing/automation,
> 
> The KernelCI project is starting to look at what's next for dashboards,
> visualization and analytics for the various Linux focused testing
> projects.
> 
> At Linux Plumbers, we launched some discussions[1] around common ways to
> collect test results, logs and metadata into a public, shared dataset,
> and we've already started collecting data from several different
> sources.
> 
> So the next question is... how do we best use all of this data?
> 
> We're beginning to brainstorm how to visualize, analyze and learn from
> this data in useful ways.
> 
> To that end, we're starting to collect a set of user stories to help us
> brainstorm a new design for web based dashboard and analytics, and we'd
> like to hear from you.
> 
> Below is the start of a list of user stories[2], but we want to grow
> this list with your ideas, so please share them on this thread. 
> 
> We're also very interested in talking with any big data people and data
> scientists who might be willing to look at this growing set of data and
> help us better plan for the future full of lots of test data.
> 
> We appreciate your ideas and feedback,
> 
> Kevin (on behalf of the KernelCI team)
> 
> 
> [1] c.f. Unifying Test Reporting with KernelCI from the testing/fuzzing
>     micro-conference:
>     https://linuxplumbersconf.org/event/7/sessions/80/#20200826
> 
> 
> 
> [2] Example user stories
> 
> A kernel developer has sent a patch which caused a regression
>    - Find the details, how to reproduce, check when it’s fixed
> 
> A maintainer is getting a branch ready for the next merge window
>   - Compare results against mainline, ensure all tests were run correctly
> 
> An OEM or SoC vendor needs to upgrade their kernel or move to upstream
>   - See all results for a particular platform on various stable releases
> 
> Regular visitors who want to know how the kernel is doing
>   - Highlight new regressions, show trends for pass/fail and performance
>   - Does kernel X work on my hardware? 
> 
> Distro vendor wants to know
>   - Is the latest kernel stable enough for my distro?
>   - Why does this test fail on my distro kernel but not stable kernel?
> 
> Stable kernel maintainer wants to know
>   - Is this kernel stable enough to release?
> 
> Kernel / subsystem maintainer wants to know
>   - Is my code/subsystem being tested?  How much coverage?
>   - Is the kernel size growing?  Why?  Which subsystems
>   - Is boot time / test execution time increasing/decreasing?  What are the historical trends?
> 

Hello

As a maintainer, viewing all boot including a specific hardware/driver.
Does my xxxx driver works everywhere ?

Recent example:
- I have added support for PRNG/hash on sun8i-ce, I want to check all boots with this driver.
More complex recent example:
- since commit bbc4d71d6354 ("net: phy: realtek: fix rtl8211e rx/tx delay config") many board could loose network, I want to see all boots including this patch with this PHY.

Regards

Re: RFC: dashboards, visualization and analytics for test results

[Chris Fiege]

Hey Kevin,

On Wed, Oct  7, 2020 at 10:47 AM, Kevin Hilman wrote:

>
> The KernelCI project is starting to look at what's next for dashboards,
> visualization and analytics for the various Linux focused testing
> projects.
> 
> (...)
> 
> So the next question is... how do we best use all of this data?
> 
> To that end, we're starting to collect a set of user stories to help us
> brainstorm a new design for web based dashboard and analytics, and we'd
> like to hear from you.

for us at Pengutronix Kernel-CI has always been a good way to present our
Kernel maintenance / development activities to our customers.

We are currently missing a few of the old views. I'll try to describe our
use-cases in some short user stories. Feel free to ask for more details.

A customer tries to evaluate whether to use  mainline Linux or a vendor kernel:
  - View of list of all hardware / SoC currently tested in all labs
  - View job history on all devices with a specific hardware / SoC
  - View statistics of failing / passing Tests over time for a specific hardware / SoC.

A member of sales department uses the contribution to Kernel-CI as an indication
for our effort to maintain systems for a long time.
  - View a list of hardware / SoC in a specific lab.
  - Send a link to said list to a costumer.
  - View the status of hardware in a specific lab:
    Are jobs scheduled for a specific hardware?
  - View the history of jobs for a specific lab.


Regards,
Chris

Re: [Automated-testing] RFC: dashboards, visualization and analytics for test results

[Nikolai Kondrashov]

Hello everyone,

On 10/7/20 8:47 PM, Kevin Hilman wrote:
 > We're beginning to brainstorm how to visualize, analyze and learn from
 > this data in useful ways.
 >
 > To that end, we're starting to collect a set of user stories to help us
 > brainstorm a new design for web based dashboard and analytics, and we'd
 > like to hear from you.
 >
 > Below is the start of a list of user stories[2], but we want to grow
 > this list with your ideas, so please share them on this thread.

I'd like to hijack this thread with a somewhat different, but relevant topic,
if I may: how to send notifications on test results arriving to KCIDB from
various CI systems.

I've thrown together a little document listing some of the user stories for
such a system, as well as a KCIDB-specific algorithm overview:

     https://docs.google.com/document/d/1rYaxfeb5-NND_ef4TkEv0ka7bGhEWc7mCPhRRNihVK8/

If you have any user stories of your own, or any ideas regarding result
notifications whatsoever, please don't hesitate to drop by and leave a
comment!

This will help us make sure we build something closer to what everyone needs.

Thank you.
Nick

On 10/7/20 8:47 PM, Kevin Hilman wrote:
 > Hello folks interested in kernel testing/automation,
 >
 > The KernelCI project is starting to look at what's next for dashboards,
 > visualization and analytics for the various Linux focused testing
 > projects.
 >
 > At Linux Plumbers, we launched some discussions[1] around common ways to
 > collect test results, logs and metadata into a public, shared dataset,
 > and we've already started collecting data from several different
 > sources.
 >
 > So the next question is... how do we best use all of this data?
 >
 > We're beginning to brainstorm how to visualize, analyze and learn from
 > this data in useful ways.
 >
 > To that end, we're starting to collect a set of user stories to help us
 > brainstorm a new design for web based dashboard and analytics, and we'd
 > like to hear from you.
 >
 > Below is the start of a list of user stories[2], but we want to grow
 > this list with your ideas, so please share them on this thread.
 >
 > We're also very interested in talking with any big data people and data
 > scientists who might be willing to look at this growing set of data and
 > help us better plan for the future full of lots of test data.
 >
 > We appreciate your ideas and feedback,
 >
 > Kevin (on behalf of the KernelCI team)
 >
 >
 > [1] c.f. Unifying Test Reporting with KernelCI from the testing/fuzzing
 >      micro-conference:
 >      https://linuxplumbersconf.org/event/7/sessions/80/#20200826
 >
 >
 >
 > [2] Example user stories
 >
 > A kernel developer has sent a patch which caused a regression
 >     - Find the details, how to reproduce, check when it’s fixed
 >
 > A maintainer is getting a branch ready for the next merge window
 >    - Compare results against mainline, ensure all tests were run correctly
 >
 > An OEM or SoC vendor needs to upgrade their kernel or move to upstream
 >    - See all results for a particular platform on various stable releases
 >
 > Regular visitors who want to know how the kernel is doing
 >    - Highlight new regressions, show trends for pass/fail and performance
 >    - Does kernel X work on my hardware?
 >
 > Distro vendor wants to know
 >    - Is the latest kernel stable enough for my distro?
 >    - Why does this test fail on my distro kernel but not stable kernel?
 >
 > Stable kernel maintainer wants to know
 >    - Is this kernel stable enough to release?
 >
 > Kernel / subsystem maintainer wants to know
 >    - Is my code/subsystem being tested?  How much coverage?
 >    - Is the kernel size growing?  Why?  Which subsystems
 >    - Is boot time / test execution time increasing/decreasing?  What are the historical trends?
 >
 >
 >
 >
 > 
 >

Re: [Automated-testing] RFC: dashboards, visualization and analytics for test results

[Chris Paterson]

Hello Kevin,

> From: automated-testing@lists.yoctoproject.org <automated-
> testing@lists.yoctoproject.org> On Behalf Of Kevin Hilman via
> lists.yoctoproject.org
> Sent: 07 October 2020 18:48

[...]

> 
> Below is the start of a list of user stories[2], but we want to grow
> this list with your ideas, so please share them on this thread.
> 

[...]

> 
> [2] Example user stories
> 
> A kernel developer has sent a patch which caused a regression
>    - Find the details, how to reproduce, check when it’s fixed
> 
> A maintainer is getting a branch ready for the next merge window
>   - Compare results against mainline, ensure all tests were run correctly

As part of this, it may be good to hide the failures reported that are also reported on mainline, so the maintainer knows the regressions directly caused by their tree.
Perhaps this change could also be made for the email reports.

Suggested-by: Geert Uytterhoeven <geert+renesas@glider.be>

Kind regards, Chris

> 
> An OEM or SoC vendor needs to upgrade their kernel or move to upstream
>   - See all results for a particular platform on various stable releases
> 
> Regular visitors who want to know how the kernel is doing
>   - Highlight new regressions, show trends for pass/fail and performance
>   - Does kernel X work on my hardware?
> 
> Distro vendor wants to know
>   - Is the latest kernel stable enough for my distro?
>   - Why does this test fail on my distro kernel but not stable kernel?
> 
> Stable kernel maintainer wants to know
>   - Is this kernel stable enough to release?
> 
> Kernel / subsystem maintainer wants to know
>   - Is my code/subsystem being tested?  How much coverage?
>   - Is the kernel size growing?  Why?  Which subsystems
>   - Is boot time / test execution time increasing/decreasing?  What are the
> historical trends?
> 

Re: [Automated-testing] RFC: dashboards, visualization and analytics for test results

[Chris Paterson]

Hello Kevin,

> From: automated-testing@lists.yoctoproject.org <automated-
> testing@lists.yoctoproject.org> On Behalf Of Kevin Hilman via
> lists.yoctoproject.org
> Sent: 07 October 2020 18:48

[...]

> 
> Below is the start of a list of user stories[2], but we want to grow
> this list with your ideas, so please share them on this thread.
> 

[...]

> 
> [2] Example user stories
> 
> A kernel developer has sent a patch which caused a regression
>    - Find the details, how to reproduce, check when it’s fixed
> 
> A maintainer is getting a branch ready for the next merge window
>   - Compare results against mainline, ensure all tests were run correctly

As part of this, it may be good to hide the failures reported that are also reported on mainline, so the maintainer knows the regressions directly caused by their tree.
Perhaps this change could also be made for the email reports.

Suggested-by: Geert Uytterhoeven <geert+renesas@glider.be>

Kind regards, Chris

> 
> An OEM or SoC vendor needs to upgrade their kernel or move to upstream
>   - See all results for a particular platform on various stable releases
> 
> Regular visitors who want to know how the kernel is doing
>   - Highlight new regressions, show trends for pass/fail and performance
>   - Does kernel X work on my hardware?
> 
> Distro vendor wants to know
>   - Is the latest kernel stable enough for my distro?
>   - Why does this test fail on my distro kernel but not stable kernel?
> 
> Stable kernel maintainer wants to know
>   - Is this kernel stable enough to release?
> 
> Kernel / subsystem maintainer wants to know
>   - Is my code/subsystem being tested?  How much coverage?
>   - Is the kernel size growing?  Why?  Which subsystems
>   - Is boot time / test execution time increasing/decreasing?  What are the
> historical trends?
>