Re: #KCIDB engagement report

Re: #KCIDB engagement report

[Guillaume Tucker]

+kernelci@lists.linux.dev (new KernelCI mailing list)

On 17/11/2022 11:28, Nikolai Kondrashov wrote:
> Hi everyone,
> 
> Below is the monthly* report on KCIDB** engagement. It lists various CI
> systems and their status of engagement with KCIDB, as well as developer
> engagement.
> 
> Lines with updates are marked with "!".
> 
> Main news are: Intel's 0day CI has been sending their data to production for a
> while, and are looking to expand their contribution, Microsoft moving closer
> to sending their data, and the proposal for known-issue schema is going out
> soon, after preliminary implementation and testing is complete.
> 
> CI Engagement Status
> 
>     KernelCI native
>         Sending (a lot of) production build and test results.
>         https://kcidb.kernelci.org/?var-origin=kernelci
> 
>     Red Hat CKI
>         Sending production results.
>         https://kcidb.kernelci.org/?var-origin=redhat
> 
>     Google Syzbot
>         Sending a subset of production results (failures only).
>         https://kcidb.kernelci.org/?var-origin=syzbot
> 
>     ARM
>         Sending production results.
>         Full commit hashes are currently not available, are spoofed, and don't
>         match the ones reported by others. There's ongoing switch to using a
>         KernelCI instance setup, which should fix this.
>         https://kcidb.kernelci.org/?var-origin=arm
> 
>     Sony Fuego
>         Internal design in progress.
> 
>     Gentoo GKernelCI
>         Sending production results.
>         Only builds (a few architectures), no configs, no logs, and no tests
>         for now, but working on growing contributions.
>         https://kcidb.kernelci.org/?var-origin=gkernelci
> 
>     Intel 0day
> !       Sending production results.
> !       Mainline builds and kselftest results.
> !       https://kcidb.kernelci.org/?var-origin=0dayci
> 
>     Linaro
>         Sending (a lot of) Tuxsuite build results to "production" KCIDB.
>         https://kcidb.kernelci.org/?var-origin=tuxsuite
> 
>     TuxML
>         Initial contact in response to a report.
>         There's a plan to approach us and start work in the coming months.
> 
>     Yocto Project
>         Initial contact in response to a report.
>         Would like to start sending build and test results, particularly for
>         older kernels. Would like to separate upstream commits from project
>         patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196
> 
>     Huawei Compass CI
>         Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
>         No response so far.
> 
>     Microsoft
>         Johnson George was filled in on the details of submission process.
> 
> Please respond with corrections or suggestions of other CI systems to contact.
> 
> Thank you!
> Nick
> 
> *More-or-less :)
> 
> **KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
>   Foundation's KernelCI project:
>   https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Groups.io Links: You receive all messages sent to this group.
> View/Reply Online (#1619): https://groups.io/g/kernelci/message/1619
> Mute This Topic: https://groups.io/mt/95087111/924702
> Mute #kcidb:https://groups.io/g/kernelci/mutehashtag/kcidb
> Group Owner: kernelci+owner@groups.io
> Unsubscribe: https://groups.io/g/kernelci/unsub [guillaume.tucker@collabora.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 
> 

#KCIDB engagement report

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly* report on KCIDB** engagement. It lists various CI
systems and their status of engagement with KCIDB, as well as developer
engagement.

Lines with updates are marked with "!".

Intel's Philip Li has reached out, asking for submission credentials and
instructions, and the 0day CI has been submitting their results to our
"playground" setup for a few days now:

https://kcidb.kernelci.org/?var-origin=0dayci&var-dataset=playground_kcidb_01

It's already looking pretty good!

I'm working on the design of known-issue handling, fleshing out smaller
details now, and looking for the fastest path to providing value.

CI Engagement Status

     KernelCI native
         Sending (a lot of) production build and test results.
         https://kcidb.kernelci.org/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://kcidb.kernelci.org/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://kcidb.kernelci.org/?var-origin=syzbot

     ARM
         Sending production results.
         Full commit hashes are currently not available, are spoofed, and don't
         match the ones reported by others. There's ongoing switch to using a
         KernelCI instance setup, which should fix this.
         https://kcidb.kernelci.org/?var-origin=arm

     Sony Fuego
         Internal design in progress.

     Gentoo GKernelCI
         Sending production results.
         Only builds (a few architectures), no configs, no logs, and no tests
         for now, but working on growing contributions.
         https://kcidb.kernelci.org/?var-origin=gkernelci

     Intel 0day
!       Sending test results to our playground setup.
! 
https://kcidb.kernelci.org/?var-origin=0dayci&var-dataset=playground_kcidb_01

     Linaro
         Sending (a lot of) Tuxsuite build results to "production" KCIDB.
         https://kcidb.kernelci.org/?var-origin=tuxsuite

     TuxML
         Initial contact in response to a report.
         There's a plan to approach us and start work in the coming months.

     Yocto Project
         Initial contact in response to a report.
         Would like to start sending build and test results, particularly for
         older kernels. Would like to separate upstream commits from project
         patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

     Huawei Compass CI
         Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
         No response so far.

     Microsoft
!       Johnson George is investigating KCIDB.

Please respond with corrections or suggestions of other CI systems to contact.

Thank you!
Nick

*More-or-less :)

**KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
   Foundation's KernelCI project:
   https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

#KCIDB engagement report

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly* report on KCIDB** engagement. It lists various CI
systems and their status of engagement with KCIDB, as well as developer
engagement.

Lines with updates are marked with "!".

Microsoft's Liz Zhang left the team and has handed her work to Johnson George.
He has started experimenting with KCIDB and looking at the database schema.

I implemented a dashboard summarizing data for a particular (sub)test on 
request from Daniel Stone. Here's an example:

https://kcidb.kernelci.org/d/test-node/test-node?orgId=1&var-test_path=igt-gpu-panfrost

The tests (and links to this dashboard) are now also visible from the Grafana 
home page.

I've successfully managed to remove BigQuery from our notification-generation 
workflow, switching to PostgreSQL instead, and now our Google Cloud costs are 
in check. The long-term data is still stored in BigQuery and is used for the 
dashboard (the latter might change eventually).

Now I'm mostly done with smaller pending issues and will switch back to 
working on handling known issues.

CI Engagement Status

     KernelCI native
         Sending (a lot of) production build and test results.
         https://kcidb.kernelci.org/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://kcidb.kernelci.org/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://kcidb.kernelci.org/?var-origin=syzbot

     ARM
         Sending production results.
         Full commit hashes are currently not available, are spoofed, and don't
         match the ones reported by others. There's ongoing switch to using a
         KernelCI instance setup, which should fix this.
         https://kcidb.kernelci.org/?var-origin=arm

     Sony Fuego
         Internal design in progress.

     Gentoo GKernelCI
         Sending production results.
         Only builds (a few architectures), no configs, no logs, and no tests
         for now, but working on growing contributions.
         https://kcidb.kernelci.org/?var-origin=gkernelci

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

     Linaro
         Sending (a lot of) Tuxsuite build results to "production" KCIDB.
         https://kcidb.kernelci.org/?var-origin=tuxsuite

     TuxML
         Initial contact in response to a report.
         There's a plan to approach us and start work in the coming months.

     Yocto Project
         Initial contact in response to a report.
         Would like to start sending build and test results, particularly for
         older kernels. Would like to separate upstream commits from project
         patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

     Huawei Compass CI
         Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
         No response so far.

     Microsoft
         I provided Liz Zhang with instructions and credentials for submitting
!       Penguinator data to KCIDB. Johnson George took over from Liz.

Please respond with corrections or suggestions of other CI systems to contact.

Thank you!
Nick

*More-or-less :)

**KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
   Foundation's KernelCI project:
   https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

#KCIDB engagement report

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly* report on KCIDB** engagement. It lists various CI
systems and their status of engagement with KCIDB, as well as developer
engagement.

Lines with updates are marked with "!".

The main news is Microsoft has started work on sending reports from their
in-house Penguinator system to KCIDB.

Daniel Stone has requested the ability to browse the test tree in KCIDB
dashboards, which I considered for a while as well. He'd like to be able
to browse graphics suite results across various builds and revisions.

However, I'll have to do that only after I'm done with the current task:
improving our use of BigQuery, so we don't have to pay so much for our
queries.

After both of these I'll return to working on known issue handling/masking.

CI Engagement Status

     KernelCI native
         Sending (a lot of) production build and test results.
         https://kcidb.kernelci.org/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://kcidb.kernelci.org/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://kcidb.kernelci.org/?var-origin=syzbot

     ARM
         Sending production results.
         Full commit hashes are currently not available, are spoofed, and don't
         match the ones reported by others. There's ongoing switch to using a
         KernelCI instance setup, which should fix this.
         https://kcidb.kernelci.org/?var-origin=arm

     Sony Fuego
         Internal design in progress.

     Gentoo GKernelCI
         Sending production results.
         Only builds (a few architectures), no configs, no logs, and no tests
         for now, but working on growing contributions.
         https://kcidb.kernelci.org/?var-origin=gkernelci

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

     Linaro
         Sending (a lot of) Tuxsuite build results to "production" KCIDB.
         https://kcidb.kernelci.org/?var-origin=tuxsuite

     TuxML
         Initial contact in response to a report.
         There's a plan to approach us and start work in the coming months.

     Yocto Project
         Initial contact in response to a report.
         Would like to start sending build and test results, particularly for
         older kernels. Would like to separate upstream commits from project
         patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

     Huawei Compass CI
         Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
         No response so far.

!   Microsoft
!       I provided Liz Zhang with instructions and credentials for submitting
!       Penguinator data to KCIDB.

Please respond with corrections or suggestions of other CI systems to contact.

Thank you!
Nick

*More-or-less :)

**KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
   Foundation's KernelCI project:
   https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

#KCIDB engagement report

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly* report on KCIDB** engagement. It lists various CI
systems and their status of engagement with KCIDB, as well as developer
engagement.

We have no news on CI system engagement, but we're working with our KernelCI
colleague Mark Brown on figuring out suitable notifications for him.

We've implemented sending notifications one hour after the last update for a
revision arrives, thus approximating the traditional behavior of "send report
when testing is done". Here's an example:

     https://groups.io/g/kernelci-results-staging/message/11899

However, this is still hardly useful, as there are so many failures. Thus we
again arrive at the inescapable need for masking known failures somehow.

To that end, I went over how various kernel CI systems do it, to try to find
a solution which would accommodate everyone, and wrapped that into a FOSDEM
talk:

 
https://fosdem.org/2022/schedule/event/masking_known_issues_across_six_kernel_ci_systems/

I have a few ideas I'll be presenting there, as well as a few problems, of
course, and would appreciate your feedback. I'd be glad to hear any
corrections to my summaries of how CI systems operate, too, as I had to
prepare those in a rush.

CI Engagement Status

     KernelCI native
         Sending (a lot of) production build and test results.
         https://kcidb.kernelci.org/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://kcidb.kernelci.org/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://kcidb.kernelci.org/?var-origin=syzbot

     ARM
         Sending production results.
         Full commit hashes are currently not available, are spoofed, and don't
         match the ones reported by others. There's ongoing switch to using a
         KernelCI instance setup, which should fix this.
         https://kcidb.kernelci.org/?var-origin=arm

     Sony Fuego
         Internal design in progress.

     Gentoo GKernelCI
         Sending production results.
         Only builds (a few architectures), no configs, no logs, and no tests
         for now, but working on growing contributions.
         https://kcidb.kernelci.org/?var-origin=gkernelci

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

     Linaro
         Sending (a lot of) Tuxsuite build results to "production" KCIDB.
         https://kcidb.kernelci.org/?var-origin=tuxsuite

     TuxML
         Initial contact in response to a report.
         There's a plan to approach us and start work in the coming months.

     Yocto Project
         Initial contact in response to a report.
         Would like to start sending build and test results, particularly for
         older kernels. Would like to separate upstream commits from project
         patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

     Huawei Compass CI
         Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
         No response so far.

Please respond with corrections or suggestions of other CI systems to contact.

Thank you!
Nick

*More-or-less :)

**KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
   Foundation's KernelCI project:
   https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

#KCIDB engagement report

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly* report on KCIDB* engagement. It lists various CI systems
and their status of engagement with KCIDB, as well as developer engagement.

This report doesn't have engagement news, as we're fully concentrating on
getting the next release out, so that we can start reaching out to developers.

Right now we have test streams working taking in both the old and the new
schemas and generating notifications, and are working on upgrading our Grafana
dashboards to the new schema as well. Expect the release this Autumn.

Additionally, we're processing contributions from Outreachy internship
candidates and we've got resources for three interns the round starting in
December.

There's one small bit of engagement news, though: ARM is working on switching
to using a KernelCI instance internally, which should fix the faked commit
hashes.

CI Engagement Status

     KernelCI native
         Sending (a lot of) production build and test results.
         https://kcidb.kernelci.org/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://kcidb.kernelci.org/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://kcidb.kernelci.org/?var-origin=syzbot

     ARM
         Sending production results.
!       Full commit hashes are currently not available, are spoofed, and don't
!       match the ones reported by others. There's ongoing switch to using a
!       KernelCI instance setup, which should fix this.
         https://kcidb.kernelci.org/?var-origin=arm

     Sony Fuego
         Internal design in progress.

     Gentoo GKernelCI
         Sending production results.
         Only builds (a few architectures), no configs, no logs, and no tests
         for now, but working on growing contributions.
         https://kcidb.kernelci.org/?var-origin=gkernelci

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

     Linaro
         Sending (a lot of) Tuxsuite build results to "production" KCIDB.
         https://kcidb.kernelci.org/?var-origin=tuxsuite

     TuxML
         Initial contact in response to a report.
         There's a plan to approach us and start work in the coming months.

     Yocto Project
         Initial contact in response to a report.
         Would like to start sending build and test results, particularly for
         older kernels. Would like to separate upstream commits from project
         patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

     Huawei Compass CI
         Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
         No response so far.

Please respond with corrections or suggestions of other CI systems to contact.

Nick

*Sorry, skipped last month update due to Linux Plumbers!

*KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
  Foundation's KernelCI project:
  https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

#KCIDB engagement report

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly report on KCIDB* engagement. It lists various CI systems
and their status of engagement with KCIDB, as well as developer engagement.

This is a belated report, and has no news, as I was on vacation half of the
time, and the other half was occupied with CKI Project. Now my breaks
are over, and I hope to wrap up CKI work soon, to get back to KCIDB.

CI Engagement Status

     KernelCI native
         Sending (a lot of) production build and test results.
         https://kcidb.kernelci.org/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://kcidb.kernelci.org/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://kcidb.kernelci.org/?var-origin=syzbot

     ARM
         Sending production results.
         Full commit hashes are currently not available, are spoofed, and don't
         match the ones reported by others. To be fixed soon.
         https://kcidb.kernelci.org/?var-origin=arm

     Sony Fuego
         Internal design in progress.

     Gentoo GKernelCI
         Sending production results.
         Only builds (a few architectures), no configs, no logs, and no tests
         for now, but working on growing contributions.
         https://kcidb.kernelci.org/?var-origin=gkernelci

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

     Linaro
         Sending (a lot of) Tuxsuite build results to "production" KCIDB.
         https://kcidb.kernelci.org/?var-origin=tuxsuite

     TuxML
         Initial contact in response to a report.
         There's a plan to approach us and start work in the coming months.

     Yocto Project
         Initial contact in response to a report.
         Would like to start sending build and test results, particularly for
         older kernels. Would like to separate upstream commits from project
         patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

     Huawei Compass CI
         Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
         No response so far.

Please respond with corrections or suggestions of other CI systems to contact.

Nick

*KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
  Foundation's KernelCI project:
  https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

Re: #KCIDB engagement report

[Nick Desaulniers]

On Mon, Aug 23, 2021 at 5:37 AM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> Hi everyone,
>
> Below is the monthly report on KCIDB* engagement. It lists various CI systems
> and their status of engagement with KCIDB, as well as developer engagement.
>
> This is a belated report, and has no news, as I was on vacation half of the
> time, and the other half was occupied with CKI Project. Now my breaks
> are over, and I hope to wrap up CKI work soon, to get back to KCIDB.
>
> CI Engagement Status
>
>      KernelCI native
>          Sending (a lot of) production build and test results.
>          https://kcidb.kernelci.org/?var-origin=kernelci
>
>      Red Hat CKI
>          Sending production results.
>          https://kcidb.kernelci.org/?var-origin=redhat
>
>      Google Syzbot
>          Sending a subset of production results (failures only).
>          https://kcidb.kernelci.org/?var-origin=syzbot
>
>      ARM
>          Sending production results.
>          Full commit hashes are currently not available, are spoofed, and don't
>          match the ones reported by others. To be fixed soon.
>          https://kcidb.kernelci.org/?var-origin=arm
>
>      Sony Fuego
>          Internal design in progress.
>
>      Gentoo GKernelCI
>          Sending production results.
>          Only builds (a few architectures), no configs, no logs, and no tests
>          for now, but working on growing contributions.
>          https://kcidb.kernelci.org/?var-origin=gkernelci
>
>      Intel 0day
>          Initial conversation concluded, general interest expressed,
>          no contact since.

+ 0 day folks (probably just need to reach out again)

>
>      Linaro
>          Sending (a lot of) Tuxsuite build results to "production" KCIDB.
>          https://kcidb.kernelci.org/?var-origin=tuxsuite
>
>      TuxML
>          Initial contact in response to a report.
>          There's a plan to approach us and start work in the coming months.
>
>      Yocto Project
>          Initial contact in response to a report.
>          Would like to start sending build and test results, particularly for
>          older kernels. Would like to separate upstream commits from project
>          patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196
>
>      Huawei Compass CI
>          Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
>          No response so far.
>
> Please respond with corrections or suggestions of other CI systems to contact.
>
> Nick
>
> *KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
>   Foundation's KernelCI project:
>   https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
>
>
>
> 
>
>


-- 
Thanks,
~Nick Desaulniers

#KCIDB engagement report

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly report on KCIDB* engagement. It lists various CI systems
and their status of engagement with KCIDB, as well as developer engagement.

No news on CI engagement. I'm still working on the next release, which will
certainly cause more updates.

However, I got a few requests from Nick Desaulniers regarding the KCIDB
dashboards, and after a few iterations we now have a few more dashboards,
panels, parameters, and graphs, which would hopefully help him in his
kernel development work.

We've also got a dedicated domain name for the dashboards:

     https://kcidb.kernelci.org/

CI Engagement Status

     KernelCI native
         Sending (a lot of) production build and test results.
         https://kcidb.kernelci.org/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://kcidb.kernelci.org/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://kcidb.kernelci.org/?var-origin=syzbot

     ARM
         Sending production results.
         Full commit hashes are currently not available, are spoofed, and don't
         match the ones reported by others. To be fixed soon.
         https://kcidb.kernelci.org/?var-origin=arm

     Sony Fuego
         Internal design in progress.

     Gentoo GKernelCI
         Sending production results.
         Only builds (a few architectures), no configs, no logs, and no tests
         for now, but working on growing contributions.
         https://kcidb.kernelci.org/?var-origin=gkernelci

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

     Linaro
         Sending (a lot of) Tuxsuite build results to "production" KCIDB.
         https://kcidb.kernelci.org/?var-origin=tuxsuite

     TuxML
         Initial contact in response to a report.
         There's a plan to approach us and start work in the coming months.

     Yocto Project
         Initial contact in response to a report.
         Would like to start sending build and test results, particularly for
         older kernels. Would like to separate upstream commits from project
         patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

     Huawei Compass CI
         Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
         No response so far.

Please respond with corrections or suggestions of other CI systems to contact.

Nick

*KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
  Foundation's KernelCI project:
  https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

#KCIDB engagement report

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly report on KCIDB* engagement. It lists various CI systems
and their status of engagement with KCIDB, and once we get to that, will list
developer engagement.

Lines with updates are marked with "!".

Not much news this time, as I had to tend to CKI matters, and had a couple 
weeks of vacation. I still have to tie some loose CKI ends before I return to
working on a new KCIDB release and reaching developers with e-mail
notifications.

However, I did try to contact Huawei's Compass CI with an invitation for 
cooperation, but got no response so far.

     KernelCI native
         Sending (a lot of) production build and test results.
         https://staging.kernelci.org:3000/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://staging.kernelci.org:3000/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://staging.kernelci.org:3000/?var-origin=syzbot

     ARM
         Sending production results.
         Full commit hashes are currently not available, are spoofed, and don't
         match the ones reported by others. To be fixed soon.
         https://staging.kernelci.org:3000/?var-origin=arm

     Sony Fuego
         Internal design in progress.

     Gentoo GKernelCI
         Sending production results.
         Only builds (a few architectures), no configs, no logs, and no tests
         for now, but working on growing contributions.
         https://staging.kernelci.org:3000/?var-origin=gkernelci

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

     Linaro
         Sending (a lot of) Tuxsuite build results to "production" KCIDB.
         https://staging.kernelci.org:3000/?var-origin=tuxsuite

     TuxML
         Initial contact in response to a report.
         There's a plan to approach us and start work in the coming months.

     Yocto Project
         Initial contact in response to a report.
         Would like to start sending build and test results, particularly for
         older kernels. Would like to separate upstream commits from project
         patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

!   Huawei Compass CI
!       Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
!       No response so far.

Please respond with corrections or suggestions of other CI systems to contact.

Nick

*KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
  Foundation's KernelCI project:
  https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

Re: #KCIDB engagement report

[Nick Desaulniers]

On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> Hi everyone,
>
> Below is the monthly report on KCIDB* engagement. It lists various CI systems
> and their status of engagement with KCIDB, and once we get to that, will list
> developer engagement.
>
> Lines with updates are marked with "!".
>
> Not much news this time, as I had to tend to CKI matters, and had a couple
> weeks of vacation. I still have to tie some loose CKI ends before I return to
> working on a new KCIDB release and reaching developers with e-mail
> notifications.
>
> However, I did try to contact Huawei's Compass CI with an invitation for
> cooperation, but got no response so far.
>
>      KernelCI native
>          Sending (a lot of) production build and test results.
>          https://staging.kernelci.org:3000/?var-origin=kernelci
>
>      Red Hat CKI
>          Sending production results.
>          https://staging.kernelci.org:3000/?var-origin=redhat
>
>      Google Syzbot
>          Sending a subset of production results (failures only).
>          https://staging.kernelci.org:3000/?var-origin=syzbot
>
>      ARM
>          Sending production results.
>          Full commit hashes are currently not available, are spoofed, and don't
>          match the ones reported by others. To be fixed soon.
>          https://staging.kernelci.org:3000/?var-origin=arm
>
>      Sony Fuego
>          Internal design in progress.
>
>      Gentoo GKernelCI
>          Sending production results.
>          Only builds (a few architectures), no configs, no logs, and no tests
>          for now, but working on growing contributions.
>          https://staging.kernelci.org:3000/?var-origin=gkernelci
>
>      Intel 0day
>          Initial conversation concluded, general interest expressed,
>          no contact since.
>
>      Linaro
>          Sending (a lot of) Tuxsuite build results to "production" KCIDB.
>          https://staging.kernelci.org:3000/?var-origin=tuxsuite

Hi Nikolai,
It's nice to see our results getting collected; it looks for a given
tree I can even see the build results of different compilers.

For example, here's a recent run of mainline:
https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-dataset=kernelci04&var-id=c4681547bcce777daf576925a966ffa824edd09d

One thing we need to be able to quickly triage when we see a build
failure with one toolchain is "is this toolchain specific or not?"  I
figure KCIDB has the data; is there a way to surface the results of
such a query quickly?  If not, that would really help us.

>
>      TuxML
>          Initial contact in response to a report.
>          There's a plan to approach us and start work in the coming months.
>
>      Yocto Project
>          Initial contact in response to a report.
>          Would like to start sending build and test results, particularly for
>          older kernels. Would like to separate upstream commits from project
>          patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196
>
> !   Huawei Compass CI
> !       Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
> !       No response so far.
>
> Please respond with corrections or suggestions of other CI systems to contact.
>
> Nick
>
> *KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
>   Foundation's KernelCI project:
>   https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
>
>
>
> 
>
>


-- 
Thanks,
~Nick Desaulniers

Re: #KCIDB engagement report

[Nikolai Kondrashov]

Hi Nick,

On 5/24/21 8:38 PM, Nick Desaulniers via groups.io wrote:
 > Hi Nikolai,
 > It's nice to see our results getting collected; it looks for a given
 > tree I can even see the build results of different compilers.
 >
 > For example, here's a recent run of mainline:
 > 
https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-dataset=kernelci04&var-id=c4681547bcce777daf576925a966ffa824edd09d
 >
 > One thing we need to be able to quickly triage when we see a build
 > failure with one toolchain is "is this toolchain specific or not?"  I
 > figure KCIDB has the data; is there a way to surface the results of
 > such a query quickly?  If not, that would really help us.

We don't have a ready-made UI for this, but I think I can add a Grafana
panel/dashboard for that rather quickly. What would be most helpful?

How about having a list of "Compilers" below the "Builds" on the page
you link? Each line in that list could correspond to a unique value of
the "Compiler" field, and give an aggregated summary of various
parameters, including build/test results. We can also have a summary per
architecture, or per "Configuration".

Or maybe something else would help you better?

Nick

On 5/24/21 8:38 PM, Nick Desaulniers via groups.io wrote:
> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
> <Nikolai.Kondrashov@redhat.com> wrote:
>>
>> Hi everyone,
>>
>> Below is the monthly report on KCIDB* engagement. It lists various CI systems
>> and their status of engagement with KCIDB, and once we get to that, will list
>> developer engagement.
>>
>> Lines with updates are marked with "!".
>>
>> Not much news this time, as I had to tend to CKI matters, and had a couple
>> weeks of vacation. I still have to tie some loose CKI ends before I return to
>> working on a new KCIDB release and reaching developers with e-mail
>> notifications.
>>
>> However, I did try to contact Huawei's Compass CI with an invitation for
>> cooperation, but got no response so far.
>>
>>       KernelCI native
>>           Sending (a lot of) production build and test results.
>>           https://staging.kernelci.org:3000/?var-origin=kernelci
>>
>>       Red Hat CKI
>>           Sending production results.
>>           https://staging.kernelci.org:3000/?var-origin=redhat
>>
>>       Google Syzbot
>>           Sending a subset of production results (failures only).
>>           https://staging.kernelci.org:3000/?var-origin=syzbot
>>
>>       ARM
>>           Sending production results.
>>           Full commit hashes are currently not available, are spoofed, and don't
>>           match the ones reported by others. To be fixed soon.
>>           https://staging.kernelci.org:3000/?var-origin=arm
>>
>>       Sony Fuego
>>           Internal design in progress.
>>
>>       Gentoo GKernelCI
>>           Sending production results.
>>           Only builds (a few architectures), no configs, no logs, and no tests
>>           for now, but working on growing contributions.
>>           https://staging.kernelci.org:3000/?var-origin=gkernelci
>>
>>       Intel 0day
>>           Initial conversation concluded, general interest expressed,
>>           no contact since.
>>
>>       Linaro
>>           Sending (a lot of) Tuxsuite build results to "production" KCIDB.
>>           https://staging.kernelci.org:3000/?var-origin=tuxsuite
> 
> Hi Nikolai,
> It's nice to see our results getting collected; it looks for a given
> tree I can even see the build results of different compilers.
> 
> For example, here's a recent run of mainline:
> https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-dataset=kernelci04&var-id=c4681547bcce777daf576925a966ffa824edd09d
> 
> One thing we need to be able to quickly triage when we see a build
> failure with one toolchain is "is this toolchain specific or not?"  I
> figure KCIDB has the data; is there a way to surface the results of
> such a query quickly?  If not, that would really help us.
> 
>>
>>       TuxML
>>           Initial contact in response to a report.
>>           There's a plan to approach us and start work in the coming months.
>>
>>       Yocto Project
>>           Initial contact in response to a report.
>>           Would like to start sending build and test results, particularly for
>>           older kernels. Would like to separate upstream commits from project
>>           patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196
>>
>> !   Huawei Compass CI
>> !       Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
>> !       No response so far.
>>
>> Please respond with corrections or suggestions of other CI systems to contact.
>>
>> Nick
>>
>> *KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
>>    Foundation's KernelCI project:
>>    https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
>>
>>
>>
>>
>>
>>
> 
> 

Re: #KCIDB engagement report

[Nick Desaulniers]

On Tue, May 25, 2021 at 3:32 AM Nikolai Kondrashov <spbnick@gmail.com> wrote:
>
> Hi Nick,
>
> On 5/24/21 8:38 PM, Nick Desaulniers via groups.io wrote:
>  > Hi Nikolai,
>  > It's nice to see our results getting collected; it looks for a given
>  > tree I can even see the build results of different compilers.
>  >
>  > For example, here's a recent run of mainline:
>  >
> https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-dataset=kernelci04&var-id=c4681547bcce777daf576925a966ffa824edd09d
>  >
>  > One thing we need to be able to quickly triage when we see a build
>  > failure with one toolchain is "is this toolchain specific or not?"  I
>  > figure KCIDB has the data; is there a way to surface the results of
>  > such a query quickly?  If not, that would really help us.
>
> We don't have a ready-made UI for this, but I think I can add a Grafana
> panel/dashboard for that rather quickly. What would be most helpful?

I think so.

For a given tuple of (tree, branch, configuration), it would be neat
to be able to link to a deterministic URL to quickly check who else
may have built this recently, and what was the result.

> How about having a list of "Compilers" below the "Builds" on the page
> you link? Each line in that list could correspond to a unique value of
> the "Compiler" field, and give an aggregated summary of various
> parameters, including build/test results. We can also have a summary per
> architecture, or per "Configuration".
>
> Or maybe something else would help you better?

Hard to imagine, but maybe we can iterate on something?

>
> Nick
>
> On 5/24/21 8:38 PM, Nick Desaulniers via groups.io wrote:
> > On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
> > <Nikolai.Kondrashov@redhat.com> wrote:
> >>
> >> Hi everyone,
> >>
> >> Below is the monthly report on KCIDB* engagement. It lists various CI systems
> >> and their status of engagement with KCIDB, and once we get to that, will list
> >> developer engagement.
> >>
> >> Lines with updates are marked with "!".
> >>
> >> Not much news this time, as I had to tend to CKI matters, and had a couple
> >> weeks of vacation. I still have to tie some loose CKI ends before I return to
> >> working on a new KCIDB release and reaching developers with e-mail
> >> notifications.
> >>
> >> However, I did try to contact Huawei's Compass CI with an invitation for
> >> cooperation, but got no response so far.
> >>
> >>       KernelCI native
> >>           Sending (a lot of) production build and test results.
> >>           https://staging.kernelci.org:3000/?var-origin=kernelci
> >>
> >>       Red Hat CKI
> >>           Sending production results.
> >>           https://staging.kernelci.org:3000/?var-origin=redhat
> >>
> >>       Google Syzbot
> >>           Sending a subset of production results (failures only).
> >>           https://staging.kernelci.org:3000/?var-origin=syzbot
> >>
> >>       ARM
> >>           Sending production results.
> >>           Full commit hashes are currently not available, are spoofed, and don't
> >>           match the ones reported by others. To be fixed soon.
> >>           https://staging.kernelci.org:3000/?var-origin=arm
> >>
> >>       Sony Fuego
> >>           Internal design in progress.
> >>
> >>       Gentoo GKernelCI
> >>           Sending production results.
> >>           Only builds (a few architectures), no configs, no logs, and no tests
> >>           for now, but working on growing contributions.
> >>           https://staging.kernelci.org:3000/?var-origin=gkernelci
> >>
> >>       Intel 0day
> >>           Initial conversation concluded, general interest expressed,
> >>           no contact since.
> >>
> >>       Linaro
> >>           Sending (a lot of) Tuxsuite build results to "production" KCIDB.
> >>           https://staging.kernelci.org:3000/?var-origin=tuxsuite
> >
> > Hi Nikolai,
> > It's nice to see our results getting collected; it looks for a given
> > tree I can even see the build results of different compilers.
> >
> > For example, here's a recent run of mainline:
> > https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-dataset=kernelci04&var-id=c4681547bcce777daf576925a966ffa824edd09d
> >
> > One thing we need to be able to quickly triage when we see a build
> > failure with one toolchain is "is this toolchain specific or not?"  I
> > figure KCIDB has the data; is there a way to surface the results of
> > such a query quickly?  If not, that would really help us.
> >
> >>
> >>       TuxML
> >>           Initial contact in response to a report.
> >>           There's a plan to approach us and start work in the coming months.
> >>
> >>       Yocto Project
> >>           Initial contact in response to a report.
> >>           Would like to start sending build and test results, particularly for
> >>           older kernels. Would like to separate upstream commits from project
> >>           patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196
> >>
> >> !   Huawei Compass CI
> >> !       Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
> >> !       No response so far.
> >>
> >> Please respond with corrections or suggestions of other CI systems to contact.
> >>
> >> Nick
> >>
> >> *KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
> >>    Foundation's KernelCI project:
> >>    https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
>


-- 
Thanks,
~Nick Desaulniers

Re: #KCIDB engagement report

[Nikolai Kondrashov]

Hi Nick,

 >> We don't have a ready-made UI for this, but I think I can add a Grafana
 >> panel/dashboard for that rather quickly. What would be most helpful?
 >
 > I think so.
 >
 > For a given tuple of (tree, branch, configuration), it would be neat
 > to be able to link to a deterministic URL to quickly check who else
 > may have built this recently, and what was the result.

I made a stab at it. I added "Repository" and "Branch" dashboards, showing 
revisions for a particular repository and branch respectively. They are 
accessible from the dropdown menu in the top left corner.

Repositories are also linked from the "Home" dashboard, branches - from 
"Repository" dashboard, and both are linked from "Revision" dashboard.

Additionally, "Home", "Repository", "Branch", and "Revision" dashboards now 
allow filtering builds by architecture and configuration name. Please be aware 
that neither are really standardized across submitters yet.

Finally, whoever is reading this, please be aware of the time range selector 
in the top right corner. It affects every dashboard.

 >> How about having a list of "Compilers" below the "Builds" on the page
 >> you link? Each line in that list could correspond to a unique value of
 >> the "Compiler" field, and give an aggregated summary of various
 >> parameters, including build/test results. We can also have a summary per
 >> architecture, or per "Configuration".
 >>
 >> Or maybe something else would help you better?
 >
 > Hard to imagine, but maybe we can iterate on something?

Sure, check it out and tell me what you'd like done differently :)

Nick

On 6/1/21 10:10 PM, Nick Desaulniers wrote:
> On Tue, May 25, 2021 at 3:32 AM Nikolai Kondrashov <spbnick@gmail.com> wrote:
>>
>> Hi Nick,
>>
>> On 5/24/21 8:38 PM, Nick Desaulniers via groups.io wrote:
>>   > Hi Nikolai,
>>   > It's nice to see our results getting collected; it looks for a given
>>   > tree I can even see the build results of different compilers.
>>   >
>>   > For example, here's a recent run of mainline:
>>   >
>> https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-dataset=kernelci04&var-id=c4681547bcce777daf576925a966ffa824edd09d
>>   >
>>   > One thing we need to be able to quickly triage when we see a build
>>   > failure with one toolchain is "is this toolchain specific or not?"  I
>>   > figure KCIDB has the data; is there a way to surface the results of
>>   > such a query quickly?  If not, that would really help us.
>>
>> We don't have a ready-made UI for this, but I think I can add a Grafana
>> panel/dashboard for that rather quickly. What would be most helpful?
> 
> I think so.
> 
> For a given tuple of (tree, branch, configuration), it would be neat
> to be able to link to a deterministic URL to quickly check who else
> may have built this recently, and what was the result.
> 
>> How about having a list of "Compilers" below the "Builds" on the page
>> you link? Each line in that list could correspond to a unique value of
>> the "Compiler" field, and give an aggregated summary of various
>> parameters, including build/test results. We can also have a summary per
>> architecture, or per "Configuration".
>>
>> Or maybe something else would help you better?
> 
> Hard to imagine, but maybe we can iterate on something?
> 
>>
>> Nick
>>
>> On 5/24/21 8:38 PM, Nick Desaulniers via groups.io wrote:
>>> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
>>> <Nikolai.Kondrashov@redhat.com> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> Below is the monthly report on KCIDB* engagement. It lists various CI systems
>>>> and their status of engagement with KCIDB, and once we get to that, will list
>>>> developer engagement.
>>>>
>>>> Lines with updates are marked with "!".
>>>>
>>>> Not much news this time, as I had to tend to CKI matters, and had a couple
>>>> weeks of vacation. I still have to tie some loose CKI ends before I return to
>>>> working on a new KCIDB release and reaching developers with e-mail
>>>> notifications.
>>>>
>>>> However, I did try to contact Huawei's Compass CI with an invitation for
>>>> cooperation, but got no response so far.
>>>>
>>>>        KernelCI native
>>>>            Sending (a lot of) production build and test results.
>>>>            https://staging.kernelci.org:3000/?var-origin=kernelci
>>>>
>>>>        Red Hat CKI
>>>>            Sending production results.
>>>>            https://staging.kernelci.org:3000/?var-origin=redhat
>>>>
>>>>        Google Syzbot
>>>>            Sending a subset of production results (failures only).
>>>>            https://staging.kernelci.org:3000/?var-origin=syzbot
>>>>
>>>>        ARM
>>>>            Sending production results.
>>>>            Full commit hashes are currently not available, are spoofed, and don't
>>>>            match the ones reported by others. To be fixed soon.
>>>>            https://staging.kernelci.org:3000/?var-origin=arm
>>>>
>>>>        Sony Fuego
>>>>            Internal design in progress.
>>>>
>>>>        Gentoo GKernelCI
>>>>            Sending production results.
>>>>            Only builds (a few architectures), no configs, no logs, and no tests
>>>>            for now, but working on growing contributions.
>>>>            https://staging.kernelci.org:3000/?var-origin=gkernelci
>>>>
>>>>        Intel 0day
>>>>            Initial conversation concluded, general interest expressed,
>>>>            no contact since.
>>>>
>>>>        Linaro
>>>>            Sending (a lot of) Tuxsuite build results to "production" KCIDB.
>>>>            https://staging.kernelci.org:3000/?var-origin=tuxsuite
>>>
>>> Hi Nikolai,
>>> It's nice to see our results getting collected; it looks for a given
>>> tree I can even see the build results of different compilers.
>>>
>>> For example, here's a recent run of mainline:
>>> https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-dataset=kernelci04&var-id=c4681547bcce777daf576925a966ffa824edd09d
>>>
>>> One thing we need to be able to quickly triage when we see a build
>>> failure with one toolchain is "is this toolchain specific or not?"  I
>>> figure KCIDB has the data; is there a way to surface the results of
>>> such a query quickly?  If not, that would really help us.
>>>
>>>>
>>>>        TuxML
>>>>            Initial contact in response to a report.
>>>>            There's a plan to approach us and start work in the coming months.
>>>>
>>>>        Yocto Project
>>>>            Initial contact in response to a report.
>>>>            Would like to start sending build and test results, particularly for
>>>>            older kernels. Would like to separate upstream commits from project
>>>>            patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196
>>>>
>>>> !   Huawei Compass CI
>>>> !       Sent a message to Fengguang Wu, who was presenting it at LVC 2021.
>>>> !       No response so far.
>>>>
>>>> Please respond with corrections or suggestions of other CI systems to contact.
>>>>
>>>> Nick
>>>>
>>>> *KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
>>>>     Foundation's KernelCI project:
>>>>     https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
> 
> 

Re: #KCIDB engagement report

[Nick Desaulniers]

On Mon, Jun 7, 2021 at 4:13 AM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
>
> Hi Nick,
>
>  >> We don't have a ready-made UI for this, but I think I can add a Grafana
>  >> panel/dashboard for that rather quickly. What would be most helpful?
>  >
>  > I think so.
>  >
>  > For a given tuple of (tree, branch, configuration), it would be neat
>  > to be able to link to a deterministic URL to quickly check who else
>  > may have built this recently, and what was the result.
>
> I made a stab at it. I added "Repository" and "Branch" dashboards, showing
> revisions for a particular repository and branch respectively. They are
> accessible from the dropdown menu in the top left corner.
>
> Repositories are also linked from the "Home" dashboard, branches - from
> "Repository" dashboard, and both are linked from "Revision" dashboard.
>
> Additionally, "Home", "Repository", "Branch", and "Revision" dashboards now
> allow filtering builds by architecture and configuration name. Please be aware
> that neither are really standardized across submitters yet.
>
> Finally, whoever is reading this, please be aware of the time range selector
> in the top right corner. It affects every dashboard.

Cool.  Some notes from playing around with it:
- might need to de-duplicate
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git vs
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
- consider sorting list of branches in drop down
- "tuxsuite" seems to get truncated to "..." for me.
- builds over time are useful, but it would be more empowering to know
which of those builds were green vs red.

I'm not sure quite yet how to drill down to see which builds were from
which toolchain, but this looks pretty close to what I had in mind.
-- 
Thanks,
~Nick Desaulniers

Re: #KCIDB engagement report

[Nikolai Kondrashov]

Hi Nick,

 > Cool.  Some notes from playing around with it:
 > - might need to de-duplicate
 > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git vs
 > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

We have a preference for https:// URLs in the schema, but sometimes submitters 
overlook that, and submit git:// anyway. It's a catch-up game, and I ping 
submitters when I see this happening, later fixing URLs in the database.

Similar thing happens with other fields (e.g. architecture). We'll be 
tightening the requirements as we converge on a common, better defined schema, 
but for now permissible schema (and messy data) allows us to ramp up 
participation quickly.

 > - consider sorting list of branches in drop down

They were sorted in descending order to have things like queue/5.9 on top, but 
I switched that to ascending, as that's more universal and easier to understand.

 > - "tuxsuite" seems to get truncated to "..." for me.

I assume you mean the "Origin" cell in the "Revisions" table. I substitute ". 
. . . ." for origin, when there's more than one, as more than one wouldn't fit 
the table. Full list is shown when you open the revision. You can also use the 
"Origin" filter on top of the page, if you're looking for data from specific 
origin.

 > - builds over time are useful, but it would be more empowering to know
 > which of those builds were green vs red.

Yes, I was thinking about that as well.

I added graphs for build status, as well as test status, over time, to the 
"Home", "Repository", and "Branch" dashboards. However, please be aware, that 
those are results over time the *result* was produced, not commit order. 
Strictly speaking, that can be any order, and results for some older commits 
might pop up e.g. a couple days later. Normally, though, they wouldn't, and 
there's some use in these graphs.

We don't have commit order/connectivity data in the database (at least yet), 
so this is the best I can do so far.

I also added a few basic graphs to the "Revision" dashboard, please tell me if 
any of those are useful.

Thanks for the feedback and the requests.
Give me some more, if you have them :)
Nick

On 6/7/21 9:09 PM, Nick Desaulniers wrote:
> On Mon, Jun 7, 2021 at 4:13 AM Nikolai Kondrashov
> <Nikolai.Kondrashov@redhat.com> wrote:
>>
>> Hi Nick,
>>
>>   >> We don't have a ready-made UI for this, but I think I can add a Grafana
>>   >> panel/dashboard for that rather quickly. What would be most helpful?
>>   >
>>   > I think so.
>>   >
>>   > For a given tuple of (tree, branch, configuration), it would be neat
>>   > to be able to link to a deterministic URL to quickly check who else
>>   > may have built this recently, and what was the result.
>>
>> I made a stab at it. I added "Repository" and "Branch" dashboards, showing
>> revisions for a particular repository and branch respectively. They are
>> accessible from the dropdown menu in the top left corner.
>>
>> Repositories are also linked from the "Home" dashboard, branches - from
>> "Repository" dashboard, and both are linked from "Revision" dashboard.
>>
>> Additionally, "Home", "Repository", "Branch", and "Revision" dashboards now
>> allow filtering builds by architecture and configuration name. Please be aware
>> that neither are really standardized across submitters yet.
>>
>> Finally, whoever is reading this, please be aware of the time range selector
>> in the top right corner. It affects every dashboard.
> 
> Cool.  Some notes from playing around with it:
> - might need to de-duplicate
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git vs
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> - consider sorting list of branches in drop down
> - "tuxsuite" seems to get truncated to "..." for me.
> - builds over time are useful, but it would be more empowering to know
> which of those builds were green vs red.
> 
> I'm not sure quite yet how to drill down to see which builds were from
> which toolchain, but this looks pretty close to what I had in mind.
> 

Re: #KCIDB engagement report

[Nick Desaulniers]

On Thu, Jun 10, 2021 at 2:15 AM Nikolai Kondrashov
<Nikolai.Kondrashov@redhat.com> wrote:
> Thanks for the feedback and the requests.
> Give me some more, if you have them :)

Awesome!!!! I really like clearly seeing the number of builds that
succeeded vs failed.  The numbers for the "Top 10 architecture build
failures" add up to the total number of build failures which is great.

We can clearly see which toolchain was used in the table.

Oh, I clicked something and can't back the nice histograms.
https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-dataset=kernelci04&var-id=c4681547bcce777daf576925a966ffa824edd09d
vs
https://kcidb.kernelci.org/d/branch/branch?orgId=1&var-dataset=kernelci04&var-git_repository_url=https:%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git&var-git_repository_branch=master&var-origin=All&var-build_architecture=All&var-build_config_name=All

The first URL was in my history, so I just went directly there; but I
can't figure out how to get back there from the existing UI elements.
I click "Home" (top left) > Branch > Repository URL >
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

(the "name" field defaults to "kunit" rather than "master"???)  Oh, is
it "Revision" rather than "Branch" that I should be using? No, that
doesn't seem to be it...hmm...

I really really like the histograms for build failures; I'm most
interested in seeing one by toolchain (we have bugs specific to just
newer vs older versions of clang all of the time); perhaps folks might
like to slice along any of the columns in the table?

The build status in the latter link with red vs green area was exactly
what I was imagining. Great work!
-- 
Thanks,
~Nick Desaulniers

Re: #KCIDB engagement report

[Nikolai Kondrashov]

Hi Nick,

On 6/11/21 2:38 AM, Nick Desaulniers wrote:
 > On Thu, Jun 10, 2021 at 2:15 AM Nikolai Kondrashov
 > <Nikolai.Kondrashov@redhat.com> wrote:
 >> Thanks for the feedback and the requests.
 >> Give me some more, if you have them :)
 >
 > Awesome!!!! I really like clearly seeing the number of builds that
 > succeeded vs failed.  The numbers for the "Top 10 architecture build
 > failures" add up to the total number of build failures which is great.

Eh, the sum would stop matching as soon as there are more than 10 failed 
architectures :D I can't really squeeze all of them in there, as it would 
become unreadable on at least some displays, that's worse for configurations, 
and would be much worse for compilers.

 >
 > We can clearly see which toolchain was used in the table.
 >
 > Oh, I clicked something and can't back the nice histograms.
 > 
https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-dataset=kernelci04&var-id=c4681547bcce777daf576925a966ffa824edd09d
 > vs
 > 
https://kcidb.kernelci.org/d/branch/branch?orgId=1&var-dataset=kernelci04&var-git_repository_url=https:%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git&var-git_repository_branch=master&var-origin=All&var-build_architecture=All&var-build_config_name=All
 >
 > The first URL was in my history, so I just went directly there; but I
 > can't figure out how to get back there from the existing UI elements.
 > I click "Home" (top left) > Branch > Repository URL >
 > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
 >
 > (the "name" field defaults to "kunit" rather than "master"???)  Oh, is
 > it "Revision" rather than "Branch" that I should be using? No, that
 > doesn't seem to be it...hmm...

Those "histograms" are only added to the "Revision" dashboard. Watch the name 
of the dashboard in the top left corner. I guess I can add them to the "Repo" 
and "Branch" dashboards as well. Would the graphs make sense when they are 
across multiple revisions?

 > I really really like the histograms for build failures; I'm most
 > interested in seeing one by toolchain (we have bugs specific to just
 > newer vs older versions of clang all of the time); perhaps folks might
 > like to slice along any of the columns in the table?

Grafana doesn't seem to allow doing that dynamically, but maybe I can leave 
only one graph, and add another drop-down list on top for selecting which 
metric you want the graph for.

Meanwhile I'll just add another graph for "Top 5 compiler build failures".

 > The build status in the latter link with red vs green area was exactly
 > what I was imagining. Great work!

Cool :) Unfortunately, when the ratio is too big, the small bar can be really 
hard to see and hit with the mouse. Grafana doesn't have pie charts by 
default, that would've worked better, but I'd rather wait with adding an extra 
pie chart plugin before we upgrade to a newer version.

Nick

Re: #KCIDB engagement report

[Nikolai Kondrashov]

On 6/11/21 2:38 AM, Nick Desaulniers wrote:
> Oh, I clicked something and can't back the nice histograms.
> https://kcidb.kernelci.org/d/revision/revision?orgId=1&var-dataset=kernelci04&var-id=c4681547bcce777daf576925a966ffa824edd09d
> vs
> https://kcidb.kernelci.org/d/branch/branch?orgId=1&var-dataset=kernelci04&var-git_repository_url=https:%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git&var-git_repository_branch=master&var-origin=All&var-build_architecture=All&var-build_config_name=All
> 
> The first URL was in my history, so I just went directly there; but I
> can't figure out how to get back there from the existing UI elements.

You can get to the "Revision" dashboard by clicking any line in a "Revisions" 
table, which appears on "Home", "Repository", and "Branch" dashboards.

Nick

Re: #KCIDB engagement report

[Kees Cook]

On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang Built Linux wrote:
> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
> <Nikolai.Kondrashov@redhat.com> wrote:
> > [...]
> >      KernelCI native
> >          Sending (a lot of) production build and test results.
> >          https://staging.kernelci.org:3000/?var-origin=kernelci
> > [...]

Apologies for the thread hijack, but does anyone know what's happening
with kselftest? It seems missing from the listed[1] build artifacts, but
it is actually present[2] (and I see the logs for generating the tarball
there too), but I can't find any builds that actually run the tests?

(Or how do I see a top-level list of all tests and search it?)

Thanks!

-Kees

[1] https://kcidb.kernelci.org/d/build/build?orgId=1&var-dataset=kernelci04&var-id=kernelci:kernelci.org:60b654321456eb7654b3afa6&fullscreen&panelId=17
[2] https://storage.kernelci.org//mainline/master/v5.13-rc4-11-gc2131f7e73c9/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-8/

-- 
Kees Cook

Re: #KCIDB engagement report

[Guillaume Tucker]

Hi Kees,

On 01/06/2021 21:26, Kees Cook wrote:
> On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang Built Linux wrote:
>> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
>> <Nikolai.Kondrashov@redhat.com> wrote:
>>> [...]
>>>      KernelCI native
>>>          Sending (a lot of) production build and test results.
>>>          https://staging.kernelci.org:3000/?var-origin=kernelci
>>> [...]
> 
> Apologies for the thread hijack, but does anyone know what's happening
> with kselftest? It seems missing from the listed[1] build artifacts, but
> it is actually present[2] (and I see the logs for generating the tarball
> there too), but I can't find any builds that actually run the tests?
> 
> (Or how do I see a top-level list of all tests and search it?)

The kselftest results are all there on the KernelCI native
dashboard, for example the futex tests:

  https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.13-rc5-74-g06af8679449d/plan/kselftest-futex/


Here's a set of passing results on a "coral" x86 Chromebook, with
a bunch of unknowns but that's because other kselftests are being
run when they shouldn't (net, mqueue, ptrace) so it's noise which
should get resolved with a fix soon:

  https://linux.kernelci.org/test/plan/id/60c2bf67ed48b86ffe0c0df8/


And here are the full kernel build details:

  https://linux.kernelci.org/build/id/60c2bdeea60229633d0c0f0c/

and artifacts (logs, binaries, meta-data in JSON):

  https://storage.kernelci.org/mainline/master/v5.13-rc5-74-g06af8679449d/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-8/


So this is the original data, now let's look at what we have in
KCIDB.  Here's the matching build:

  https://kcidb.kernelci.org/d/build/build?orgId=1&var-dataset=kernelci04&var-id=kernelci:kernelci.org:60c2bdeea60229633d0c0f0c

However there's no results, probably because submitting the data
failed for some reason.  It could be due to some invalid
characters in the test names.  The Log link works though, it
takes you to the directory with all the log files - to be
improved as it's advertised as a single build log...

So we'll take a closer look, see if there were any errors in the
logs to find out why the results aren't in KCIDB.  But the
kselftests were definitely run.


Thanks,
Guillaume

> [1] https://kcidb.kernelci.org/d/build/build?orgId=1&var-dataset=kernelci04&var-id=kernelci:kernelci.org:60b654321456eb7654b3afa6&fullscreen&panelId=17
> [2] https://storage.kernelci.org//mainline/master/v5.13-rc4-11-gc2131f7e73c9/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-8/

Re: #KCIDB engagement report

[Nikolai Kondrashov]

Guillaume,

I checked the database, and the submission queue logs and could only find
the build itself, but no tests. They were probably lost somewhere before
KCIDB.

Nick
P.S. Sorry Debian broke Thunderbird with GMail, so I have to use WebUI and
my messages might be extra ugly.

On Fri, 11 Jun 2021 at 19:12, Guillaume Tucker <
guillaume.tucker@collabora.com> wrote:

> Hi Kees,
>
> On 01/06/2021 21:26, Kees Cook wrote:
> > On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang
> Built Linux wrote:
> >> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
> >> <Nikolai.Kondrashov@redhat.com> wrote:
> >>> [...]
> >>>      KernelCI native
> >>>          Sending (a lot of) production build and test results.
> >>>          https://staging.kernelci.org:3000/?var-origin=kernelci
> >>> [...]
> >
> > Apologies for the thread hijack, but does anyone know what's happening
> > with kselftest? It seems missing from the listed[1] build artifacts, but
> > it is actually present[2] (and I see the logs for generating the tarball
> > there too), but I can't find any builds that actually run the tests?
> >
> > (Or how do I see a top-level list of all tests and search it?)
>
> The kselftest results are all there on the KernelCI native
> dashboard, for example the futex tests:
>
>
> https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.13-rc5-74-g06af8679449d/plan/kselftest-futex/
>
>
> Here's a set of passing results on a "coral" x86 Chromebook, with
> a bunch of unknowns but that's because other kselftests are being
> run when they shouldn't (net, mqueue, ptrace) so it's noise which
> should get resolved with a fix soon:
>
>   https://linux.kernelci.org/test/plan/id/60c2bf67ed48b86ffe0c0df8/
>
>
> And here are the full kernel build details:
>
>   https://linux.kernelci.org/build/id/60c2bdeea60229633d0c0f0c/
>
> and artifacts (logs, binaries, meta-data in JSON):
>
>
> https://storage.kernelci.org/mainline/master/v5.13-rc5-74-g06af8679449d/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-8/
>
>
> So this is the original data, now let's look at what we have in
> KCIDB.  Here's the matching build:
>
>
> https://kcidb.kernelci.org/d/build/build?orgId=1&var-dataset=kernelci04&var-id=kernelci:kernelci.org:60c2bdeea60229633d0c0f0c
>
> However there's no results, probably because submitting the data
> failed for some reason.  It could be due to some invalid
> characters in the test names.  The Log link works though, it
> takes you to the directory with all the log files - to be
> improved as it's advertised as a single build log...
>
> So we'll take a closer look, see if there were any errors in the
> logs to find out why the results aren't in KCIDB.  But the
> kselftests were definitely run.
>
>
> Thanks,
> Guillaume
>
> > [1]
> https://kcidb.kernelci.org/d/build/build?orgId=1&var-dataset=kernelci04&var-id=kernelci:kernelci.org:60b654321456eb7654b3afa6&fullscreen&panelId=17
> > [2]
> https://storage.kernelci.org//mainline/master/v5.13-rc4-11-gc2131f7e73c9/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-8/
>
>
> 
>
>
>

Re: #KCIDB engagement report

[Guillaume Tucker]

On 15/06/2021 10:58, Nikolai Kondrashov wrote:
> Guillaume,
> 
> I checked the database, and the submission queue logs and could only find the build itself, but no tests. They were probably lost somewhere before KCIDB.

Right, I think this is at least part of the issue:

  ValidationError: 'kselftest-futex.breakpoints:step_after_suspend_test' does not match '^[.a-zA-Z0-9_-]*$'

Michał, is this going to get resolved with the fix you're making
for invalid characters in test case names?

Cheers,
Guillaume

> On Fri, 11 Jun 2021 at 19:12, Guillaume Tucker <guillaume.tucker@collabora.com <mailto:guillaume.tucker@collabora.com>> wrote:
> 
>     Hi Kees,
> 
>     On 01/06/2021 21:26, Kees Cook wrote:
>     > On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang Built Linux wrote:
>     >> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
>     >> <Nikolai.Kondrashov@redhat.com <mailto:Nikolai.Kondrashov@redhat.com>> wrote:
>     >>> [...]
>     >>>      KernelCI native
>     >>>          Sending (a lot of) production build and test results.
>     >>>          https://staging.kernelci.org:3000/?var-origin=kernelci
>     >>> [...]
>     >
>     > Apologies for the thread hijack, but does anyone know what's happening
>     > with kselftest? It seems missing from the listed[1] build artifacts, but
>     > it is actually present[2] (and I see the logs for generating the tarball
>     > there too), but I can't find any builds that actually run the tests?
>     >
>     > (Or how do I see a top-level list of all tests and search it?)
> 
>     The kselftest results are all there on the KernelCI native
>     dashboard, for example the futex tests:
> 
>       https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.13-rc5-74-g06af8679449d/plan/kselftest-futex/
> 
> 
>     Here's a set of passing results on a "coral" x86 Chromebook, with
>     a bunch of unknowns but that's because other kselftests are being
>     run when they shouldn't (net, mqueue, ptrace) so it's noise which
>     should get resolved with a fix soon:
> 
>       https://linux.kernelci.org/test/plan/id/60c2bf67ed48b86ffe0c0df8/
> 
> 
>     And here are the full kernel build details:
> 
>       https://linux.kernelci.org/build/id/60c2bdeea60229633d0c0f0c/
> 
>     and artifacts (logs, binaries, meta-data in JSON):
> 
>       https://storage.kernelci.org/mainline/master/v5.13-rc5-74-g06af8679449d/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-8/
> 
> 
>     So this is the original data, now let's look at what we have in
>     KCIDB.  Here's the matching build:
> 
>       https://kcidb.kernelci.org/d/build/build?orgId=1&var-dataset=kernelci04&var-id=kernelci:kernelci.org:60c2bdeea60229633d0c0f0c
> 
>     However there's no results, probably because submitting the data
>     failed for some reason.  It could be due to some invalid
>     characters in the test names.  The Log link works though, it
>     takes you to the directory with all the log files - to be
>     improved as it's advertised as a single build log...
> 
>     So we'll take a closer look, see if there were any errors in the
>     logs to find out why the results aren't in KCIDB.  But the
>     kselftests were definitely run.
> 
> 
>     Thanks,
>     Guillaume
> 
>     > [1] https://kcidb.kernelci.org/d/build/build?orgId=1&var-dataset=kernelci04&var-id=kernelci:kernelci.org:60b654321456eb7654b3afa6&fullscreen&panelId=17
>     > [2] https://storage.kernelci.org//mainline/master/v5.13-rc4-11-gc2131f7e73c9/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-8/
> 
> 
>     
> 
> 

Re: #KCIDB engagement report

[Kees Cook]

On Fri, Jun 11, 2021 at 05:11:59PM +0100, Guillaume Tucker wrote:
> Hi Kees,
> 
> On 01/06/2021 21:26, Kees Cook wrote:
> > On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang Built Linux wrote:
> >> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
> >> <Nikolai.Kondrashov@redhat.com> wrote:
> >>> [...]
> >>>      KernelCI native
> >>>          Sending (a lot of) production build and test results.
> >>>          https://staging.kernelci.org:3000/?var-origin=kernelci
> >>> [...]
> > 
> > Apologies for the thread hijack, but does anyone know what's happening
> > with kselftest? It seems missing from the listed[1] build artifacts, but
> > it is actually present[2] (and I see the logs for generating the tarball
> > there too), but I can't find any builds that actually run the tests?
> > 
> > (Or how do I see a top-level list of all tests and search it?)
> 
> The kselftest results are all there on the KernelCI native
> dashboard, for example the futex tests:
> 
>   https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.13-rc5-74-g06af8679449d/plan/kselftest-futex/

Thanks for looking at this for me! :)

How do I find the other kselftest stuff? I just see "kselftest-futex"
and "kselftest-filesystem". I was expecting _all_ of the kselftests, but
I can't find them.

(Specifically, I can't find a top-level "list of all test plans")

-Kees

-- 
Kees Cook

Re: #KCIDB engagement report

[Guillaume Tucker]

+alex 

On 15/06/2021 23:03, Kees Cook wrote:
> On Fri, Jun 11, 2021 at 05:11:59PM +0100, Guillaume Tucker wrote:
>> Hi Kees,
>>
>> On 01/06/2021 21:26, Kees Cook wrote:
>>> On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang Built Linux wrote:
>>>> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
>>>> <Nikolai.Kondrashov@redhat.com> wrote:
>>>>> [...]
>>>>>      KernelCI native
>>>>>          Sending (a lot of) production build and test results.
>>>>>          https://staging.kernelci.org:3000/?var-origin=kernelci
>>>>> [...]
>>>
>>> Apologies for the thread hijack, but does anyone know what's happening
>>> with kselftest? It seems missing from the listed[1] build artifacts, but
>>> it is actually present[2] (and I see the logs for generating the tarball
>>> there too), but I can't find any builds that actually run the tests?
>>>
>>> (Or how do I see a top-level list of all tests and search it?)
>>
>> The kselftest results are all there on the KernelCI native
>> dashboard, for example the futex tests:
>>
>>   https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.13-rc5-74-g06af8679449d/plan/kselftest-futex/
> 
> Thanks for looking at this for me! :)
> 
> How do I find the other kselftest stuff? I just see "kselftest-futex"
> and "kselftest-filesystem". I was expecting _all_ of the kselftests, but
> I can't find them.
> 
> (Specifically, I can't find a top-level "list of all test plans")

That's because kselftest is rather large, and we're only enabling
subsets of it one at a time.  As more test labs and more devices
become available, we'll gradually expand coverage.  We might also
choose to have full coverage only on say, linux-next, mainline
and LTS branches but not everywhere to not overload the labs.

To answer your question about "all the tests", well you can look
at any kernel revision to see the tests that were run for it
since it won't be the same for all of them.  Typically,
linux-next has the highest number of tests so here's an example:

  https://linux.kernelci.org/test/job/next/branch/master/kernel/next-20210615/

As you've already found, there are only 3 kselftest subsets
or "collections" being run there at the moment.  That's by design
in the KernelCI configuration, but at least we have good enough
support for running kselftest now which wasn't completely
trivial to put in place...

There are still a few issues to fix, but I would expect kselftest
coverage to keep growing over the coming weeks.

If there are kselftest collections you really want to have
enabled, you can always make a PR to add them to this file:

  https://github.com/kernelci/kernelci-core/blob/main/config/core/test-configs.yaml#L187

As long as there's capacity for it at least on some types of
devices and it runs as expected, we should be able to get this
deployed in production pretty easily.


Thanks,
Guillaume

Re: #KCIDB engagement report

[Kees Cook]

On Tue, Jun 15, 2021 at 11:23:35PM +0100, Guillaume Tucker wrote:
> +alex 
> 
> On 15/06/2021 23:03, Kees Cook wrote:
> > On Fri, Jun 11, 2021 at 05:11:59PM +0100, Guillaume Tucker wrote:
> >> Hi Kees,
> >>
> >> On 01/06/2021 21:26, Kees Cook wrote:
> >>> On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang Built Linux wrote:
> >>>> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
> >>>> <Nikolai.Kondrashov@redhat.com> wrote:
> >>>>> [...]
> >>>>>      KernelCI native
> >>>>>          Sending (a lot of) production build and test results.
> >>>>>          https://staging.kernelci.org:3000/?var-origin=kernelci
> >>>>> [...]
> >>>
> >>> Apologies for the thread hijack, but does anyone know what's happening
> >>> with kselftest? It seems missing from the listed[1] build artifacts, but
> >>> it is actually present[2] (and I see the logs for generating the tarball
> >>> there too), but I can't find any builds that actually run the tests?
> >>>
> >>> (Or how do I see a top-level list of all tests and search it?)
> >>
> >> The kselftest results are all there on the KernelCI native
> >> dashboard, for example the futex tests:
> >>
> >>   https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.13-rc5-74-g06af8679449d/plan/kselftest-futex/
> > 
> > Thanks for looking at this for me! :)
> > 
> > How do I find the other kselftest stuff? I just see "kselftest-futex"
> > and "kselftest-filesystem". I was expecting _all_ of the kselftests, but
> > I can't find them.
> > 
> > (Specifically, I can't find a top-level "list of all test plans")
> 
> That's because kselftest is rather large, and we're only enabling
> subsets of it one at a time.  As more test labs and more devices

Ah-ha! Okay.

> become available, we'll gradually expand coverage.  We might also
> choose to have full coverage only on say, linux-next, mainline
> and LTS branches but not everywhere to not overload the labs.

Doing this for -next, mainline, and LTS would be extremely helpful for
me, though I suppose I mostly only care about the lkdtm, seccomp, and
pstore tests.

> To answer your question about "all the tests", well you can look
> at any kernel revision to see the tests that were run for it
> since it won't be the same for all of them.  Typically,
> linux-next has the highest number of tests so here's an example:
> 
>   https://linux.kernelci.org/test/job/next/branch/master/kernel/next-20210615/

Right, that's helpful, but I need to know which kernel to test. It'd be
nice to have a top-level "all the tests", and for each test, it should
list the kernels that run those tests, etc.

> As you've already found, there are only 3 kselftest subsets
> or "collections" being run there at the moment.  That's by design
> in the KernelCI configuration, but at least we have good enough
> support for running kselftest now which wasn't completely
> trivial to put in place...

Right, totally understood. I spent time tweaking those pieces too. :)

> There are still a few issues to fix, but I would expect kselftest
> coverage to keep growing over the coming weeks.
> 
> If there are kselftest collections you really want to have
> enabled, you can always make a PR to add them to this file:
> 
>   https://github.com/kernelci/kernelci-core/blob/main/config/core/test-configs.yaml#L187
> 
> As long as there's capacity for it at least on some types of
> devices and it runs as expected, we should be able to get this
> deployed in production pretty easily.

Awesome. I will do so immediately. :)

Thanks!

-Kees

-- 
Kees Cook

Re: #KCIDB engagement report

[Guillaume Tucker]

+collabora

On 16/06/2021 00:02, Kees Cook wrote:
> On Tue, Jun 15, 2021 at 11:23:35PM +0100, Guillaume Tucker wrote:
>> +alex 
>>
>> On 15/06/2021 23:03, Kees Cook wrote:
>>> On Fri, Jun 11, 2021 at 05:11:59PM +0100, Guillaume Tucker wrote:
>>>> Hi Kees,
>>>>
>>>> On 01/06/2021 21:26, Kees Cook wrote:
>>>>> On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang Built Linux wrote:
>>>>>> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
>>>>>> <Nikolai.Kondrashov@redhat.com> wrote:
>>>>>>> [...]
>>>>>>>      KernelCI native
>>>>>>>          Sending (a lot of) production build and test results.
>>>>>>>          https://staging.kernelci.org:3000/?var-origin=kernelci
>>>>>>> [...]
>>>>>
>>>>> Apologies for the thread hijack, but does anyone know what's happening
>>>>> with kselftest? It seems missing from the listed[1] build artifacts, but
>>>>> it is actually present[2] (and I see the logs for generating the tarball
>>>>> there too), but I can't find any builds that actually run the tests?
>>>>>
>>>>> (Or how do I see a top-level list of all tests and search it?)
>>>>
>>>> The kselftest results are all there on the KernelCI native
>>>> dashboard, for example the futex tests:
>>>>
>>>>   https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.13-rc5-74-g06af8679449d/plan/kselftest-futex/
>>>
>>> Thanks for looking at this for me! :)
>>>
>>> How do I find the other kselftest stuff? I just see "kselftest-futex"
>>> and "kselftest-filesystem". I was expecting _all_ of the kselftests, but
>>> I can't find them.
>>>
>>> (Specifically, I can't find a top-level "list of all test plans")
>>
>> That's because kselftest is rather large, and we're only enabling
>> subsets of it one at a time.  As more test labs and more devices
> 
> Ah-ha! Okay.
> 
>> become available, we'll gradually expand coverage.  We might also
>> choose to have full coverage only on say, linux-next, mainline
>> and LTS branches but not everywhere to not overload the labs.
> 
> Doing this for -next, mainline, and LTS would be extremely helpful for
> me, though I suppose I mostly only care about the lkdtm, seccomp, and
> pstore tests.
> 
>> To answer your question about "all the tests", well you can look
>> at any kernel revision to see the tests that were run for it
>> since it won't be the same for all of them.  Typically,
>> linux-next has the highest number of tests so here's an example:
>>
>>   https://linux.kernelci.org/test/job/next/branch/master/kernel/next-20210615/
> 
> Right, that's helpful, but I need to know which kernel to test. It'd be
> nice to have a top-level "all the tests", and for each test, it should
> list the kernels that run those tests, etc.
> 
>> As you've already found, there are only 3 kselftest subsets
>> or "collections" being run there at the moment.  That's by design
>> in the KernelCI configuration, but at least we have good enough
>> support for running kselftest now which wasn't completely
>> trivial to put in place...
> 
> Right, totally understood. I spent time tweaking those pieces too. :)
> 
>> There are still a few issues to fix, but I would expect kselftest
>> coverage to keep growing over the coming weeks.
>>
>> If there are kselftest collections you really want to have
>> enabled, you can always make a PR to add them to this file:
>>
>>   https://github.com/kernelci/kernelci-core/blob/main/config/core/test-configs.yaml#L187
>>
>> As long as there's capacity for it at least on some types of
>> devices and it runs as expected, we should be able to get this
>> deployed in production pretty easily.
> 
> Awesome. I will do so immediately. :)

Closing the loop here, it's now all enabled in production.
Thanks Kees for all the patches both in KernelCI and kselftest.

Here's some sample results on mainline:

  lkdtm https://linux.kernelci.org/test/plan/id/60dbfb7de0e18e28fc23bc03/
  seccomp https://linux.kernelci.org/test/plan/id/60dbfbe2a9a5def16e23bbeb/


As a bonus, here's a regression already on linux-next:

  https://linux.kernelci.org/test/case/id/60db556ec143e8c85323bbf6/

It's passing with next-20210628:

19:26:49.968767  # selftests: lkdtm: READ_AFTER_FREE.sh
19:26:49.978731  # [   40.808124] lkdtm: Performing d[   41.274300] lkdtm: Performing direct entry SLAB_INIT_ON_ALLOC
19:26:49.982030  irect entry READ_AFTER_FREE
19:26:49.985157  # [   40.813688] lkdtm: Value in memory before free: 12345678
19:26:49.991294  # [   40.841184] lkdtm: Attempting bad read from freed memory
19:26:49.995147  # [   40.868690] lkdtm: Memory correctly poisoned (0)

Full log:  https://storage.kernelci.org/next/master/next-20210628/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-8/lab-collabora/kselftest-lkdtm-hp-11A-G6-EE-grunt.html#L3880

And failing with next-20210629:

17:15:39.454516  # selftests: lkdtm: READ_AFTER_FREE.sh
17:15:39.458520  # [   55.832953] lkdtm: Performing direct entry READ_AFTER_FREE
17:15:39.462522  # [   55.852501] lkdtm: Value in memory before free: 12345678
17:15:39.470520  # [   55.879964] lkdtm: Attempting bad read from freed memory
17:15:39.474530  # [   55.907455] lkdtm: FAIL: Memory was not poisoned!
17:15:39.490501  # [   55.934343] lkdtm: This is probably expected, since this kernel was built *without* CONFIG_INIT_ON_FREE_DEFAULT_ON=y (and booted without 'init_on_free' specified)
17:15:39.498502  # READ_AFTER_FREE: missing 'call trace:|Memory correctly poisoned': [FAIL]

Full log: https://storage.kernelci.org/next/master/next-20210629/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-8/lab-collabora/kselftest-lkdtm-hp-11A-G6-EE-grunt.html#L3879

Does this look legit?

I haven't checked if there was a patch to actually disable
CONFIG_INIT_ON_FREE_DEFAULT_ON=y and no automated bisection has
been run yet.  I'll share any results we may get.

Thanks,
Guillaume

Re: #KCIDB engagement report

[Kees Cook]

On Wed, Jun 30, 2021 at 09:54:31AM +0100, Guillaume Tucker wrote:
> +collabora
> 
> On 16/06/2021 00:02, Kees Cook wrote:
> > On Tue, Jun 15, 2021 at 11:23:35PM +0100, Guillaume Tucker wrote:
> >> +alex 
> >>
> >> On 15/06/2021 23:03, Kees Cook wrote:
> >>> On Fri, Jun 11, 2021 at 05:11:59PM +0100, Guillaume Tucker wrote:
> >>>> Hi Kees,
> >>>>
> >>>> On 01/06/2021 21:26, Kees Cook wrote:
> >>>>> On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang Built Linux wrote:
> >>>>>> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
> >>>>>> <Nikolai.Kondrashov@redhat.com> wrote:
> >>>>>>> [...]
> >>>>>>>      KernelCI native
> >>>>>>>          Sending (a lot of) production build and test results.
> >>>>>>>          https://staging.kernelci.org:3000/?var-origin=kernelci
> >>>>>>> [...]
> >>>>>
> >>>>> Apologies for the thread hijack, but does anyone know what's happening
> >>>>> with kselftest? It seems missing from the listed[1] build artifacts, but
> >>>>> it is actually present[2] (and I see the logs for generating the tarball
> >>>>> there too), but I can't find any builds that actually run the tests?
> >>>>>
> >>>>> (Or how do I see a top-level list of all tests and search it?)
> >>>>
> >>>> The kselftest results are all there on the KernelCI native
> >>>> dashboard, for example the futex tests:
> >>>>
> >>>>   https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.13-rc5-74-g06af8679449d/plan/kselftest-futex/
> >>>
> >>> Thanks for looking at this for me! :)
> >>>
> >>> How do I find the other kselftest stuff? I just see "kselftest-futex"
> >>> and "kselftest-filesystem". I was expecting _all_ of the kselftests, but
> >>> I can't find them.
> >>>
> >>> (Specifically, I can't find a top-level "list of all test plans")
> >>
> >> That's because kselftest is rather large, and we're only enabling
> >> subsets of it one at a time.  As more test labs and more devices
> > 
> > Ah-ha! Okay.
> > 
> >> become available, we'll gradually expand coverage.  We might also
> >> choose to have full coverage only on say, linux-next, mainline
> >> and LTS branches but not everywhere to not overload the labs.
> > 
> > Doing this for -next, mainline, and LTS would be extremely helpful for
> > me, though I suppose I mostly only care about the lkdtm, seccomp, and
> > pstore tests.
> > 
> >> To answer your question about "all the tests", well you can look
> >> at any kernel revision to see the tests that were run for it
> >> since it won't be the same for all of them.  Typically,
> >> linux-next has the highest number of tests so here's an example:
> >>
> >>   https://linux.kernelci.org/test/job/next/branch/master/kernel/next-20210615/
> > 
> > Right, that's helpful, but I need to know which kernel to test. It'd be
> > nice to have a top-level "all the tests", and for each test, it should
> > list the kernels that run those tests, etc.
> > 
> >> As you've already found, there are only 3 kselftest subsets
> >> or "collections" being run there at the moment.  That's by design
> >> in the KernelCI configuration, but at least we have good enough
> >> support for running kselftest now which wasn't completely
> >> trivial to put in place...
> > 
> > Right, totally understood. I spent time tweaking those pieces too. :)
> > 
> >> There are still a few issues to fix, but I would expect kselftest
> >> coverage to keep growing over the coming weeks.
> >>
> >> If there are kselftest collections you really want to have
> >> enabled, you can always make a PR to add them to this file:
> >>
> >>   https://github.com/kernelci/kernelci-core/blob/main/config/core/test-configs.yaml#L187
> >>
> >> As long as there's capacity for it at least on some types of
> >> devices and it runs as expected, we should be able to get this
> >> deployed in production pretty easily.
> > 
> > Awesome. I will do so immediately. :)
> 
> Closing the loop here, it's now all enabled in production.
> Thanks Kees for all the patches both in KernelCI and kselftest.
> 
> Here's some sample results on mainline:
> 
>   lkdtm https://linux.kernelci.org/test/plan/id/60dbfb7de0e18e28fc23bc03/
>   seccomp https://linux.kernelci.org/test/plan/id/60dbfbe2a9a5def16e23bbeb/
> 
> 
> As a bonus, here's a regression already on linux-next:
> 
>   https://linux.kernelci.org/test/case/id/60db556ec143e8c85323bbf6/
> 
> It's passing with next-20210628:
> 
> 19:26:49.968767  # selftests: lkdtm: READ_AFTER_FREE.sh
> 19:26:49.978731  # [   40.808124] lkdtm: Performing d[   41.274300] lkdtm: Performing direct entry SLAB_INIT_ON_ALLOC
> 19:26:49.982030  irect entry READ_AFTER_FREE
> 19:26:49.985157  # [   40.813688] lkdtm: Value in memory before free: 12345678
> 19:26:49.991294  # [   40.841184] lkdtm: Attempting bad read from freed memory
> 19:26:49.995147  # [   40.868690] lkdtm: Memory correctly poisoned (0)
> 
> Full log:  https://storage.kernelci.org/next/master/next-20210628/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-8/lab-collabora/kselftest-lkdtm-hp-11A-G6-EE-grunt.html#L3880
> 
> And failing with next-20210629:
> 
> 17:15:39.454516  # selftests: lkdtm: READ_AFTER_FREE.sh
> 17:15:39.458520  # [   55.832953] lkdtm: Performing direct entry READ_AFTER_FREE
> 17:15:39.462522  # [   55.852501] lkdtm: Value in memory before free: 12345678
> 17:15:39.470520  # [   55.879964] lkdtm: Attempting bad read from freed memory
> 17:15:39.474530  # [   55.907455] lkdtm: FAIL: Memory was not poisoned!
> 17:15:39.490501  # [   55.934343] lkdtm: This is probably expected, since this kernel was built *without* CONFIG_INIT_ON_FREE_DEFAULT_ON=y (and booted without 'init_on_free' specified)
> 17:15:39.498502  # READ_AFTER_FREE: missing 'call trace:|Memory correctly poisoned': [FAIL]
> 
> Full log: https://storage.kernelci.org/next/master/next-20210629/x86_64/x86_64_defconfig+x86-chromebook+kselftest/gcc-8/lab-collabora/kselftest-lkdtm-hp-11A-G6-EE-grunt.html#L3879
> 
> Does this look legit?
> 
> I haven't checked if there was a patch to actually disable
> CONFIG_INIT_ON_FREE_DEFAULT_ON=y and no automated bisection has
> been run yet.  I'll share any results we may get.

Right -- this is probably a CONFIG regression, rather than a kernel code
regression, but yeah, it's quite nice to have this all visible now!

I've got three more changes ready, but I'm waiting for the merge window
to end before I send them for linux-next:

https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git/log/?h=for-next/lkdtm

Thank you again for all your help with this!

Some ideas for future improvements that I might try poking at if someone
doesn't beat me to it:

- Find a way to separate test output from dmesg output (so the interleaved
  lines don't make log reading so hard any more).

- Have the Web UI for a specific test show just that test's output
  (instead of just having a link to the entire boot log).

- Have a way to do side-by-side comparisons across kernel versions and/or
  architectures, so there's an easy way to have a URL for a dashboard that
  shows "these tests all pass on x86_64, but arm64 is failing on that one",
  etc. Something that would look like:

	  v5.13-rc7

				x86_64		arm64		s390
	  lkdtm.ARRAY_BOUNDS	 pass           pass		xfail
	  lkdtm.EXEC_RO		 pass		FAIL		pass



-Kees

-- 
Kees Cook

#KCIDB engagement report

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly report on KCIDB* engagement. It lists various CI systems
and their status of engagement with KCIDB, and once we get to that, will list
developer engagement.

Lines with updates are marked with "!".

Main news are KernelCI now sending all production data, which boosted the
average number of reports we're getting every day: over 50 revisions, some 10K
builds, and around 100K tests.

We're also getting closer to a new release and starting reaching developers
with e-mail notifications.

     KernelCI native
!       Sending (a lot of) production build and test results.
!       https://staging.kernelci.org:3000/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://staging.kernelci.org:3000/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://staging.kernelci.org:3000/?var-origin=syzbot

     ARM
         Sending production results.
         Full commit hashes are currently not available, are spoofed, and don't
         match the ones reported by others. To be fixed soon.
         https://staging.kernelci.org:3000/?var-origin=arm

     Sony Fuego
         Internal design in progress.

     Gentoo GKernelCI
         Sending production results.
         Only builds (a few architectures), no configs, no logs, and no tests
         for now, but working on growing contributions.
         https://staging.kernelci.org:3000/?var-origin=gkernelci

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

     Linaro
         Sending (a lot of) Tuxsuite build results to "production" KCIDB.
         https://staging.kernelci.org:3000/?var-origin=tuxsuite

     TuxML
         Initial contact in response to a report.
         There's a plan to approach us and start work in the coming months.

     Yocto Project
         Initial contact in response to a report.
         Would like to start sending build and test results, particularly for
         older kernels. Would like to separate upstream commits from project
         patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

Please respond with corrections or suggestions of other CI systems to contact.

Nick

*KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
  Foundation's KernelCI project:
  https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

#KCIDB engagement report

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly report on KCIDB* engagement. It lists various CI systems
and their status of engagement with KCIDB, and once we get to that, will list
developer engagement.

Lines with updates are marked with "!".

Main news are KernelCI closing in on sending production data, Tuxsuite sending
to production, and GKernelCI building more architectures.

     KernelCI native
!       Improved-throughput code is in the last stages of testing, sending
!       results to "playground" right now. Will start sending production
!       data in a few days.
!       https://staging.kernelci.org:3000/?var-origin=kernelci&var-dataset=playground_kernelci04

     Red Hat CKI
         Sending production results.
         https://staging.kernelci.org:3000/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://staging.kernelci.org:3000/?var-origin=syzbot

     ARM
         Sending production results.
         Full commit hashes are currently not available, are spoofed, and don't
         match the ones reported by others. To be fixed soon.
         https://staging.kernelci.org:3000/?var-origin=arm

     Sony Fuego
         Internal design in progress.

     Gentoo GKernelCI
         Sending production results.
!       Only builds (a few architectures), no configs, no logs, and no tests
!       for now, but working on growing contributions.
         https://staging.kernelci.org:3000/?var-origin=gkernelci

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

     Linaro
!       Sending (a lot of) Tuxsuite build results to "production" KCIDB.
!       https://staging.kernelci.org:3000/?var-origin=tuxsuite

     TuxML
         Initial contact in response to a report.
         There's a plan to approach us and start work in the coming months.

     Yocto Project
         Initial contact in response to a report.
         Would like to start sending build and test results, particularly for
         older kernels. Would like to separate upstream commits from project
         patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

Please respond with corrections or suggestions of other CI systems to contact.

Nick

*KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
  Foundation's KernelCI project:
  https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

#KCIDB engagement report

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly report on KCIDB* engagement. It lists various CI systems
and their status of engagement with KCIDB, and once we get to that, will list
developer engagement.

Lines with updates are marked with "!".

The main news is that Gentoo GKernelCI is now sending data to "production",
Linaro Tuxsuite is sending data to "playground" and is almost ready for
"production", and TuxML and Yocto Project have expressed their interest
in joining.

     KernelCI native
         Sending staging results.
         Transitioning to production results is at 75%
         (KCIDB support done, backend work in progress).
         https://staging.kernelci.org:3000/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://staging.kernelci.org:3000/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://staging.kernelci.org:3000/?var-origin=syzbot

     ARM
         Sending production results to "production" KCIDB.
         Full commit hashes are currently not available, are spoofed, and don't
         match the ones reported by others. To be fixed soon.
         https://staging.kernelci.org:3000/?var-origin=arm

     Sony Fuego
         Internal design in progress.

!   Gentoo GKernelCI
!       Sending production results to "production" KCIDB.
!       Distro patches are nicely separated from upstream commits.
!       Only x86_64 builds, no configs, no logs, and no tests for now, but
!       working on growing contributions.
!       https://staging.kernelci.org:3000/d/home/home?var-origin=gkernelci

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

!   Linaro
!       Preliminary Tuxsuite (https://tuxsuite.com/) build results are flowing
!       into "playground" KCIDB. Almost ready for "production".
!       https://staging.kernelci.org:3000/d/home/home?var-origin=tuxsuite&var-dataset=playground_kernelci04

!   TuxML
!       Initial contact in response to the last report.
!       There's a plan to approach us and start work in the coming months.

!   Yocto Project
!       Initial contact in response to the last report.
!       Would like to start sending build and test results, particularly for
!       older kernels. Would like to separate upstream commits from project
!       patches first: https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

Please respond with corrections or suggestions of other CI systems to contact.

Nick

*KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
  Foundation's KernelCI project:
  https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/