KCIDB engagement report #kcidb

KCIDB engagement report #kcidb

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly report on KCIDB* engagement. It lists various CI systems
and their status of engagement with KCIDB, and once we get to that, will list
developer engagement.

Lines with updates are marked with "!".

The main news is that Gentoo GKernelCI have started sending their preliminary
data to our "playground" setup. Linaro have also started work on sending
tuxmake build results, but no reports have arrived yet.

     KernelCI native
         Sending staging results.
         Transitioning to production results is at 50%
         (KCIDB support done, backend work started).
         https://staging.kernelci.org:3000/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://staging.kernelci.org:3000/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://staging.kernelci.org:3000/?var-origin=syzbot

     ARM
         Sending production results to "production" KCIDB.
         Full commit hashes are currently not available, are spoofed, and don't
         match the ones reported by others. To be fixed soon.
         https://staging.kernelci.org:3000/?var-origin=arm

     Sony Fuego
         Internal design in progress.

!   Gentoo GKernelCI
!       Preliminary data (revisions and builds) have started flowing into
!       "playground" KCIDB. No timestamps yet.
!       https://staging.kernelci.org:3000/d/home/home?var-origin=gkernelci&var-dataset=playground_kernelci04

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

!   Linaro
!       Work has started on sending tuxmake build results
!       (a part of https://tuxsuite.com/). No reports have arrived yet.

Please respond with corrections or suggestions of other CI systems to contact.

Nick

*KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
  Foundation's KernelCI project:
  https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/

Re: KCIDB engagement report #kcidb

[Mathieu Acher]

Hi, 

TuxML https://github.com/TuxML/tuxml has a plan to synchronize with KCIDB in the upcoming months. 
Our focus is on testing random configurations of the Linux kernel in the large. 
We have two use cases (at least): 
 * configuration bug prevention/prediction/fixing (when the kernel config. does not build)
 * binary kernel size prediction (for whatever kernel version and compiler) 

We have some data here: https://tuxmlweb.istic.univ-rennes1.fr/
We plan to discuss this in february with KernelCI's team for further refining how we can integrate/interoperate. 

Best, 

--
Dr. Mathieu ACHER, Associate Professor
Univ Rennes, Inria, CNRS, IRISA, France (DiverSE team)
http://www.mathieuacher.com/

----- Mail original -----
> De: "Nikolai Kondrashov" <Nikolai.Kondrashov@redhat.com>
> À: "kernelci" <kernelci@groups.io>, automated-testing@yoctoproject.org
> Envoyé: Jeudi 21 Janvier 2021 11:07:02
> Objet: KCIDB engagement report

> Hi everyone,
> 
> Below is the monthly report on KCIDB* engagement. It lists various CI systems
> and their status of engagement with KCIDB, and once we get to that, will list
> developer engagement.
> 
> Lines with updates are marked with "!".
> 
> The main news is that Gentoo GKernelCI have started sending their preliminary
> data to our "playground" setup. Linaro have also started work on sending
> tuxmake build results, but no reports have arrived yet.
> 
>     KernelCI native
>         Sending staging results.
>         Transitioning to production results is at 50%
>         (KCIDB support done, backend work started).
>         https://staging.kernelci.org:3000/?var-origin=kernelci
> 
>     Red Hat CKI
>         Sending production results.
>         https://staging.kernelci.org:3000/?var-origin=redhat
> 
>     Google Syzbot
>         Sending a subset of production results (failures only).
>         https://staging.kernelci.org:3000/?var-origin=syzbot
> 
>     ARM
>         Sending production results to "production" KCIDB.
>         Full commit hashes are currently not available, are spoofed, and don't
>         match the ones reported by others. To be fixed soon.
>         https://staging.kernelci.org:3000/?var-origin=arm
> 
>     Sony Fuego
>         Internal design in progress.
> 
> !   Gentoo GKernelCI
> !       Preliminary data (revisions and builds) have started flowing into
> !       "playground" KCIDB. No timestamps yet.
> !
> https://staging.kernelci.org:3000/d/home/home?var-origin=gkernelci&var-dataset=playground_kernelci04
> 
>     Intel 0day
>         Initial conversation concluded, general interest expressed,
>         no contact since.
> 
> !   Linaro
> !       Work has started on sending tuxmake build results
> !       (a part of https://tuxsuite.com/). No reports have arrived yet.
> 
> Please respond with corrections or suggestions of other CI systems to contact.
> 
> Nick
> 
> *KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
>  Foundation's KernelCI project:
>  https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
> 
> 
> 
> 

Re: KCIDB engagement report #kcidb

[Nikolai Kondrashov]

Hi Mathieu,

On 1/21/21 12:17 PM, Mathieu Acher wrote:
 > TuxML https://github.com/TuxML/tuxml has a plan to synchronize with KCIDB in the upcoming months.

Wonderful, would be happy to help you!
Don't hesitate to contact us on the maillist and on the freenode's #kernelci channel, anytime!

Nick

On 1/21/21 12:17 PM, Mathieu Acher wrote:
> Hi,
> 
> TuxML https://github.com/TuxML/tuxml has a plan to synchronize with KCIDB in the upcoming months.
> Our focus is on testing random configurations of the Linux kernel in the large.
> We have two use cases (at least):
>   * configuration bug prevention/prediction/fixing (when the kernel config. does not build)
>   * binary kernel size prediction (for whatever kernel version and compiler)
> 
> We have some data here: https://tuxmlweb.istic.univ-rennes1.fr/
> We plan to discuss this in february with KernelCI's team for further refining how we can integrate/interoperate.
> 
> Best,
> 
> --
> Dr. Mathieu ACHER, Associate Professor
> Univ Rennes, Inria, CNRS, IRISA, France (DiverSE team)
> http://www.mathieuacher.com/
> 
> ----- Mail original -----
>> De: "Nikolai Kondrashov" <Nikolai.Kondrashov@redhat.com>
>> À: "kernelci" <kernelci@groups.io>, automated-testing@yoctoproject.org
>> Envoyé: Jeudi 21 Janvier 2021 11:07:02
>> Objet: KCIDB engagement report
> 
>> Hi everyone,
>>
>> Below is the monthly report on KCIDB* engagement. It lists various CI systems
>> and their status of engagement with KCIDB, and once we get to that, will list
>> developer engagement.
>>
>> Lines with updates are marked with "!".
>>
>> The main news is that Gentoo GKernelCI have started sending their preliminary
>> data to our "playground" setup. Linaro have also started work on sending
>> tuxmake build results, but no reports have arrived yet.
>>
>>      KernelCI native
>>          Sending staging results.
>>          Transitioning to production results is at 50%
>>          (KCIDB support done, backend work started).
>>          https://staging.kernelci.org:3000/?var-origin=kernelci
>>
>>      Red Hat CKI
>>          Sending production results.
>>          https://staging.kernelci.org:3000/?var-origin=redhat
>>
>>      Google Syzbot
>>          Sending a subset of production results (failures only).
>>          https://staging.kernelci.org:3000/?var-origin=syzbot
>>
>>      ARM
>>          Sending production results to "production" KCIDB.
>>          Full commit hashes are currently not available, are spoofed, and don't
>>          match the ones reported by others. To be fixed soon.
>>          https://staging.kernelci.org:3000/?var-origin=arm
>>
>>      Sony Fuego
>>          Internal design in progress.
>>
>> !   Gentoo GKernelCI
>> !       Preliminary data (revisions and builds) have started flowing into
>> !       "playground" KCIDB. No timestamps yet.
>> !
>> https://staging.kernelci.org:3000/d/home/home?var-origin=gkernelci&var-dataset=playground_kernelci04
>>
>>      Intel 0day
>>          Initial conversation concluded, general interest expressed,
>>          no contact since.
>>
>> !   Linaro
>> !       Work has started on sending tuxmake build results
>> !       (a part of https://tuxsuite.com/). No reports have arrived yet.
>>
>> Please respond with corrections or suggestions of other CI systems to contact.
>>
>> Nick
>>
>> *KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
>>   Foundation's KernelCI project:
>>   https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
>>
>>
>>
>>
> 
> 
> 
> 
> 

Re: [Automated-testing] KCIDB engagement report #kcidb

[Richard Purdie]

On Thu, 2021-01-21 at 12:07 +0200, Nikolai Kondrashov wrote:
> Hi everyone,
> 
> Below is the monthly report on KCIDB* engagement. It lists various CI systems
> and their status of engagement with KCIDB, and once we get to that, will list
> developer engagement.
> 
> Lines with updates are marked with "!".
> 
> The main news is that Gentoo GKernelCI have started sending their preliminary
> data to our "playground" setup. Linaro have also started work on sending
> tuxmake build results, but no reports have arrived yet.
> 
>      KernelCI native
>          Sending staging results.
>          Transitioning to production results is at 50%
>          (KCIDB support done, backend work started).
>          https://staging.kernelci.org:3000/?var-origin=kernelci
> 
>      Red Hat CKI
>          Sending production results.
>          https://staging.kernelci.org:3000/?var-origin=redhat
> 
>      Google Syzbot
>          Sending a subset of production results (failures only).
>          https://staging.kernelci.org:3000/?var-origin=syzbot
> 
>      ARM
>          Sending production results to "production" KCIDB.
>          Full commit hashes are currently not available, are spoofed, and don't
>          match the ones reported by others. To be fixed soon.
>          https://staging.kernelci.org:3000/?var-origin=arm
> 
>      Sony Fuego
>          Internal design in progress.
> 
> !   Gentoo GKernelCI
> !       Preliminary data (revisions and builds) have started flowing into
> !       "playground" KCIDB. No timestamps yet.
> !       https://staging.kernelci.org:3000/d/home/home?var-origin=gkernelci&var-dataset=playground_kernelci04
> 
>      Intel 0day
>          Initial conversation concluded, general interest expressed,
>          no contact since.
> 
> !   Linaro
> !       Work has started on sending tuxmake build results
> !       (a part of https://tuxsuite.com/). No reports have arrived yet.
> 
> Please respond with corrections or suggestions of other CI systems to contact.

I noticed Yocto Project isn't mentioned here but we do a lot of builds
of complete systems for multiple architectures and we include kernels
in those. We probably have two kernel versions under testing in any
given project release series.

Would it be useful to get data from us? If so, what kind of data?

We're generally building linux-yocto from:
https://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto/
which is derived from upstream kernel releases as their baseline.

Testing wise, we build them, then usually boot them under qemu with a
variety of userspaces. We also run ltp, some lsb and the tests from a
variety of projects like lttng.

An example output report we use is:

https://autobuilder.yocto.io/pub/non-release/20210120-9/testresults/testresult-report.txt

So far I'd assumed you were interested in bleeding edge kernel CI but
if there is interest around general kernel testing of older kernels, we
do have a lot of data. There are some details we don't currently log
(e.g. which kernel version in in a given test target) but we can change
that if we need it.

Cheers,

Richard

Re: [Automated-testing] KCIDB engagement report #kcidb

[Nikolai Kondrashov]

Hi Richard,

On 1/21/21 1:01 PM, Richard Purdie wrote:
 > On Thu, 2021-01-21 at 12:07 +0200, Nikolai Kondrashov wrote:
 >> Please respond with corrections or suggestions of other CI systems to contact.
 >
 > I noticed Yocto Project isn't mentioned here but we do a lot of builds
 > of complete systems for multiple architectures and we include kernels
 > in those. We probably have two kernel versions under testing in any
 > given project release series.
 >
 > Would it be useful to get data from us? If so, what kind of data?

Absolutely! Thanks for reaching out :)

 > We're generally building linux-yocto from:
 > https://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto/
 > which is derived from upstream kernel releases as their baseline.
 >
 > Testing wise, we build them, then usually boot them under qemu with a
 > variety of userspaces. We also run ltp, some lsb and the tests from a
 > variety of projects like lttng.
 >
 > An example output report we use is:
 >
 > https://autobuilder.yocto.io/pub/non-release/20210120-9/testresults/testresult-report.txt
 >
 > So far I'd assumed you were interested in bleeding edge kernel CI but
 > if there is interest around general kernel testing of older kernels, we
 > do have a lot of data. There are some details we don't currently log
 > (e.g. which kernel version in in a given test target) but we can change
 > that if we need it.

Wow, that's a lot of tests, very cool!

We would definitely welcome reports for older kernel versions as well.
Any data helping track down issues is welcome.

We could use build/test results, and logs.

It would help if you could separate the upstream commit your branches are
based on and the patches you apply on top, and identify them separately.
However, it's OK if you can't, only mention the base version (e.g. using "git
describe"), and otherwise refer to your own repo and commits directly.

In any case, that would be good enough for the start. We're still in research
stage (and will be for a while), trying to bring the data from various sources
together and do something with it.

Our strategy is get the data first, organize later :)

We would be glad to have you on board and sending whatever data you have.

Please take a look at our Submission HOWTO for an overview of how things work:

     https://github.com/kernelci/kcidb/blob/main/SUBMISSION_HOWTO.md

If you'd like to try it out, I can send you credentials and parameters for
submitting to our "playground" instance. You can submit data there (either
manually or automatically) and experiment freely without worrying about
breaking anything. The data will appear on this dashboard (GKernelCI is
currently submitting there):

     https://staging.kernelci.org:3000/d/home/home?var-dataset=playground_kernelci04

Once you're confident with how things work, it would only take a permission
tweak on my side and one parameter change on your side to switch to
production.

Don't hesitate to ping us on the maillist,
or on the freenode's #kernelci channel!

Nick

Re: [Automated-testing] KCIDB engagement report

[Richard Purdie]

On Thu, 2021-01-21 at 14:46 +0200, Nikolai Kondrashov wrote:
> Hi Richard,
> 
> On 1/21/21 1:01 PM, Richard Purdie wrote:
>  > On Thu, 2021-01-21 at 12:07 +0200, Nikolai Kondrashov wrote:
>  >> Please respond with corrections or suggestions of other CI systems to contact.
>  >
>  > I noticed Yocto Project isn't mentioned here but we do a lot of builds
>  > of complete systems for multiple architectures and we include kernels
>  > in those. We probably have two kernel versions under testing in any
>  > given project release series.
>  >
>  > Would it be useful to get data from us? If so, what kind of data?
> 
> Absolutely! Thanks for reaching out :)
> 
>  > We're generally building linux-yocto from:
>  > https://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto/
>  > which is derived from upstream kernel releases as their baseline.
>  >
>  > Testing wise, we build them, then usually boot them under qemu with a
>  > variety of userspaces. We also run ltp, some lsb and the tests from a
>  > variety of projects like lttng.
>  >
>  > An example output report we use is:
>  >
>  > https://autobuilder.yocto.io/pub/non-release/20210120-9/testresults/testresult-report.txt
>  >
>  > So far I'd assumed you were interested in bleeding edge kernel CI but
>  > if there is interest around general kernel testing of older kernels, we
>  > do have a lot of data. There are some details we don't currently log
>  > (e.g. which kernel version in in a given test target) but we can change
>  > that if we need it.
> 
> Wow, that's a lot of tests, very cool!
> 
> We would definitely welcome reports for older kernel versions as well.
> Any data helping track down issues is welcome.
> 
> We could use build/test results, and logs.
> 
> It would help if you could separate the upstream commit your branches are
> based on and the patches you apply on top, and identify them separately.
> However, it's OK if you can't, only mention the base version (e.g. using "git
> describe"), and otherwise refer to your own repo and commits directly.
> 
> In any case, that would be good enough for the start. We're still in research
> stage (and will be for a while), trying to bring the data from various sources
> together and do something with it.
> 
> Our strategy is get the data first, organize later :)
> 
> We would be glad to have you on board and sending whatever data you have.
> 
> Please take a look at our Submission HOWTO for an overview of how things work:
> 
>      https://github.com/kernelci/kcidb/blob/main/SUBMISSION_HOWTO.md
> 
> If you'd like to try it out, I can send you credentials and parameters for
> submitting to our "playground" instance. You can submit data there (either
> manually or automatically) and experiment freely without worrying about
> breaking anything. The data will appear on this dashboard (GKernelCI is
> currently submitting there):
> 
>      https://staging.kernelci.org:3000/d/home/home?var-dataset=playground_kernelci04
> 
> Once you're confident with how things work, it would only take a permission
> tweak on my side and one parameter change on your side to switch to
> production.
> 
> Don't hesitate to ping us on the maillist,
> or on the freenode's #kernelci channel!

Thanks, that is all good info. The biggest issue right now is we don't
record the kernel version information anywhere that our test results
processing is going to be able to find it. That isn't too hard to fix
but there are a few systems involved.

I'm not sure when or who will have the time to make this happen but I
do like the idea and don't want it to get lost. I put an entry into our
bug/feature tracker:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

Cheers,

Richard

Re: [Automated-testing] KCIDB engagement report

[Nikolai Kondrashov]

Hi Richard,

On 1/26/21 5:10 PM, Richard Purdie wrote:
 > Thanks, that is all good info. The biggest issue right now is we don't
 > record the kernel version information anywhere that our test results
 > processing is going to be able to find it. That isn't too hard to fix
 > but there are a few systems involved.
 >
 > I'm not sure when or who will have the time to make this happen but I
 > do like the idea and don't want it to get lost. I put an entry into our
 > bug/feature tracker:
 >
 > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196

Great! Don't hesitate to write if you'd like to discuss something, and feel free to
join either one of the monthly "Automated Testing" conference calls,
or a weekly KernelCI call, if you'd like a quick high-bandwidth chat :)

Nick

On 1/26/21 5:10 PM, Richard Purdie wrote:
> On Thu, 2021-01-21 at 14:46 +0200, Nikolai Kondrashov wrote:
>> Hi Richard,
>>
>> On 1/21/21 1:01 PM, Richard Purdie wrote:
>>   > On Thu, 2021-01-21 at 12:07 +0200, Nikolai Kondrashov wrote:
>>   >> Please respond with corrections or suggestions of other CI systems to contact.
>>   >
>>   > I noticed Yocto Project isn't mentioned here but we do a lot of builds
>>   > of complete systems for multiple architectures and we include kernels
>>   > in those. We probably have two kernel versions under testing in any
>>   > given project release series.
>>   >
>>   > Would it be useful to get data from us? If so, what kind of data?
>>
>> Absolutely! Thanks for reaching out :)
>>
>>   > We're generally building linux-yocto from:
>>   > https://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto/
>>   > which is derived from upstream kernel releases as their baseline.
>>   >
>>   > Testing wise, we build them, then usually boot them under qemu with a
>>   > variety of userspaces. We also run ltp, some lsb and the tests from a
>>   > variety of projects like lttng.
>>   >
>>   > An example output report we use is:
>>   >
>>   > https://autobuilder.yocto.io/pub/non-release/20210120-9/testresults/testresult-report.txt
>>   >
>>   > So far I'd assumed you were interested in bleeding edge kernel CI but
>>   > if there is interest around general kernel testing of older kernels, we
>>   > do have a lot of data. There are some details we don't currently log
>>   > (e.g. which kernel version in in a given test target) but we can change
>>   > that if we need it.
>>
>> Wow, that's a lot of tests, very cool!
>>
>> We would definitely welcome reports for older kernel versions as well.
>> Any data helping track down issues is welcome.
>>
>> We could use build/test results, and logs.
>>
>> It would help if you could separate the upstream commit your branches are
>> based on and the patches you apply on top, and identify them separately.
>> However, it's OK if you can't, only mention the base version (e.g. using "git
>> describe"), and otherwise refer to your own repo and commits directly.
>>
>> In any case, that would be good enough for the start. We're still in research
>> stage (and will be for a while), trying to bring the data from various sources
>> together and do something with it.
>>
>> Our strategy is get the data first, organize later :)
>>
>> We would be glad to have you on board and sending whatever data you have.
>>
>> Please take a look at our Submission HOWTO for an overview of how things work:
>>
>>       https://github.com/kernelci/kcidb/blob/main/SUBMISSION_HOWTO.md
>>
>> If you'd like to try it out, I can send you credentials and parameters for
>> submitting to our "playground" instance. You can submit data there (either
>> manually or automatically) and experiment freely without worrying about
>> breaking anything. The data will appear on this dashboard (GKernelCI is
>> currently submitting there):
>>
>>       https://staging.kernelci.org:3000/d/home/home?var-dataset=playground_kernelci04
>>
>> Once you're confident with how things work, it would only take a permission
>> tweak on my side and one parameter change on your side to switch to
>> production.
>>
>> Don't hesitate to ping us on the maillist,
>> or on the freenode's #kernelci channel!
> 
> Thanks, that is all good info. The biggest issue right now is we don't
> record the kernel version information anywhere that our test results
> processing is going to be able to find it. That isn't too hard to fix
> but there are a few systems involved.
> 
> I'm not sure when or who will have the time to make this happen but I
> do like the idea and don't want it to get lost. I put an entry into our
> bug/feature tracker:
> 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14196
> 
> Cheers,
> 
> Richard
> 

KCIDB engagement report #kcidb

[Nikolai Kondrashov]

Hi everyone,

Below is the monthly report on KCIDB* engagement. It lists various CI systems
and their status of engagement with KCIDB, and once we get to that, will list
developer engagement.

Lines with updates are marked with "!".

The main news is that ARM are now sending their production data!

     KernelCI native
         Sending staging results.
         Transitioning to production results is at 50%
         (KCIDB support done, backend work started).
         https://staging.kernelci.org:3000/?var-origin=kernelci

     Red Hat CKI
         Sending production results.
         https://staging.kernelci.org:3000/?var-origin=redhat

     Google Syzbot
         Sending a subset of production results (failures only).
         https://staging.kernelci.org:3000/?var-origin=syzbot

     ARM
!       Sending production results to "production" KCIDB.
!       Full commit hashes are currently not available, are spoofed, and don't
!       match the ones reported by others. To be fixed soon.
!       https://staging.kernelci.org:3000/?var-origin=arm

     Sony Fuego
         Internal design in progress.

     Gentoo GKernelCI
         Prototype work has started.
         Gentoo is in a CI system transition, and progress is slow.

     Intel 0day
         Initial conversation concluded, general interest expressed,
         no contact since.

     Google OSS-Fuzz
!       Not really suited for kernel testing, although QEMU virtual device
!       fuzzing they run could be used as a kernel test.
!       Dropping Google OSS-Fuzz in the next report.

     Linaro LKFT
         Participated in KCIDB kick-off, no contact regarding reporting since.

Please respond with corrections or suggestions of other CI systems to contact.

Nick

*KCIDB is an effort to unify Linux Kernel CI reporting, maintained by Linux
  Foundation's KernelCI project:
  https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/