Investigating where to put KUnit in the KernelCI build pipeline

Investigating where to put KUnit in the KernelCI build pipeline

[Brendan Higgins]

# Recapping discussion at OSS Europe:

At OSS Europe Kevin and I talked about how to integrate KUnit into KernelCI.

It sounded like there where two distinct cases to consider:

1) Including KUnit as a stage in the normal KernelCI build and test process.

2) Figuring out presubmit for KUnit/KernelCI.

As the latter case requires a lot more work/thought/discussion, I will
focus on just the first in this email.

For integrating KUnit into the KernelCI build and test process, Kevin
suggested that we should treat KUnit like a "static analysis step"
that would run within the build pipeline before the install/test
phase.

I know that "static analysis step/pipeline" doesn't exist yet, but
Kevin suggested that KUnit could be the first prototype "static
analysis" thing and could establish the pattern that subsequent static
analysis tools would implement.

# My initial idea on how to add KUnit to KernelCI

Based on my initial investigation, it seems that this KUnit step would
probably fit somewhere in-between the "build-trigger[1]" and
"build[2]" nodes. Does this make sense? (Forgive me, I am new to
Jenkins pipelines, and the KernelCI infrastructure in general.)

It seems like it might make sense to add a new pipeline file for
"static analysis" type things, does this seem reasonable?

Thanks!

[1] https://github.com/kernelci/kernelci-core/blob/master/jenkins/build-trigger.jpl
[2] https://github.com/kernelci/kernelci-core/blob/master/jenkins/build.jpl

Re: Investigating where to put KUnit in the KernelCI build pipeline

[Guillaume Tucker]

[-- Attachment #1: Type: text/plain, Size: 4217 bytes --]

On Tue, Nov 5, 2019 at 12:40 AM Brendan Higgins via Groups.Io
<brendanhiggins=google.com@groups.io> wrote:

> # Recapping discussion at OSS Europe:
>
> At OSS Europe Kevin and I talked about how to integrate KUnit into
> KernelCI.
>
> It sounded like there where two distinct cases to consider:
>
> 1) Including KUnit as a stage in the normal KernelCI build and test
> process.
>
> 2) Figuring out presubmit for KUnit/KernelCI.
>

What do you mean by "presumbit"?  Testing patches from mailing
lists before they get applied to git branches?


> As the latter case requires a lot more work/thought/discussion, I will
> focus on just the first in this email.
>
> For integrating KUnit into the KernelCI build and test process, Kevin
> suggested that we should treat KUnit like a "static analysis step"
> that would run within the build pipeline before the install/test
> phase.
>
> I know that "static analysis step/pipeline" doesn't exist yet, but
> Kevin suggested that KUnit could be the first prototype "static
> analysis" thing and could establish the pattern that subsequent static
> analysis tools would implement.
>
> # My initial idea on how to add KUnit to KernelCI
>
> Based on my initial investigation, it seems that this KUnit step would
> probably fit somewhere in-between the "build-trigger[1]" and
> "build[2]" nodes. Does this make sense? (Forgive me, I am new to
> Jenkins pipelines, and the KernelCI infrastructure in general.)
>

I think the first step is to be able to run KUnit and send the
results to the kernelci-backend using the command line.  We may
need to add some commands to kci_test for that (i.e. kci_test
send_results).  Then we can look into running it in Jenkins.

While KUnit needs a kernel build, I think it's an implementation
detail and we can treat it as plain static analysis unlike the
other kernel builds.  Would there be any value in pushing the UML
build used by KUnit to the KernelCI storage server like we do
with other builds?

In any case, it seems like we should add a kunit test plan
definition in test-configs.yaml and a new type of lab in
lab-configs.yaml to run static analysis tests.  This could be
reused for Coccinelle or anything that takes a source tarball and
produces some test results.

Then we can decide where to run these static tests, should it be
on Jenkins builders?  We could also use some cloud VMs I guess.
Provided we had some account with credits for that (GCE,
Azure...), it would be easy to implement a kernelci.lab.cloud (or
.gce or .azure...)  module which can start a VM and run a test
there to produce a results file which can then be pushed to the
backend.

The only change we would need to make in Jenkins jobs for this is
to trigger a test-runner.jpl with no kernel build but with the
kernel source tarball URL, to run tests in a "cloud" lab.

If we wanted to run the static analysis tests on Jenkins
builders, we could also have a kernelci.lab.jenkins module which
would kind of loop back into Jenkins - or take some shortcut by
checking the type of lab in a Jenkins job and do something there
directly.  If we go down that route, based on my experience the
best way to work out a solution is to try in practice as it's
very hard to guess what would be the best Jenkins Pipeline job
design on paper.  We have staging.kernelci.org for that.

Then I also think we should try and decouple KernelCI from
Jenkins, and KUnit would seem like a good way of starting to run
jobs in cloud VMs.


> It seems like it might make sense to add a new pipeline file for
> "static analysis" type things, does this seem reasonable?
>

The current KernelCI pipeline and configuration should already be
flexible enough to cover this, and if not I would rather make it
more versatile than append extra corner cases with new pipeline
jobs.  The Jenkins implementation is a bit of a detail, but
KernelCI will need to be able to run more diverse kinds of tests
in the future than just "booting dev boards" and I see this as
one first step in this direction.

Guillaume


> Thanks!
>
> [1]
> https://github.com/kernelci/kernelci-core/blob/master/jenkins/build-trigger.jpl
> [2]
> https://github.com/kernelci/kernelci-core/blob/master/jenkins/build.jpl
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 5764 bytes --]

Re: Investigating where to put KUnit in the KernelCI build pipeline

[Brendan Higgins]

[-- Attachment #1: Type: text/plain, Size: 8936 bytes --]

On Tue, Nov 5, 2019 at 2:06 AM Guillaume Tucker <guillaume.tucker@gmail.com>
wrote:

>
>
> On Tue, Nov 5, 2019 at 12:40 AM Brendan Higgins via Groups.Io
> <brendanhiggins=google.com@groups.io> wrote:
>
>> # Recapping discussion at OSS Europe:
>>
>> At OSS Europe Kevin and I talked about how to integrate KUnit into
>> KernelCI.
>>
>> It sounded like there where two distinct cases to consider:
>>
>> 1) Including KUnit as a stage in the normal KernelCI build and test
>> process.
>>
>> 2) Figuring out presubmit for KUnit/KernelCI.
>>
>
> What do you mean by "presumbit"?  Testing patches from mailing
> lists before they get applied to git branches?
>

Yep. As I understand it, testing patches before they are applied to git
branches is not currently on the roadmap for KernelCI. Nevertheless, this
is something we, KUnit, have an interest in. I have an idea of how to do
this, but I figured it would be best to focus on getting KUnit support into
KernelCI first, but now that I have read your response, it might be good to
share what my longer term plan is for adding support for presubmit.

We already have a system (
https://prow.gflocks.com/job-history/oss-prow/pr-logs/directory/kunit-test-presubmit)
that supports presubmit on Gerrit and GitHub; it's built on an open source
project called Prow (
https://github.com/kubernetes/test-infra/tree/master/prow#readme). We have
an experimental module to allow it to work with patches from LKML. My
thought is, once we can get KUnit running on KernelCI, if you have no plan
to support presubmit, we can just use Prow with the LKML adapter and upload
the test results to KernelCI.

At a very high level, does that make sense? Am I correct in saying that you
have no plans to support presubmit? Does this conflict with anything that
you are planning on doing?

As the latter case requires a lot more work/thought/discussion, I will
>> focus on just the first in this email.
>>
>> For integrating KUnit into the KernelCI build and test process, Kevin
>> suggested that we should treat KUnit like a "static analysis step"
>> that would run within the build pipeline before the install/test
>> phase.
>>
>> I know that "static analysis step/pipeline" doesn't exist yet, but
>> Kevin suggested that KUnit could be the first prototype "static
>> analysis" thing and could establish the pattern that subsequent static
>> analysis tools would implement.
>>
>> # My initial idea on how to add KUnit to KernelCI
>>
>> Based on my initial investigation, it seems that this KUnit step would
>> probably fit somewhere in-between the "build-trigger[1]" and
>> "build[2]" nodes. Does this make sense? (Forgive me, I am new to
>> Jenkins pipelines, and the KernelCI infrastructure in general.)
>>
>
> I think the first step is to be able to run KUnit and send the
> results to the kernelci-backend using the command line.  We may
> need to add some commands to kci_test for that (i.e. kci_test
> send_results).  Then we can look into running it in Jenkins.
>

Makes sense. I imagined the first first step would be to get KUnit to
output the test_group (https://api.kernelci.org/schema-test-group.html)
format used by KernelCI; nevertheless, I think it makes sense to make KUnit
speak the format used by KernelCI rather than having a converter built in
to KernelCI.

After that, yeah, I think adding KUnit to the kernelci-backend using the
command line makes sense. I don't know what is entailed to get kci_test to
send_results, but that seems like it is probably something we will need to
tackle at this stage.

Once we do all that, I imagine we would then want to add it Jenkins, right?
(I know that there is a decent amount of work in both KernelCI and KUnit to
get to this point, but I have someone who will be joining us temporarily
who I am planning on doing this work, so I want to get an idea of what she
will be needing to do before she starts.) Either via the cloud builders as
you mention below, or using what I think you are describing as the existing
method?

While KUnit needs a kernel build, I think it's an implementation
> detail and we can treat it as plain static analysis unlike the
> other kernel builds.
>

I think that is the right approach, and Kevin seemed to also think so when
I talked with him about it.


> Would there be any value in pushing the UML
> build used by KUnit to the KernelCI storage server like we do
> with other builds?
>

Probably not. They are pretty easy to deterministically recreate if you
have the config and the source ref.

In any case, it seems like we should add a kunit test plan
> definition in test-configs.yaml and a new type of lab in
> lab-configs.yaml to run static analysis tests.  This could be
> reused for Coccinelle or anything that takes a source tarball and
> produces some test results.
>

Makes sense. For the labs-config.yaml, we are basically just using it to
provide a name and to tell it what trees to test against, right? Will we
want to provide a new lab_type called "static-analysis" or something like
that?

I think the test-configs.yaml entry would be empty, since there is no
rootfs, no bios, etc. Is that right?

Then we can decide where to run these static tests, should it be
> on Jenkins builders?
>

That's what I was thinking. Most of the cost of running the tests comes
from building the kernel. I did some back of the napkin math, and
determined that it would easily take years to accumulate enough tests for
the amount of tests to consume enough resources to match the amount of
resources consumed just by building the kernel (and I was pretty optimistic
about the rate of adding tests). I can throw up some numbers if you like.


> We could also use some cloud VMs I guess.
> Provided we had some account with credits for that (GCE,
> Azure...), it would be easy to implement a kernelci.lab.cloud (or
> .gce or .azure...)  module which can start a VM and run a test
> there to produce a results file which can then be pushed to the
> backend.
>
> The only change we would need to make in Jenkins jobs for this is
> to trigger a test-runner.jpl with no kernel build but with the
> kernel source tarball URL, to run tests in a "cloud" lab.
>

I am fine with this too. I don't really have strong feelings either way. I
think it comes down to where you want to do static analysis in the future.
The one advantage I see with this approach is it makes it easier for the
person adding support for the static analysis tool to set up a custom
environment (I assume you can provide your own Docker container, no?). I
don't think KUnit will need to take advantage of running our own Docker
container (at least not for a while), but I could see it being useful for
other static analysis tools.

If we wanted to run the static analysis tests on Jenkins
> builders, we could also have a kernelci.lab.jenkins module which
> would kind of loop back into Jenkins - or take some shortcut by
> checking the type of lab in a Jenkins job and do something there
> directly.  If we go down that route, based on my experience the
> best way to work out a solution is to try in practice as it's
> very hard to guess what would be the best Jenkins Pipeline job
> design on paper.  We have staging.kernelci.org for that.
>

Cool. Very good to know.

I imagine you administer it, and you do testing by cutting branches that
you cherry-pick experimental commits on?

Then I also think we should try and decouple KernelCI from
> Jenkins, and KUnit would seem like a good way of starting to run
> jobs in cloud VMs.
>

Can you elaborate on how you are thinking about decoupling KernelCI from
Jenkins? This doesn't sound like something I think we feel strongly about,
unless it helps/hurts the eventual goal of providing presubmit, but it
would nevertheless, be good to know about so we can make sure what we do
helps you with this goal.

It seems like it might make sense to add a new pipeline file for
>> "static analysis" type things, does this seem reasonable?
>>
>
> The current KernelCI pipeline and configuration should already be
> flexible enough to cover this, and if not I would rather make it
> more versatile than append extra corner cases with new pipeline
> jobs.
>

Sure, I don't want to add unnecessary complexity, I just thought that it
looked necessary to add what we wanted. Are you saying that KernelCI
already has a generic notion of a builder and we can just add a config that
makes KUnit a type of builder with no downstream tests? Or am I completely
misunderstanding?


> The Jenkins implementation is a bit of a detail, but
> KernelCI will need to be able to run more diverse kinds of tests
> in the future than just "booting dev boards" and I see this as
> one first step in this direction.
>

Agreed!

Cheers!

Guillaume
>
>
>> Thanks!
>>
>> [1]
>> https://github.com/kernelci/kernelci-core/blob/master/jenkins/build-trigger.jpl
>> [2]
>> https://github.com/kernelci/kernelci-core/blob/master/jenkins/build.jpl
>>
>

[-- Attachment #2: Type: text/html, Size: 13569 bytes --]