Re: Linking CIP testing with KernelCI

* Re: Linking CIP testing with KernelCI
       [not found] <OSAPR01MB2385E0367B8CA545E1D66422B7890@OSAPR01MB2385.jpnprd01.prod.outlook.com>
@ 2020-06-11 21:11 ` Guillaume Tucker
  2020-06-12  7:54   ` santiago.esteban
  2020-06-12 11:03   ` Chris Paterson
  0 siblings, 2 replies; 5+ messages in thread
From: Guillaume Tucker @ 2020-06-11 21:11 UTC (permalink / raw)
  To: Chris Paterson; +Cc: Kevin Hilman, kernelci

[-- Attachment #1: Type: text/plain, Size: 5811 bytes --]

+ kernelci@groups.io as discussed

Hi Chris,

Thanks for starting this discussion, connecting the CIP test
system with KernelCI sounds like a perfect example of how the
project can grow.

On Thu, Jun 4, 2020 at 11:15 AM Chris Paterson <Chris.Paterson2@renesas.com>
wrote:

> Hello Guillaume, Kevin,
>
> I'd really like to get CIP linked into KernelCI somehow before ELC-NA in a
> few weeks.
> Are you both available for a quick chat this week/next to determine what
> kind of integration would be suitable, and how we can go about achieving it?
>

Sure, let's try to find a slot next week.

> Some ramblings below to start the discussion...
>
> # Background on the current CIP CI setup
> - We have a GitLab mirror of our kernel.org kernel repo
> - Our branches contain a simple .gitlab-ci.yml file that links to our
> build/test GitLab CI pipeline repository
> - Kernel builds are kicked off using the various arches/configs supported
> by CIP
> - Test jobs are then submitted to our LAVA master
> - We currently run simple boot tests, spectre/meltdown checker,
> cyclictest+hackbench (RT configs only) and LTP (only run on CIP release
> candidates)
> - If the builds and tests finish okay then GitLab shows a green tick, if
> there is some kind of build error, or test infrastructure error then GitLab
> shows a red cross
> - If individual test cases fail, or there is a test regression, currently
> there is no way of knowing in the Gitlab UI. This is why I want to start
> using KernelCI
> - We follow a similar process for Greg's stable-rc 4.4 and 4.19 branches,
> but don't run LTP
>

This sounds very similar to the experiment Kevin did with his own
tree and Gitlab CI, except he's using the command line tools from
kernelci-core.

> # Options for KernelCI integration (?)
> 1) Keep our current build/test pipeline the same, just add a notify
> section to the LAVA jobs to send the results to kernelci.org
> - We'd need to include the correct metadata etc.
>

I think we might be able to have commands that generate just the
meta-data for a given kernel build without actually doing a
build.  That would allow you to keep your current build system
and also stay in sync with the kernelci meta-data definitions.

> - Would this work if there is no corresponding 'build job'?
>

I don't think anyone is doing this right now but in any case it's
something we need to support, for example with static analysis
and other tests that don't require any build.  The only thing
that should be really required is the kernel revision (tree,
branch, sha1, git describe).  So, "yes" this needs to work.

> - Would you allow test results from the CIP SLTS Kernels on kernelci.org?
> Or would we need to set up our own instance?
>

Ideally, only upstream-oriented branches should be built and
tested on kernelci.org to keep it in line with the project's
mission and upstream kernel development.  If the CIP branches
consist only of backported patches from mainline, maybe it could
be accepted in the same way that we have stable branches.  If
however there are patches that are not intended to get merged
upstream and are there for specific products, then in my opinion
that would not be acceptable - see below for an alternative as
you suggested.

> 1b) Same as 1) but using either cip.kernelci.org or
> kernelci.ciplatform.org

Yes, having a separate instance would enable a separate
configuration, separate database, special features etc... so no
possible interference with the main kernelci.org
activities.  Having it as cip.kernelci.org means it would
probably be managed by the KernelCI LF project, I guess that
would make sense for a member.  But then if you host and manage
your own instance on kernelci.ciplatform.org you're of course
free to do anything you want with it.

> 2) kernelci.org starts monitoring/building/testing the CIP SLTS Kernels
> - Tests would run through our LAVA master
> - Perhaps kernelci.org could make use of our build infrastructure?
> 2b) Same as 2) but using either cip.kernelci.org or
> kernelci.ciplatform.org
>

The question of kernelci.org vs another instance remains the
same, in my opinion it all hinges on what is on the branches.

Doing this would give you automated bisection since the KernelCI
pipeline would be able to run arbitrary jobs in the LAVA lab.
Using your own build infrastructure would probably be another
argument in favour of a separate instance (e.g. cip.kernelci.org)
as this would be a custom use-case, unless you're happy to have
any kernelci.org kernel builds run on your infrastructure.

3) kernelci.org start submitting their usual test jobs to the CIP LAVA
> master
> - CIP currently have 11 different device-types (20 boards in total) which
> should help expand what is available to KernelCI
>

Maybe we could do that anyway, with filters to limit the upstream
branches and test suites to run if you want, and also work out a
solution for the CIP SLTS branches?

> My preferred option would be 1) as it would be nice not to lose all the
> work spent on the GitLab CI setup
> (or perhaps integrating the GitLab CI approach into KernelCI's setup is a
> worthwhile task, but it'll take time)
>

Yes I think Option 1 would be good given the timeframe you've
set.  Also it's going to be a recurring use-case, KernelCI is all
about being flexible enough to integrate existing CI systems with
no real modifications to them.  We can try things out with
staging.kernelci.org initially.  Option 2 could be done as a
follow-up if you think the added features justify the extra
efforts.

And Option 3 seems really straightforward to me, any LAVA lab can
be added to kernelci.org with an API token and a bit of YAML
configuration.  That would definitely be a quick win if you want
to be sure to have something to show at ELC-NA :)

Best wishes,
Guillaume

[-- Attachment #2: Type: text/html, Size: 8972 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread