Testing with distros

Testing with distros

[Guillaume Tucker]

Hi,

One subject that comes up regularly is about testing distributions
against upstream kernels in KernelCI.  We've been historically using a
bare buildroot user-space to do boot testing and run very low-level test
cases.  We're now starting to produce Debian root file systems for more
advanced test plans and extend the coverage of kernel-user interfaces
(DRM, V4L2...).

It seems like there would also be some value in having tests that
capture distro user-space breakages due to kernel changes.  This may
mean building upstream kernels with a distro config, booting with a
fixed stable user-space from that distro and detecting new failures.

The next part to be defined would be what tests to run with these
distros, as even booting means choosing which services to start etc...
The point being to increase kernel test coverage without ending up
testing the user-space itself as this would seem outside of the scope of
KernelCI.

Does this sound like a path worth exploring?

Best wishes,
Guillaume

getting started?

[micah.parrish]

I really like what you've done with kernelci.org, and I'm interested in 
running it in a semi-isolated x86 test lab.  Do you have a recipe I can 
follow for doing this?  It seems like there are a lot of moving parts, 
but I don't quite see how they all fit together yet.

Regards,

Micah Parrish
Linux OS Engineer
Hewlett-Packard Enterprise

Re: [kernelci] getting started?

[Matt Hart]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

Hi Micah

There are indeed a lot of moving parts. You could run your own web UI and
API fairly easily but there's no guides to setup the build system and the
boot testing (LAVA) lab.
What are you trying to achieve? Could it be with KernelCI rather than your
own setup?

Matt

On 22 June 2018 at 19:35, Micah Parrish <micah.parrish@hpe.com> wrote:

> I really like what you've done with kernelci.org, and I'm interested in
> running it in a semi-isolated x86 test lab.  Do you have a recipe I can
> follow for doing this?  It seems like there are a lot of moving parts, but
> I don't quite see how they all fit together yet.
>
> Regards,
>
> Micah Parrish
> Linux OS Engineer
> Hewlett-Packard Enterprise
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 1178 bytes --]

Re: [kernelci] getting started?

[Micah Parrish]

I'm trying to redeploy a mix of prototypes and production systems as a 
very low touch "top of tree kernel test" farm.  This give us early 
warning when an kernel change has affected our HW or FW.  Kernelci.org 
does almost exactly what I'd like to do, but I need to keep an instance 
in-house because of prototype confidentiality requirements.  If that 
works out I'm open to putting some production servers in the main pool 
somehow.

Is there a diagram of how all the parts fit together?

-Micah


On 06/28/2018 07:11 AM, Matt Hart wrote:
> Hi Micah
>
> There are indeed a lot of moving parts. You could run your own web UI 
> and API fairly easily but there's no guides to setup the build system 
> and the boot testing (LAVA) lab.
> What are you trying to achieve? Could it be with KernelCI rather than 
> your own setup?
>
> Matt
>
> On 22 June 2018 at 19:35, Micah Parrish <micah.parrish@hpe.com 
> <mailto:micah.parrish@hpe.com>> wrote:
>
>     I really like what you've done with kernelci.org
>     <http://kernelci.org>, and I'm interested in running it in a
>     semi-isolated x86 test lab.  Do you have a recipe I can follow for
>     doing this?  It seems like there are a lot of moving parts, but I
>     don't quite see how they all fit together yet.
>
>     Regards,
>
>     Micah Parrish
>     Linux OS Engineer
>     Hewlett-Packard Enterprise
>
>
>
>
> 

Re: [kernelci] getting started?

[Kevin Hilman]

"Micah Parrish" <micah.parrish@hpe.com> writes:

> I'm trying to redeploy a mix of prototypes and production systems as a
> very low touch "top of tree kernel test" farm.  This give us early
> warning when an kernel change has affected our HW or FW.  Kernelci.org
> does almost exactly what I'd like to do, but I need to keep an
> instance in-house because of prototype confidentiality requirements. 
> If that works out I'm open to putting some production servers in the
> main pool somehow.
>
> Is there a diagram of how all the parts fit together?

The long delay in responding meant that no, there was no diagram.  Our
docs are pretty minimal, but we've begun an attempt to improve that
on the wiki for the kernelci-doc project: https://github.com/kernelci/kernelci-doc/wiki

The sub-page for setting up a local instance[1] would be a good start,
and to compliment that, I just whipped up this draw.io diagram[1] that
tries to show graphically how all the kernelCI pieces fit together in a
full CI loop. Most items have links to the various github projects where
the code lives.

I admit that it's very primitive, but it should at least help you get
started seeing how all the parts fit together.

Hope that helps,

Kevin

[1] https://github.com/kernelci/kernelci-doc/wiki/Setting-up-a-local-development-instance
[2] https://drive.google.com/file/d/1x2SXt24ZNzuk3SfBzsQpw_RnLOcF3vWd/view?usp=sharing


> -Micah
>
>
> On 06/28/2018 07:11 AM, Matt Hart wrote:
>> Hi Micah
>>
>> There are indeed a lot of moving parts. You could run your own web
>> UI and API fairly easily but there's no guides to setup the build
>> system and the boot testing (LAVA) lab.
>> What are you trying to achieve? Could it be with KernelCI rather
>> than your own setup?
>>
>> Matt
>>
>> On 22 June 2018 at 19:35, Micah Parrish <micah.parrish@hpe.com
>> <mailto:micah.parrish@hpe.com>> wrote:
>>
>>     I really like what you've done with kernelci.org
>>     <http://kernelci.org>, and I'm interested in running it in a
>>     semi-isolated x86 test lab.  Do you have a recipe I can follow for
>>     doing this?  It seems like there are a lot of moving parts, but I
>>     don't quite see how they all fit together yet.
>>
>>     Regards,
>>
>>     Micah Parrish
>>     Linux OS Engineer
>>     Hewlett-Packard Enterprise
>>
>>
>>
>>
>>
>
>
>
> 

Re: [kernelci] Testing with distros

[Kevin Hilman]

"Guillaume Tucker" <guillaume.tucker@gmail.com> writes:

> One subject that comes up regularly is about testing distributions
> against upstream kernels in KernelCI.  We've been historically using a
> bare buildroot user-space to do boot testing and run very low-level test
> cases.  We're now starting to produce Debian root file systems for more
> advanced test plans and extend the coverage of kernel-user interfaces
> (DRM, V4L2...).
>
> It seems like there would also be some value in having tests that
> capture distro user-space breakages due to kernel changes.  This may
> mean building upstream kernels with a distro config, booting with a
> fixed stable user-space from that distro and detecting new failures.
>
> The next part to be defined would be what tests to run with these
> distros, as even booting means choosing which services to start etc...
> The point being to increase kernel test coverage without ending up
> testing the user-space itself as this would seem outside of the scope of
> KernelCI.
>
> Does this sound like a path worth exploring?

It definitely sounds like a path worth exploring, but personally, I'd
rather get our kernel-focused testing (and reporting!) figured out using
our debian baseline before tackling distros.

In particular, I think a kselftest and/or LTP testplan should probably
be next on the list.

Kevin

Re: [kernelci] Testing with distros

[Matt Hart]

[-- Attachment #1: Type: text/plain, Size: 1613 bytes --]

On 28 June 2018 at 00:05, Kevin Hilman <khilman@baylibre.com> wrote:

> "Guillaume Tucker" <guillaume.tucker@gmail.com> writes:
>
> > One subject that comes up regularly is about testing distributions
> > against upstream kernels in KernelCI.  We've been historically using a
> > bare buildroot user-space to do boot testing and run very low-level test
> > cases.  We're now starting to produce Debian root file systems for more
> > advanced test plans and extend the coverage of kernel-user interfaces
> > (DRM, V4L2...).
> >
> > It seems like there would also be some value in having tests that
> > capture distro user-space breakages due to kernel changes.  This may
> > mean building upstream kernels with a distro config, booting with a
> > fixed stable user-space from that distro and detecting new failures.
> >
> > The next part to be defined would be what tests to run with these
> > distros, as even booting means choosing which services to start etc...
> > The point being to increase kernel test coverage without ending up
> > testing the user-space itself as this would seem outside of the scope of
> > KernelCI.
> >
> > Does this sound like a path worth exploring?
>
> It definitely sounds like a path worth exploring, but personally, I'd
> rather get our kernel-focused testing (and reporting!) figured out using
> our debian baseline before tackling distros.
>

I'd put distro testing as a long term interest. I like it, but there's a
lot of other things that should come first.



> In particular, I think a kselftest and/or LTP testplan should probably
> be next on the list.
>
> Kevin
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 2376 bytes --]

Re: [kernelci] Testing with distros

[Tomeu Vizoso]

On 06/28/2018 01:05 AM, Kevin Hilman wrote:
> "Guillaume Tucker" <guillaume.tucker@gmail.com> writes:
> 
>> One subject that comes up regularly is about testing distributions
>> against upstream kernels in KernelCI.  We've been historically using a
>> bare buildroot user-space to do boot testing and run very low-level test
>> cases.  We're now starting to produce Debian root file systems for more
>> advanced test plans and extend the coverage of kernel-user interfaces
>> (DRM, V4L2...).
>>
>> It seems like there would also be some value in having tests that
>> capture distro user-space breakages due to kernel changes.  This may
>> mean building upstream kernels with a distro config, booting with a
>> fixed stable user-space from that distro and detecting new failures.
>>
>> The next part to be defined would be what tests to run with these
>> distros, as even booting means choosing which services to start etc...
>> The point being to increase kernel test coverage without ending up
>> testing the user-space itself as this would seem outside of the scope of
>> KernelCI.
>>
>> Does this sound like a path worth exploring?
> 
> It definitely sounds like a path worth exploring, but personally, I'd
> rather get our kernel-focused testing (and reporting!) figured out using
> our debian baseline before tackling distros.

I would prefer if distros ran mainline kernels in their own CI, then 
reported the bugs upstream after triage. And ideally, added tests against 
the kernel ABI to kernelci.

Regards,

Tomeu

> In particular, I think a kselftest and/or LTP testplan should probably
> be next on the list.
> 
> Kevin
> 
> 
>