All of lore.kernel.org
 help / color / mirror / Atom feed
* Testing with distros
@ 2018-06-22 12:34 Guillaume Tucker
  2018-06-22 18:35 ` getting started? micah.parrish
  2018-06-27 23:05 ` [kernelci] Testing with distros Kevin Hilman
  0 siblings, 2 replies; 8+ messages in thread
From: Guillaume Tucker @ 2018-06-22 12:34 UTC (permalink / raw)
  To: kernelci

Hi,

One subject that comes up regularly is about testing distributions
against upstream kernels in KernelCI.  We've been historically using a
bare buildroot user-space to do boot testing and run very low-level test
cases.  We're now starting to produce Debian root file systems for more
advanced test plans and extend the coverage of kernel-user interfaces
(DRM, V4L2...).

It seems like there would also be some value in having tests that
capture distro user-space breakages due to kernel changes.  This may
mean building upstream kernels with a distro config, booting with a
fixed stable user-space from that distro and detecting new failures.

The next part to be defined would be what tests to run with these
distros, as even booting means choosing which services to start etc...
The point being to increase kernel test coverage without ending up
testing the user-space itself as this would seem outside of the scope of
KernelCI.

Does this sound like a path worth exploring?

Best wishes,
Guillaume

^ permalink raw reply	[flat|nested] 8+ messages in thread

* getting started?
  2018-06-22 12:34 Testing with distros Guillaume Tucker
@ 2018-06-22 18:35 ` micah.parrish
  2018-06-28 13:11   ` [kernelci] " Matt Hart
  2018-06-27 23:05 ` [kernelci] Testing with distros Kevin Hilman
  1 sibling, 1 reply; 8+ messages in thread
From: micah.parrish @ 2018-06-22 18:35 UTC (permalink / raw)
  To: kernelci

I really like what you've done with kernelci.org, and I'm interested in 
running it in a semi-isolated x86 test lab.  Do you have a recipe I can 
follow for doing this?  It seems like there are a lot of moving parts, 
but I don't quite see how they all fit together yet.

Regards,

Micah Parrish
Linux OS Engineer
Hewlett-Packard Enterprise

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [kernelci] Testing with distros
  2018-06-22 12:34 Testing with distros Guillaume Tucker
  2018-06-22 18:35 ` getting started? micah.parrish
@ 2018-06-27 23:05 ` Kevin Hilman
  2018-06-28 13:13   ` Matt Hart
  2018-07-02  8:07   ` Tomeu Vizoso
  1 sibling, 2 replies; 8+ messages in thread
From: Kevin Hilman @ 2018-06-27 23:05 UTC (permalink / raw)
  To: Guillaume Tucker; +Cc: kernelci

"Guillaume Tucker" <guillaume.tucker@gmail.com> writes:

> One subject that comes up regularly is about testing distributions
> against upstream kernels in KernelCI.  We've been historically using a
> bare buildroot user-space to do boot testing and run very low-level test
> cases.  We're now starting to produce Debian root file systems for more
> advanced test plans and extend the coverage of kernel-user interfaces
> (DRM, V4L2...).
>
> It seems like there would also be some value in having tests that
> capture distro user-space breakages due to kernel changes.  This may
> mean building upstream kernels with a distro config, booting with a
> fixed stable user-space from that distro and detecting new failures.
>
> The next part to be defined would be what tests to run with these
> distros, as even booting means choosing which services to start etc...
> The point being to increase kernel test coverage without ending up
> testing the user-space itself as this would seem outside of the scope of
> KernelCI.
>
> Does this sound like a path worth exploring?

It definitely sounds like a path worth exploring, but personally, I'd
rather get our kernel-focused testing (and reporting!) figured out using
our debian baseline before tackling distros.

In particular, I think a kselftest and/or LTP testplan should probably
be next on the list.

Kevin

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [kernelci] getting started?
  2018-06-22 18:35 ` getting started? micah.parrish
@ 2018-06-28 13:11   ` Matt Hart
  2018-06-28 17:53     ` Micah Parrish
  0 siblings, 1 reply; 8+ messages in thread
From: Matt Hart @ 2018-06-28 13:11 UTC (permalink / raw)
  To: kernelci

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

Hi Micah

There are indeed a lot of moving parts. You could run your own web UI and
API fairly easily but there's no guides to setup the build system and the
boot testing (LAVA) lab.
What are you trying to achieve? Could it be with KernelCI rather than your
own setup?

Matt

On 22 June 2018 at 19:35, Micah Parrish <micah.parrish@hpe.com> wrote:

> I really like what you've done with kernelci.org, and I'm interested in
> running it in a semi-isolated x86 test lab.  Do you have a recipe I can
> follow for doing this?  It seems like there are a lot of moving parts, but
> I don't quite see how they all fit together yet.
>
> Regards,
>
> Micah Parrish
> Linux OS Engineer
> Hewlett-Packard Enterprise
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 1178 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [kernelci] Testing with distros
  2018-06-27 23:05 ` [kernelci] Testing with distros Kevin Hilman
@ 2018-06-28 13:13   ` Matt Hart
  2018-07-02  8:07   ` Tomeu Vizoso
  1 sibling, 0 replies; 8+ messages in thread
From: Matt Hart @ 2018-06-28 13:13 UTC (permalink / raw)
  To: kernelci; +Cc: Guillaume Tucker

[-- Attachment #1: Type: text/plain, Size: 1613 bytes --]

On 28 June 2018 at 00:05, Kevin Hilman <khilman@baylibre.com> wrote:

> "Guillaume Tucker" <guillaume.tucker@gmail.com> writes:
>
> > One subject that comes up regularly is about testing distributions
> > against upstream kernels in KernelCI.  We've been historically using a
> > bare buildroot user-space to do boot testing and run very low-level test
> > cases.  We're now starting to produce Debian root file systems for more
> > advanced test plans and extend the coverage of kernel-user interfaces
> > (DRM, V4L2...).
> >
> > It seems like there would also be some value in having tests that
> > capture distro user-space breakages due to kernel changes.  This may
> > mean building upstream kernels with a distro config, booting with a
> > fixed stable user-space from that distro and detecting new failures.
> >
> > The next part to be defined would be what tests to run with these
> > distros, as even booting means choosing which services to start etc...
> > The point being to increase kernel test coverage without ending up
> > testing the user-space itself as this would seem outside of the scope of
> > KernelCI.
> >
> > Does this sound like a path worth exploring?
>
> It definitely sounds like a path worth exploring, but personally, I'd
> rather get our kernel-focused testing (and reporting!) figured out using
> our debian baseline before tackling distros.
>

I'd put distro testing as a long term interest. I like it, but there's a
lot of other things that should come first.



> In particular, I think a kselftest and/or LTP testplan should probably
> be next on the list.
>
> Kevin
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 2376 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [kernelci] getting started?
  2018-06-28 13:11   ` [kernelci] " Matt Hart
@ 2018-06-28 17:53     ` Micah Parrish
  2018-07-26 22:22       ` Kevin Hilman
  0 siblings, 1 reply; 8+ messages in thread
From: Micah Parrish @ 2018-06-28 17:53 UTC (permalink / raw)
  To: kernelci

I'm trying to redeploy a mix of prototypes and production systems as a 
very low touch "top of tree kernel test" farm.  This give us early 
warning when an kernel change has affected our HW or FW.  Kernelci.org 
does almost exactly what I'd like to do, but I need to keep an instance 
in-house because of prototype confidentiality requirements.  If that 
works out I'm open to putting some production servers in the main pool 
somehow.

Is there a diagram of how all the parts fit together?

-Micah


On 06/28/2018 07:11 AM, Matt Hart wrote:
> Hi Micah
>
> There are indeed a lot of moving parts. You could run your own web UI 
> and API fairly easily but there's no guides to setup the build system 
> and the boot testing (LAVA) lab.
> What are you trying to achieve? Could it be with KernelCI rather than 
> your own setup?
>
> Matt
>
> On 22 June 2018 at 19:35, Micah Parrish <micah.parrish@hpe.com 
> <mailto:micah.parrish@hpe.com>> wrote:
>
>     I really like what you've done with kernelci.org
>     <http://kernelci.org>, and I'm interested in running it in a
>     semi-isolated x86 test lab.  Do you have a recipe I can follow for
>     doing this?  It seems like there are a lot of moving parts, but I
>     don't quite see how they all fit together yet.
>
>     Regards,
>
>     Micah Parrish
>     Linux OS Engineer
>     Hewlett-Packard Enterprise
>
>
>
>
> 



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [kernelci] Testing with distros
  2018-06-27 23:05 ` [kernelci] Testing with distros Kevin Hilman
  2018-06-28 13:13   ` Matt Hart
@ 2018-07-02  8:07   ` Tomeu Vizoso
  1 sibling, 0 replies; 8+ messages in thread
From: Tomeu Vizoso @ 2018-07-02  8:07 UTC (permalink / raw)
  To: kernelci, Guillaume Tucker

On 06/28/2018 01:05 AM, Kevin Hilman wrote:
> "Guillaume Tucker" <guillaume.tucker@gmail.com> writes:
> 
>> One subject that comes up regularly is about testing distributions
>> against upstream kernels in KernelCI.  We've been historically using a
>> bare buildroot user-space to do boot testing and run very low-level test
>> cases.  We're now starting to produce Debian root file systems for more
>> advanced test plans and extend the coverage of kernel-user interfaces
>> (DRM, V4L2...).
>>
>> It seems like there would also be some value in having tests that
>> capture distro user-space breakages due to kernel changes.  This may
>> mean building upstream kernels with a distro config, booting with a
>> fixed stable user-space from that distro and detecting new failures.
>>
>> The next part to be defined would be what tests to run with these
>> distros, as even booting means choosing which services to start etc...
>> The point being to increase kernel test coverage without ending up
>> testing the user-space itself as this would seem outside of the scope of
>> KernelCI.
>>
>> Does this sound like a path worth exploring?
> 
> It definitely sounds like a path worth exploring, but personally, I'd
> rather get our kernel-focused testing (and reporting!) figured out using
> our debian baseline before tackling distros.

I would prefer if distros ran mainline kernels in their own CI, then 
reported the bugs upstream after triage. And ideally, added tests against 
the kernel ABI to kernelci.

Regards,

Tomeu

> In particular, I think a kselftest and/or LTP testplan should probably
> be next on the list.
> 
> Kevin
> 
> 
> 

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [kernelci] getting started?
  2018-06-28 17:53     ` Micah Parrish
@ 2018-07-26 22:22       ` Kevin Hilman
  0 siblings, 0 replies; 8+ messages in thread
From: Kevin Hilman @ 2018-07-26 22:22 UTC (permalink / raw)
  To: Micah Parrish; +Cc: kernelci

"Micah Parrish" <micah.parrish@hpe.com> writes:

> I'm trying to redeploy a mix of prototypes and production systems as a
> very low touch "top of tree kernel test" farm.  This give us early
> warning when an kernel change has affected our HW or FW.  Kernelci.org
> does almost exactly what I'd like to do, but I need to keep an
> instance in-house because of prototype confidentiality requirements. 
> If that works out I'm open to putting some production servers in the
> main pool somehow.
>
> Is there a diagram of how all the parts fit together?

The long delay in responding meant that no, there was no diagram.  Our
docs are pretty minimal, but we've begun an attempt to improve that
on the wiki for the kernelci-doc project: https://github.com/kernelci/kernelci-doc/wiki

The sub-page for setting up a local instance[1] would be a good start,
and to compliment that, I just whipped up this draw.io diagram[1] that
tries to show graphically how all the kernelCI pieces fit together in a
full CI loop. Most items have links to the various github projects where
the code lives.

I admit that it's very primitive, but it should at least help you get
started seeing how all the parts fit together.

Hope that helps,

Kevin

[1] https://github.com/kernelci/kernelci-doc/wiki/Setting-up-a-local-development-instance
[2] https://drive.google.com/file/d/1x2SXt24ZNzuk3SfBzsQpw_RnLOcF3vWd/view?usp=sharing


> -Micah
>
>
> On 06/28/2018 07:11 AM, Matt Hart wrote:
>> Hi Micah
>>
>> There are indeed a lot of moving parts. You could run your own web
>> UI and API fairly easily but there's no guides to setup the build
>> system and the boot testing (LAVA) lab.
>> What are you trying to achieve? Could it be with KernelCI rather
>> than your own setup?
>>
>> Matt
>>
>> On 22 June 2018 at 19:35, Micah Parrish <micah.parrish@hpe.com
>> <mailto:micah.parrish@hpe.com>> wrote:
>>
>>     I really like what you've done with kernelci.org
>>     <http://kernelci.org>, and I'm interested in running it in a
>>     semi-isolated x86 test lab.  Do you have a recipe I can follow for
>>     doing this?  It seems like there are a lot of moving parts, but I
>>     don't quite see how they all fit together yet.
>>
>>     Regards,
>>
>>     Micah Parrish
>>     Linux OS Engineer
>>     Hewlett-Packard Enterprise
>>
>>
>>
>>
>>
>
>
>
> 

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2018-07-26 22:22 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-06-22 12:34 Testing with distros Guillaume Tucker
2018-06-22 18:35 ` getting started? micah.parrish
2018-06-28 13:11   ` [kernelci] " Matt Hart
2018-06-28 17:53     ` Micah Parrish
2018-07-26 22:22       ` Kevin Hilman
2018-06-27 23:05 ` [kernelci] Testing with distros Kevin Hilman
2018-06-28 13:13   ` Matt Hart
2018-07-02  8:07   ` Tomeu Vizoso

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.