Re: #KCIDB engagement report

From: "Guillaume Tucker" <guillaume.tucker@collabora.com>
To: Kees Cook <keescook@chromium.org>
Cc: kernelci@groups.io, Nick Desaulniers <ndesaulniers@google.com>,
	Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>,
	"automated-testing@yoctoproject.org"
	<automated-testing@yoctoproject.org>,
	clang-built-linux <clang-built-linux@googlegroups.com>,
	Vishal Bhoj <vishal.bhoj@linaro.org>,
	Antonio Terceiro <antonio.terceiro@linaro.org>,
	Remi Duraffort <remi.duraffort@linaro.org>,
	Alexandra da Silva Pereira <alexandra.pereira@collabora.com>
Subject: Re: #KCIDB engagement report
Date: Tue, 15 Jun 2021 23:23:35 +0100	[thread overview]
Message-ID: <f68058fe-a047-2c02-ce35-8349a95f69b8@collabora.com> (raw)
In-Reply-To: <202106151501.235746C5@keescook>

+alex 

On 15/06/2021 23:03, Kees Cook wrote:
> On Fri, Jun 11, 2021 at 05:11:59PM +0100, Guillaume Tucker wrote:
>> Hi Kees,
>>
>> On 01/06/2021 21:26, Kees Cook wrote:
>>> On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang Built Linux wrote:
>>>> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
>>>> <Nikolai.Kondrashov@redhat.com> wrote:
>>>>> [...]
>>>>>      KernelCI native
>>>>>          Sending (a lot of) production build and test results.
>>>>>          https://staging.kernelci.org:3000/?var-origin=kernelci
>>>>> [...]
>>>
>>> Apologies for the thread hijack, but does anyone know what's happening
>>> with kselftest? It seems missing from the listed[1] build artifacts, but
>>> it is actually present[2] (and I see the logs for generating the tarball
>>> there too), but I can't find any builds that actually run the tests?
>>>
>>> (Or how do I see a top-level list of all tests and search it?)
>>
>> The kselftest results are all there on the KernelCI native
>> dashboard, for example the futex tests:
>>
>>   https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.13-rc5-74-g06af8679449d/plan/kselftest-futex/
> 
> Thanks for looking at this for me! :)
> 
> How do I find the other kselftest stuff? I just see "kselftest-futex"
> and "kselftest-filesystem". I was expecting _all_ of the kselftests, but
> I can't find them.
> 
> (Specifically, I can't find a top-level "list of all test plans")

That's because kselftest is rather large, and we're only enabling
subsets of it one at a time.  As more test labs and more devices
become available, we'll gradually expand coverage.  We might also
choose to have full coverage only on say, linux-next, mainline
and LTS branches but not everywhere to not overload the labs.

To answer your question about "all the tests", well you can look
at any kernel revision to see the tests that were run for it
since it won't be the same for all of them.  Typically,
linux-next has the highest number of tests so here's an example:

  https://linux.kernelci.org/test/job/next/branch/master/kernel/next-20210615/

As you've already found, there are only 3 kselftest subsets
or "collections" being run there at the moment.  That's by design
in the KernelCI configuration, but at least we have good enough
support for running kselftest now which wasn't completely
trivial to put in place...

There are still a few issues to fix, but I would expect kselftest
coverage to keep growing over the coming weeks.

If there are kselftest collections you really want to have
enabled, you can always make a PR to add them to this file:

  https://github.com/kernelci/kernelci-core/blob/main/config/core/test-configs.yaml#L187

As long as there's capacity for it at least on some types of
devices and it runs as expected, we should be able to get this
deployed in production pretty easily.

Thanks,
Guillaume