From mboxrd@z Thu Jan 1 00:00:00 1970 Subject: Re: #KCIDB engagement report References: <5a9bf050-0671-3273-cc4f-1b131445c1fe@redhat.com> <202106011315.432A65D6@keescook> <774899c5-c20a-3d7e-3289-ee257b86e06e@collabora.com> <202106151501.235746C5@keescook> From: "Guillaume Tucker" Message-ID: Date: Tue, 15 Jun 2021 23:23:35 +0100 MIME-Version: 1.0 In-Reply-To: <202106151501.235746C5@keescook> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit List-ID: To: Kees Cook Cc: kernelci@groups.io, Nick Desaulniers , Nikolai Kondrashov , "automated-testing@yoctoproject.org" , clang-built-linux , Vishal Bhoj , Antonio Terceiro , Remi Duraffort , Alexandra da Silva Pereira +alex On 15/06/2021 23:03, Kees Cook wrote: > On Fri, Jun 11, 2021 at 05:11:59PM +0100, Guillaume Tucker wrote: >> Hi Kees, >> >> On 01/06/2021 21:26, Kees Cook wrote: >>> On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang Built Linux wrote: >>>> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov >>>> wrote: >>>>> [...] >>>>> KernelCI native >>>>> Sending (a lot of) production build and test results. >>>>> https://staging.kernelci.org:3000/?var-origin=kernelci >>>>> [...] >>> >>> Apologies for the thread hijack, but does anyone know what's happening >>> with kselftest? It seems missing from the listed[1] build artifacts, but >>> it is actually present[2] (and I see the logs for generating the tarball >>> there too), but I can't find any builds that actually run the tests? >>> >>> (Or how do I see a top-level list of all tests and search it?) >> >> The kselftest results are all there on the KernelCI native >> dashboard, for example the futex tests: >> >> https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.13-rc5-74-g06af8679449d/plan/kselftest-futex/ > > Thanks for looking at this for me! :) > > How do I find the other kselftest stuff? I just see "kselftest-futex" > and "kselftest-filesystem". I was expecting _all_ of the kselftests, but > I can't find them. > > (Specifically, I can't find a top-level "list of all test plans") That's because kselftest is rather large, and we're only enabling subsets of it one at a time. As more test labs and more devices become available, we'll gradually expand coverage. We might also choose to have full coverage only on say, linux-next, mainline and LTS branches but not everywhere to not overload the labs. To answer your question about "all the tests", well you can look at any kernel revision to see the tests that were run for it since it won't be the same for all of them. Typically, linux-next has the highest number of tests so here's an example: https://linux.kernelci.org/test/job/next/branch/master/kernel/next-20210615/ As you've already found, there are only 3 kselftest subsets or "collections" being run there at the moment. That's by design in the KernelCI configuration, but at least we have good enough support for running kselftest now which wasn't completely trivial to put in place... There are still a few issues to fix, but I would expect kselftest coverage to keep growing over the coming weeks. If there are kselftest collections you really want to have enabled, you can always make a PR to add them to this file: https://github.com/kernelci/kernelci-core/blob/main/config/core/test-configs.yaml#L187 As long as there's capacity for it at least on some types of devices and it runs as expected, we should be able to get this deployed in production pretty easily. Thanks, Guillaume