RE: RFC - kernel selftest result documentation (KTAP)

From: "Bird, Tim" <Tim.Bird@sony.com>
To: Kees Cook <keescook@chromium.org>
Cc: Brendan Higgins <brendanhiggins@google.com>,
	"shuah@kernel.org" <shuah@kernel.org>,
	"linux-kselftest@vger.kernel.org"
	<linux-kselftest@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Paolo Bonzini <pbonzini@redhat.com>,
	David Gow <davidgow@google.com>
Subject: RE: RFC - kernel selftest result documentation (KTAP)
Date: Wed, 17 Jun 2020 02:30:45 +0000	[thread overview]
Message-ID: <CY4PR13MB1175DCC4066FC0839A6B2E84FD9A0@CY4PR13MB1175.namprd13.prod.outlook.com> (raw)
In-Reply-To: <202006161703.B2E51605@keescook>


> -----Original Message-----
> From: Kees Cook
> 
> On Tue, Jun 16, 2020 at 09:16:01PM +0000, Bird, Tim wrote:
> > So far, most of the CI systems don't parse out diagnostic data, so it doesn't
> > really matter what the format is.  If it's useful for humans, it's valuable as is.
> > However, it would be nice if that could change.  But without some formalization
> > of the format of the diagnostic data, it's an intractable problem for CI systems
> > to parse it.  So it's really a chicken and egg problem.  To solve it, we would have
> > to determine what exactly needs to be provided on a consistent basis for diagnostic
> > data across many tests.  I think that it's too big a problem to handle right now.
> > I'm not opposed to migrating to some structure with yaml in the future, but free
> > form text output seems OK for now.
> 
> For a CI system, if I see a test has failed, I expect to be able to
> click a link to get the log of that test, which includes the diagnostic
> lines. The other reason to have them there is to show progress during a
> manual run.
Agreed.  You only need machine-parsable data if you expect the CI
system to do something more with the data than just present it.
What that would be, that would be common for all tests (or at least
many test), is unclear.  Maybe there are patterns in the diagnostic
data that could lead to higher-level analysis, or even automated
fixes, that don't become apparent if the data is unstructured.  But
it's hard to know until you have lots of data.  I think just getting
the other things consistent is a good priority right now.
 -- Tim

> 
> > > Yeah, I think it would be nice if all test frameworks/libraries for the
> > > kernel output tests in the same language.
> > Agreed.
> 
> $ git grep "TAP version"
> exec/binfmt_script:print("TAP version 1.3")
> kselftest.h:            printf("TAP version 13\n");
> kselftest/runner.sh:    echo "TAP version 13"
> resctrl/resctrl_tests.c:        printf("TAP version 13\n");
> size/get_size.c:        print("TAP version 13\n");
> 
> Looks like there are 2 tests to convert to kselftest.h, and then we can
> just change the version to 14 in the header and the runner. ;)
> 
> --
> Kees Cook