Re: RFC - kernel selftest result documentation (KTAP)

From: Kees Cook <keescook@chromium.org>
To: "Bird, Tim" <Tim.Bird@sony.com>
Cc: "shuah@kernel.org" <shuah@kernel.org>,
	"linux-kselftest@vger.kernel.org"
	<linux-kselftest@vger.kernel.org>,
	Brendan Higgins <brendanhiggins@google.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	David Gow <davidgow@google.com>,
	Paolo Bonzini <pbonzini@redhat.com>
Subject: Re: RFC - kernel selftest result documentation (KTAP)
Date: Tue, 16 Jun 2020 16:52:10 -0700	[thread overview]
Message-ID: <202006161642.A5F50ED07@keescook> (raw)
In-Reply-To: <CY4PR13MB11757D57CD441C5CAEC3F257FD9C0@CY4PR13MB1175.namprd13.prod.outlook.com>

On Mon, Jun 15, 2020 at 07:07:34PM +0000, Bird, Tim wrote:
> From: Kees Cook <keescook@chromium.org>
> > Note: making the plan line required differs from TAP13 and TAP14. I
> > think it's the right choice, but we should be clear.
> 
> [...]
> With regards to making it optional or not, I don't have a strong
> preference.  The extra info seems helpful in some circumstances.
> I don't know if it's too onerous to make it a requirement or not.
> I'd prefer if it was always there (either at the beginning or the end),
> but if there is some situation where it's quite difficult to calculate,
> then it would be best not to mandate it. I can't think of any impossible
> situations at the moment.

I think we should require one of:

- starting plan line
- ending plan line
- ending with something that indicates "I'm done, but I have no idea how
  many tests actually ran" (Maybe "1..?")

To me, the point of the plan line is to be able to say "this test did,
in fact, finish". So even if some test can't even count how many tests
it _ran_, it can at least say "I am now finished".

> > TAP13/14 makes description optional, are we making it required (I think
> > we should). There seems to be a TAP13/14 "convention" of starting
> > <description> with "- ", which I'm on the fence about it. It does make
> > parsing maybe a little easier.
> 
> I would like the description to be required.
> I don't have a strong opinion on the dash.  I'm OK with either one (dash
> or no dash), but we should make kselftest and KUnit consistent.

I find the dash to be distracting -- it doesn't help me scan it, and it
doesn't help a parser (which only needs to find "#").

> > > Differences between kernel test result format and TAP13:
> > >  - in KTAP the "# SKIP" directive is placed after the description on
> > >    the test result line

I sent a bunch of clean-ups for kselftest.h recently[1], but it looks
like we'll need more for adding "description" to skip (right now it only
prints the SKIP reason).

[1] https://lore.kernel.org/lkml/20200611224028.3275174-1-keescook@chromium.org/

> > Yes Documentation/*.rst Not sure on name yet, but where do kselftest
> > docs live? :)
> Documentation/dev-tools/kselftest.rst
> 
> I'll put this at: Documentation/dev-tools/test-results-format.rst

Sounds good!

-- 
Kees Cook