From: Brendan Higgins <brendanhiggins@google.com>
To: "Bird, Tim" <Tim.Bird@sony.com>
Cc: "shuah@kernel.org" <shuah@kernel.org>,
"linux-kselftest@vger.kernel.org"
<linux-kselftest@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: RFC - kernel selftest result documentation (KTAP)
Date: Tue, 16 Jun 2020 13:48:17 -0700 [thread overview]
Message-ID: <20200616204817.GA212825@google.com> (raw)
In-Reply-To: <CY4PR13MB1175B804E31E502221BC8163FD830@CY4PR13MB1175.namprd13.prod.outlook.com>
On Wed, Jun 10, 2020 at 06:11:06PM +0000, Bird, Tim wrote:
> Some months ago I started work on a document to formalize how
> kselftest implements the TAP specification. However, I didn't finish
> that work. Maybe it's time to do so now.
>
> kselftest has developed a few differences from the original
> TAP specification, and some extensions that I believe are worth
> documenting.
>
> Essentially, we have created our own KTAP (kernel TAP)
> format. I think it is worth documenting our conventions, in order to
> keep everyone on the same page.
>
> Below is a partially completed document on my understanding
> of KTAP, based on examination of some of the kselftest test
> output. I have not reconciled this with the kunit output format,
> which I believe has some differences (which maybe we should
> resolve before we get too far into this).
>
> I submit the document now, before it is finished, because a patch
> was recently introduced to alter one of the result conventions
> (from SKIP='not ok' to SKIP='ok').
>
> See the document include inline below
>
> ====== start of ktap-doc-rfc.txt ======
[...]
> --- from here on is not-yet-organized material
>
> Tip:
> - don't change the test plan based on skipped tests.
> - it is better to report that a test case was skipped, than to
> not report it
> - that is, don't adjust the number of test cases based on skipped
> tests
>
> Other things to mention:
> TAP13 elements not used:
> - yaml for diagnostic messages
We talked about this before, but I would like some way to get failed
expectation/assertion information in the test in a consistent machine
parsible way. Currently we do the following:
# Subtest: example
1..1
# example_simple_test: initializing
# example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
Expected 1 + 1 == 3, but
1 + 1 == 2
3 == 3
not ok 1 - example_simple_test
not ok 5 - example
Technically not TAP compliant, but no one seems to mind. I am okay with
keeping it the way it is, but if we don't want it in the KTAP spec, we
will need some kind of recourse.
> - reason: try to keep things line-based, since output from other things
> may be interspersed with messages from the test itself
> - TODO directive
Is this more of stating a fact or desire? We don't use TODO either, but
it looks like it could be useful.
> KTAP Extensions beyond TAP13:
> - nesting
> - via indentation
> - indentation makes it easier for humans to read
> - test identifier
> - multiple parts, separated by ':'
Can you elabroate on this more? I am not sure what you mean.
> - summary lines
> - can be skipped by CI systems that do their own calculations
>
> Other notes:
> - automatic assignment of result status based on exit code
>
> Tips:
> - do NOT describe the result in the test line
> - the test case description should be the same whether the test
> succeeds or fails
> - use diagnostic lines to describe or explain results, if this is
> desirable
> - test numbers are considered harmful
> - test harnesses should use the test description as the identifier
> - test numbers change when testcases are added or removed
> - which means that results can't be compared between different
> versions of the test
> - recommendations for diagnostic messages:
> - reason for failure
> - reason for skip
> - diagnostic data should always preceding the result line
> - problem: harness may emit result before test can do assessment
> to determine reason for result
> - this is what the kernel uses
>
> Differences between kernel test result format and TAP13:
> - in KTAP the "# SKIP" directive is placed after the description on
> the test result line
>
> ====== start of ktap-doc-rfc.txt ======
> OK - that's the end of the RFC doc.
>
> Here are a few questions:
> - is this document desired or not?
> - is it too long or too short?
> - if the document is desired, where should it be placed?
I like it. I don't think we can rely on the TAP people updating their
stuff based on my interactions with them. So having a spec which is
actually maintained would be nice.
Maybe in Documentation/dev-tools/ ?
> I assume somewhere under Documentation, and put into
> .rst format. Suggestions for a name and location are welcome.
> - is this document accurate?
> I think KUNIT does a few things differently than this description.
> - is the intent to have kunit and kselftest have the same output format?
> if so, then these should be rationalized.
Yeah, I think it would be nice if all test frameworks/libraries for the
kernel output tests in the same language.
Cheers
next prev parent reply other threads:[~2020-06-16 20:48 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-10 18:11 RFC - kernel selftest result documentation (KTAP) Bird, Tim
2020-06-13 5:07 ` David Gow
2020-06-15 17:34 ` Bird, Tim
2020-06-16 20:03 ` Brendan Higgins
2020-06-16 20:37 ` Bird, Tim
2020-06-17 0:02 ` Kees Cook
2020-06-19 19:32 ` Brendan Higgins
2020-06-19 18:17 ` Frank Rowand
2020-06-14 18:17 ` Kees Cook
2020-06-15 17:45 ` Bird, Tim
2020-06-15 18:44 ` Kees Cook
2020-06-14 18:39 ` Kees Cook
2020-06-15 19:07 ` Bird, Tim
2020-06-16 12:08 ` Paolo Bonzini
2020-06-16 16:42 ` Bird, Tim
2020-06-16 19:44 ` Brendan Higgins
2020-06-16 20:30 ` Bird, Tim
2020-06-16 23:58 ` Kees Cook
2020-06-19 18:47 ` Frank Rowand
2020-06-19 19:11 ` Kees Cook
2020-06-19 22:58 ` Paolo Bonzini
2020-06-20 14:51 ` Frank Rowand
2020-06-19 18:33 ` Frank Rowand
2020-06-19 17:58 ` Frank Rowand
2020-06-20 6:44 ` David Gow
2020-06-20 15:03 ` Frank Rowand
2020-06-23 2:58 ` David Gow
2020-06-16 23:52 ` Kees Cook
2020-06-19 18:52 ` Frank Rowand
2020-06-19 19:50 ` Brendan Higgins
2020-06-19 19:49 ` Frank Rowand
2020-06-16 20:48 ` Brendan Higgins [this message]
2020-06-16 21:16 ` Bird, Tim
2020-06-16 21:19 ` Bird, Tim
2020-06-17 0:06 ` Kees Cook
2020-06-17 2:30 ` Bird, Tim
2020-06-17 3:36 ` Kees Cook
2020-06-17 4:05 ` David Gow
2020-06-19 19:44 ` Brendan Higgins
2020-06-19 20:19 ` Frank Rowand
2020-06-19 23:47 ` Bird, Tim
2020-06-19 19:39 ` Brendan Higgins
2020-06-19 17:13 ` Frank Rowand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200616204817.GA212825@google.com \
--to=brendanhiggins@google.com \
--cc=Tim.Bird@sony.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=shuah@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).