From: Brendan Higgins <brendanhiggins@google.com>
To: "Bird, Tim" <Tim.Bird@sony.com>, Alan Maguire <alan.maguire@oracle.com>
Cc: "shuah@kernel.org" <shuah@kernel.org>,
"linux-kselftest@vger.kernel.org"
<linux-kselftest@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
Kees Cook <keescook@chromium.org>,
Paolo Bonzini <pbonzini@redhat.com>,
David Gow <davidgow@google.com>
Subject: Re: RFC - kernel selftest result documentation (KTAP)
Date: Fri, 19 Jun 2020 12:39:53 -0700 [thread overview]
Message-ID: <CAFd5g44fueJUMYxWoEa6YEW_9+LjZ7XWQQB17e1cQtZ911KBWg@mail.gmail.com> (raw)
In-Reply-To: <CY4PR13MB1175A71AE3937C0786721ABFFD9D0@CY4PR13MB1175.namprd13.prod.outlook.com>
On Tue, Jun 16, 2020 at 2:16 PM Bird, Tim <Tim.Bird@sony.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Brendan Higgins
> >
> > On Wed, Jun 10, 2020 at 06:11:06PM +0000, Bird, Tim wrote:
> > > Some months ago I started work on a document to formalize how
> > > kselftest implements the TAP specification. However, I didn't finish
> > > that work. Maybe it's time to do so now.
> > >
> > > kselftest has developed a few differences from the original
> > > TAP specification, and some extensions that I believe are worth
> > > documenting.
> > >
> > > Essentially, we have created our own KTAP (kernel TAP)
> > > format. I think it is worth documenting our conventions, in order to
> > > keep everyone on the same page.
> > >
> > > Below is a partially completed document on my understanding
> > > of KTAP, based on examination of some of the kselftest test
> > > output. I have not reconciled this with the kunit output format,
> > > which I believe has some differences (which maybe we should
> > > resolve before we get too far into this).
> > >
> > > I submit the document now, before it is finished, because a patch
> > > was recently introduced to alter one of the result conventions
> > > (from SKIP='not ok' to SKIP='ok').
> > >
> > > See the document include inline below
> > >
> > > ====== start of ktap-doc-rfc.txt ======
> >
> > [...]
> >
> > > --- from here on is not-yet-organized material
> > >
> > > Tip:
> > > - don't change the test plan based on skipped tests.
> > > - it is better to report that a test case was skipped, than to
> > > not report it
> > > - that is, don't adjust the number of test cases based on skipped
> > > tests
> > >
> > > Other things to mention:
> > > TAP13 elements not used:
> > > - yaml for diagnostic messages
> >
> > We talked about this before, but I would like some way to get failed
> > expectation/assertion information in the test in a consistent machine
> > parsible way. Currently we do the following:
> >
> > # Subtest: example
> > 1..1
> > # example_simple_test: initializing
> > # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29
> > Expected 1 + 1 == 3, but
> > 1 + 1 == 2
> > 3 == 3
> > not ok 1 - example_simple_test
> > not ok 5 - example
> >
> > Technically not TAP compliant, but no one seems to mind. I am okay with
> > keeping it the way it is, but if we don't want it in the KTAP spec, we
> > will need some kind of recourse.
>
> So far, most of the CI systems don't parse out diagnostic data, so it doesn't
> really matter what the format is. If it's useful for humans, it's valuable as is.
> However, it would be nice if that could change. But without some formalization
> of the format of the diagnostic data, it's an intractable problem for CI systems
> to parse it. So it's really a chicken and egg problem. To solve it, we would have
> to determine what exactly needs to be provided on a consistent basis for diagnostic
> data across many tests. I think that it's too big a problem to handle right now.
> I'm not opposed to migrating to some structure with yaml in the future, but free
> form text output seems OK for now.
Well as long as everyone is cool with it for now we can put the
problem for later.
> > > - reason: try to keep things line-based, since output from other things
> > > may be interspersed with messages from the test itself
> > > - TODO directive
> >
> > Is this more of stating a fact or desire? We don't use TODO either, but
> > it looks like it could be useful.
> Just stating a fact. I didn't find TODO in either KUnit or selftest in
> November when I initially wrote this up. If TODO serves as a kind
> of XFAIL, it could be useful. I have nothing against it.
Fair enough.
> > > KTAP Extensions beyond TAP13:
> > > - nesting
> > > - via indentation
> > > - indentation makes it easier for humans to read
> > > - test identifier
> > > - multiple parts, separated by ':'
> >
> > Can you elabroate on this more? I am not sure what you mean.
> An individual test case can have a name that is scoped by a containing
> test or test suite. For example: selftests: cpufreq: main.sh
> This test identifier consists of the test system (selftests), the test
> area (cpufreq), and the test case name (main.sh). This one's a bit
> weird because the test case name is just the name of the program
> in that test area. The program itself doesn't output data in TAP format,
> and the harness uses it's exit code to detect PASS/FAIL. if main.sh had
> multiple test cases, it might produce test identifiers like this:
> selftests: cpufreq: main: check_change_afinity_mask
> selftests: cpufreq: main: check_permissions_for_mask_operation
> (Or it might just produce the last part of these strings, the
> testcase names, and the testcase id might be something generated
> by the harness or CI system.)
+Alan Maguire
Aha, that is something we (Alan, David, Kees, and myself) were talking
about on another thread:
https://lore.kernel.org/linux-kselftest/CABVgOSnjrzfFOMF0VE1-5Ks-e40fc0XZsNZ92jE60ZOhYmZWog@mail.gmail.com/T/#m682be9f9103f7b363b702e49c137d83a4833fcae
I think that makes a lot of sense if it isn't too hard in practice.
> The value of having a single string to identify the testcase (like a
> uniform resource locator), is that it's easier to use the string to
> correlate results produced from different CI system that are executing
> the same test.
Makes sense.
> > > - summary lines
> > > - can be skipped by CI systems that do their own calculations
> > >
> > > Other notes:
> > > - automatic assignment of result status based on exit code
> > >
> > > Tips:
> > > - do NOT describe the result in the test line
> > > - the test case description should be the same whether the test
> > > succeeds or fails
> > > - use diagnostic lines to describe or explain results, if this is
> > > desirable
> > > - test numbers are considered harmful
> > > - test harnesses should use the test description as the identifier
> > > - test numbers change when testcases are added or removed
> > > - which means that results can't be compared between different
> > > versions of the test
> > > - recommendations for diagnostic messages:
> > > - reason for failure
> > > - reason for skip
> > > - diagnostic data should always preceding the result line
> > > - problem: harness may emit result before test can do assessment
> > > to determine reason for result
> > > - this is what the kernel uses
> > >
> > > Differences between kernel test result format and TAP13:
> > > - in KTAP the "# SKIP" directive is placed after the description on
> > > the test result line
> > >
> > > ====== start of ktap-doc-rfc.txt ======
> > > OK - that's the end of the RFC doc.
> > >
> > > Here are a few questions:
> > > - is this document desired or not?
> > > - is it too long or too short?
> > > - if the document is desired, where should it be placed?
> >
> > I like it. I don't think we can rely on the TAP people updating their
> > stuff based on my interactions with them. So having a spec which is
> > actually maintained would be nice.
> >
> > Maybe in Documentation/dev-tools/ ?
> I'm leaning towards Documentation/dev-tools/test-results_format.rst
SGTM.
> > > I assume somewhere under Documentation, and put into
> > > .rst format. Suggestions for a name and location are welcome.
> > > - is this document accurate?
> > > I think KUNIT does a few things differently than this description.
> > > - is the intent to have kunit and kselftest have the same output format?
> > > if so, then these should be rationalized.
> >
> > Yeah, I think it would be nice if all test frameworks/libraries for the
> > kernel output tests in the same language.
> Agreed.
Cheers
next prev parent reply other threads:[~2020-06-19 19:40 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-10 18:11 RFC - kernel selftest result documentation (KTAP) Bird, Tim
2020-06-13 5:07 ` David Gow
2020-06-15 17:34 ` Bird, Tim
2020-06-16 20:03 ` Brendan Higgins
2020-06-16 20:37 ` Bird, Tim
2020-06-17 0:02 ` Kees Cook
2020-06-19 19:32 ` Brendan Higgins
2020-06-19 18:17 ` Frank Rowand
2020-06-14 18:17 ` Kees Cook
2020-06-15 17:45 ` Bird, Tim
2020-06-15 18:44 ` Kees Cook
2020-06-14 18:39 ` Kees Cook
2020-06-15 19:07 ` Bird, Tim
2020-06-16 12:08 ` Paolo Bonzini
2020-06-16 16:42 ` Bird, Tim
2020-06-16 19:44 ` Brendan Higgins
2020-06-16 20:30 ` Bird, Tim
2020-06-16 23:58 ` Kees Cook
2020-06-19 18:47 ` Frank Rowand
2020-06-19 19:11 ` Kees Cook
2020-06-19 22:58 ` Paolo Bonzini
2020-06-20 14:51 ` Frank Rowand
2020-06-19 18:33 ` Frank Rowand
2020-06-19 17:58 ` Frank Rowand
2020-06-20 6:44 ` David Gow
2020-06-20 15:03 ` Frank Rowand
2020-06-23 2:58 ` David Gow
2020-06-16 23:52 ` Kees Cook
2020-06-19 18:52 ` Frank Rowand
2020-06-19 19:50 ` Brendan Higgins
2020-06-19 19:49 ` Frank Rowand
2020-06-16 20:48 ` Brendan Higgins
2020-06-16 21:16 ` Bird, Tim
2020-06-16 21:19 ` Bird, Tim
2020-06-17 0:06 ` Kees Cook
2020-06-17 2:30 ` Bird, Tim
2020-06-17 3:36 ` Kees Cook
2020-06-17 4:05 ` David Gow
2020-06-19 19:44 ` Brendan Higgins
2020-06-19 20:19 ` Frank Rowand
2020-06-19 23:47 ` Bird, Tim
2020-06-19 19:39 ` Brendan Higgins [this message]
2020-06-19 17:13 ` Frank Rowand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAFd5g44fueJUMYxWoEa6YEW_9+LjZ7XWQQB17e1cQtZ911KBWg@mail.gmail.com \
--to=brendanhiggins@google.com \
--cc=Tim.Bird@sony.com \
--cc=alan.maguire@oracle.com \
--cc=davidgow@google.com \
--cc=keescook@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=pbonzini@redhat.com \
--cc=shuah@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).