From: Frank Rowand <frowand.list@gmail.com>
To: Kees Cook <keescook@chromium.org>,
Brendan Higgins <brendanhiggins@google.com>
Cc: "Bird, Tim" <Tim.Bird@sony.com>,
Paolo Bonzini <pbonzini@redhat.com>,
"shuah@kernel.org" <shuah@kernel.org>,
"linux-kselftest@vger.kernel.org"
<linux-kselftest@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
David Gow <davidgow@google.com>
Subject: Re: RFC - kernel selftest result documentation (KTAP)
Date: Fri, 19 Jun 2020 13:47:29 -0500 [thread overview]
Message-ID: <398200b2-f8bc-894d-6d6f-366ff98a490e@gmail.com> (raw)
In-Reply-To: <202006161653.15C278A5@keescook>
On 2020-06-16 18:58, Kees Cook wrote:
> On Tue, Jun 16, 2020 at 12:44:28PM -0700, Brendan Higgins wrote:
>> On Tue, Jun 16, 2020 at 9:42 AM Bird, Tim <Tim.Bird@sony.com> wrote:
>>>> From: Paolo Bonzini <pbonzini@redhat.com>
>>>>
>>>> On 15/06/20 21:07, Bird, Tim wrote:
>>>>>> Note: making the plan line required differs from TAP13 and TAP14. I
>>>>>> think it's the right choice, but we should be clear.
>>>>
>>>> As an aside, where is TAP14?
>>> By TAP14, I was referring to the current, undocumented, KUnit
>>> conventions.
>>
>> Not so. TAP14 is the proposed next version of TAP13:
>>
>> https://github.com/TestAnything/testanything.github.io/pull/36
>> https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md
>
> I was reading this (I haven't compared to the blob above):
>
> https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md
>
>> Based on the discussion, it seems like most of the things we wanted
>> from TAP14 would probably make it in if TAP ever accepts another pull
>> request.
>
> Were our leading diagnostic lines part of their latest spec? I thought
> we were pretty far off in left field for that particular bit.
>
>>> My personal preference is to have the dash. I think it's more human readable.
>>> I note that the TAP spec has examples of result lines both with and without
>>> the dash, so even the spec is ambiguous on this. I think not mandating it
>>> either way is probably best. For regex parsers, it's easy to ignore with '[-]?'
>>> outside the pattern groups that grab the number and description.
>>
>> I don't think we care, because we don't use it.
>
> Yeah, I'm in the same place. I don't care -- I would just like a
> determination. (The "implied" nature of it in TAP14 bothers me.)
>
>>>> XFAIL/XPASS are different from SKIP. I personally don't have a need for
>>>> them, but kselftests includes XFAIL/XPASS exit codes and they aren't
>>>> reflected into selftests/kselftest/runner.sh.
>>>>
>>>> Likewise, kselftest.h has ksft_inc_xfail_cnt but not
>>>> ksft_test_result_xfail/ksft_test_result_xpass.
>
> I proposed fixing that recently[1]. seccomp uses XFAIL for "I have
> detected you lack the config to test this, so I can't say it's working
> or not, because it only looks like a failure without the config."
Based on that description, the case sounds like it should be a skip.
Or if the entire test depends on the missing config then Bail out might
be appropriate.
>
> [1] https://lore.kernel.org/lkml/20200611224028.3275174-7-keescook@chromium.org/
>
next prev parent reply other threads:[~2020-06-19 18:47 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-10 18:11 RFC - kernel selftest result documentation (KTAP) Bird, Tim
2020-06-13 5:07 ` David Gow
2020-06-15 17:34 ` Bird, Tim
2020-06-16 20:03 ` Brendan Higgins
2020-06-16 20:37 ` Bird, Tim
2020-06-17 0:02 ` Kees Cook
2020-06-19 19:32 ` Brendan Higgins
2020-06-19 18:17 ` Frank Rowand
2020-06-14 18:17 ` Kees Cook
2020-06-15 17:45 ` Bird, Tim
2020-06-15 18:44 ` Kees Cook
2020-06-14 18:39 ` Kees Cook
2020-06-15 19:07 ` Bird, Tim
2020-06-16 12:08 ` Paolo Bonzini
2020-06-16 16:42 ` Bird, Tim
2020-06-16 19:44 ` Brendan Higgins
2020-06-16 20:30 ` Bird, Tim
2020-06-16 23:58 ` Kees Cook
2020-06-19 18:47 ` Frank Rowand [this message]
2020-06-19 19:11 ` Kees Cook
2020-06-19 22:58 ` Paolo Bonzini
2020-06-20 14:51 ` Frank Rowand
2020-06-19 18:33 ` Frank Rowand
2020-06-19 17:58 ` Frank Rowand
2020-06-20 6:44 ` David Gow
2020-06-20 15:03 ` Frank Rowand
2020-06-23 2:58 ` David Gow
2020-06-16 23:52 ` Kees Cook
2020-06-19 18:52 ` Frank Rowand
2020-06-19 19:50 ` Brendan Higgins
2020-06-19 19:49 ` Frank Rowand
2020-06-16 20:48 ` Brendan Higgins
2020-06-16 21:16 ` Bird, Tim
2020-06-16 21:19 ` Bird, Tim
2020-06-17 0:06 ` Kees Cook
2020-06-17 2:30 ` Bird, Tim
2020-06-17 3:36 ` Kees Cook
2020-06-17 4:05 ` David Gow
2020-06-19 19:44 ` Brendan Higgins
2020-06-19 20:19 ` Frank Rowand
2020-06-19 23:47 ` Bird, Tim
2020-06-19 19:39 ` Brendan Higgins
2020-06-19 17:13 ` Frank Rowand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=398200b2-f8bc-894d-6d6f-366ff98a490e@gmail.com \
--to=frowand.list@gmail.com \
--cc=Tim.Bird@sony.com \
--cc=brendanhiggins@google.com \
--cc=davidgow@google.com \
--cc=keescook@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=pbonzini@redhat.com \
--cc=shuah@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).