archive mirror
 help / color / mirror / Atom feed
From: Paolo Bonzini <>
To: "Bird, Tim" <>,
	Shuah Khan <>,
	"" <>,
	"" <>
Cc: "" <>,
Subject: Re: [PATCH] kselftest: runner: fix TAP output for skipped tests
Date: Wed, 10 Jun 2020 20:22:51 +0200	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On 10/06/20 19:43, Bird, Tim wrote:
>>>   		(rc=$?;	\
>>>   		if [ $rc -eq $skip_rc ]; then	\
>>> -			echo "not ok $test_num $TEST_HDR_MSG # SKIP"
>>> +			echo "ok $test_num $TEST_HDR_MSG # SKIP"
> This is a pretty big change, and might break upstream CIs that have come to
> rely on kselftest's existing behavior.  I know it's going to break Fuego's parsing
> of results.

Do you have a pointer to this code?

> kselftest has a few conventions that are different from the TAP spec, 
> and a few items it does that are extensions to the TAP spec.

Yes, there are extensions to directives are not a problem and parsers
might raise an error on them.  That can be an issue, but it's a separate
one (and it's easier to ignore it as long as test pass...).

> IMHO, the TAP spec got this one wrong, but I could be convinced
> otherwise.

Here the TAP spec says that a skip starts with "ok" and has a "SKIP"
directive, and anyone can parse it to treat as it as a failure if
desirable.  But doing something else should be treated simply as a
violation of the spec, it's not a matter of "right" or "wrong".

So, if you want to use "not ok ... # SKIP", don't call it TAP.

However, I noticed now that there is another instance of "not ok.*SKIP"
in testing/selftests/kselftest.h (and also one in a comment).  So they
should all be fixed at the same time, and I'm okay with holding this patch.


> But I think we should discuss this among CI users of
> kselftest before making the change.
> I started work quite a while ago on an effort to document the
> conventions used by kselftest (particularly where it deviates
> from the TAP spec),  but never submitted it.
> I'm going to submit what I've got as an RFC now, for discussion,
> even though it's not finished.  I'll do that in a separate thread.
>>>   		elif [ $rc -eq $timeout_rc ]; then \
>>>   			echo "#"
>>>   			echo "not ok $test_num $TEST_HDR_MSG # TIMEOUT"
>> Thanks. I will pull this in for Linux 5.8-rc2
> Shuah - can you hold off on this until we discuss it?
> Thanks,
>  -- Tim

  reply	other threads:[~2020-06-10 18:23 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-10 15:44 Paolo Bonzini
2020-06-10 16:19 ` Shuah Khan
2020-06-10 17:43   ` Bird, Tim
2020-06-10 18:22     ` Paolo Bonzini [this message]
2020-06-15 19:28       ` Bird, Tim
2020-06-10 20:02     ` Shuah Khan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \
    --subject='Re: [PATCH] kselftest: runner: fix TAP output for skipped tests' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).