linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Shuah Khan <skhan@linuxfoundation.org>
To: "Bird, Tim" <Tim.Bird@sony.com>,
	Paolo Bonzini <pbonzini@redhat.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>
Cc: "shuah@kernel.org" <shuah@kernel.org>,
	"linux-kselftest@vger.kernel.org"
	<linux-kselftest@vger.kernel.org>,
	Shuah Khan <skhan@linuxfoundation.org>
Subject: Re: [PATCH] kselftest: runner: fix TAP output for skipped tests
Date: Wed, 10 Jun 2020 14:02:37 -0600	[thread overview]
Message-ID: <fbce656a-fdf0-7662-2fcf-4f0daae8cf33@linuxfoundation.org> (raw)
In-Reply-To: <CY4PR13MB1175A17F29B281642DF05A6DFD830@CY4PR13MB1175.namprd13.prod.outlook.com>

On 6/10/20 11:43 AM, Bird, Tim wrote:
> 
> 
>> -----Original Message-----
>> From: linux-kselftest-owner@vger.kernel.org <linux-kselftest-owner@vger.kernel.org> On Behalf Of Shuah Khan
>>
>> On 6/10/20 9:44 AM, Paolo Bonzini wrote:
>>> According to the TAP specification, a skipped test must be marked as "ok"
>>> and annotated with the SKIP directive, for example
>>>
>>>      ok 23 # skip Insufficient flogiston pressure.
>>>      (https://testanything.org/tap-specification.html)
>>>
>>> Fix the runner script to match this.
>>>
>>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>>> ---
>>>    tools/testing/selftests/kselftest/runner.sh | 2 +-
>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
>>> index 676b3a8b114d..f4815cbcd60f 100644
>>> --- a/tools/testing/selftests/kselftest/runner.sh
>>> +++ b/tools/testing/selftests/kselftest/runner.sh
>>> @@ -77,7 +77,7 @@ run_one()
>>>    		echo "ok $test_num $TEST_HDR_MSG") ||
>>>    		(rc=$?;	\
>>>    		if [ $rc -eq $skip_rc ]; then	\
>>> -			echo "not ok $test_num $TEST_HDR_MSG # SKIP"
>>> +			echo "ok $test_num $TEST_HDR_MSG # SKIP"
> 
> This is a pretty big change, and might break upstream CIs that have come to
> rely on kselftest's existing behavior.  I know it's going to break Fuego's parsing
> of results.
> 

Thanks for chiming in. We don't want to break CI workflow.

> kselftest has a few conventions that are different from the TAP spec,
> and a few items it does that are extensions to the TAP spec.
> IMHO, the TAP spec got this one wrong, but I could be convinced
> otherwise.  But I think we should discuss this among CI users of
> kselftest before making the change.
> 
> I started work quite a while ago on an effort to document the
> conventions used by kselftest (particularly where it deviates
> from the TAP spec),  but never submitted it.
> 
> I'm going to submit what I've got as an RFC now, for discussion,
> even though it's not finished.  I'll do that in a separate thread.
> 
> 
>>>    		elif [ $rc -eq $timeout_rc ]; then \
>>>    			echo "#"
>>>    			echo "not ok $test_num $TEST_HDR_MSG # TIMEOUT"
>>>
>>
>> Thanks. I will pull this in for Linux 5.8-rc2
> Shuah - can you hold off on this until we discuss it?
> 

Of course. Thanks for getting my attention before I pulled it in.

thanks,
-- Shuah


      parent reply	other threads:[~2020-06-10 20:02 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-10 15:44 Paolo Bonzini
2020-06-10 16:19 ` Shuah Khan
2020-06-10 17:43   ` Bird, Tim
2020-06-10 18:22     ` Paolo Bonzini
2020-06-15 19:28       ` Bird, Tim
2020-06-10 20:02     ` Shuah Khan [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=fbce656a-fdf0-7662-2fcf-4f0daae8cf33@linuxfoundation.org \
    --to=skhan@linuxfoundation.org \
    --cc=Tim.Bird@sony.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=pbonzini@redhat.com \
    --cc=shuah@kernel.org \
    --subject='Re: [PATCH] kselftest: runner: fix TAP output for skipped tests' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).