linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Bird, Tim" <Tim.Bird@sony.com>
To: Shuah Khan <skhan@linuxfoundation.org>,
	Paolo Bonzini <pbonzini@redhat.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>
Cc: "shuah@kernel.org" <shuah@kernel.org>,
	"linux-kselftest@vger.kernel.org"
	<linux-kselftest@vger.kernel.org>
Subject: RE: [PATCH] kselftest: runner: fix TAP output for skipped tests
Date: Wed, 10 Jun 2020 17:43:27 +0000	[thread overview]
Message-ID: <CY4PR13MB1175A17F29B281642DF05A6DFD830@CY4PR13MB1175.namprd13.prod.outlook.com> (raw)
In-Reply-To: <ac2c1eaa-acd7-7ac6-0666-6e6c0cbd546b@linuxfoundation.org>



> -----Original Message-----
> From: linux-kselftest-owner@vger.kernel.org <linux-kselftest-owner@vger.kernel.org> On Behalf Of Shuah Khan
> 
> On 6/10/20 9:44 AM, Paolo Bonzini wrote:
> > According to the TAP specification, a skipped test must be marked as "ok"
> > and annotated with the SKIP directive, for example
> >
> >     ok 23 # skip Insufficient flogiston pressure.
> >     (https://testanything.org/tap-specification.html)
> >
> > Fix the runner script to match this.
> >
> > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> > ---
> >   tools/testing/selftests/kselftest/runner.sh | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
> > index 676b3a8b114d..f4815cbcd60f 100644
> > --- a/tools/testing/selftests/kselftest/runner.sh
> > +++ b/tools/testing/selftests/kselftest/runner.sh
> > @@ -77,7 +77,7 @@ run_one()
> >   		echo "ok $test_num $TEST_HDR_MSG") ||
> >   		(rc=$?;	\
> >   		if [ $rc -eq $skip_rc ]; then	\
> > -			echo "not ok $test_num $TEST_HDR_MSG # SKIP"
> > +			echo "ok $test_num $TEST_HDR_MSG # SKIP"

This is a pretty big change, and might break upstream CIs that have come to
rely on kselftest's existing behavior.  I know it's going to break Fuego's parsing
of results.

kselftest has a few conventions that are different from the TAP spec, 
and a few items it does that are extensions to the TAP spec.
IMHO, the TAP spec got this one wrong, but I could be convinced
otherwise.  But I think we should discuss this among CI users of
kselftest before making the change.

I started work quite a while ago on an effort to document the
conventions used by kselftest (particularly where it deviates
from the TAP spec),  but never submitted it.

I'm going to submit what I've got as an RFC now, for discussion,
even though it's not finished.  I'll do that in a separate thread.


> >   		elif [ $rc -eq $timeout_rc ]; then \
> >   			echo "#"
> >   			echo "not ok $test_num $TEST_HDR_MSG # TIMEOUT"
> >
> 
> Thanks. I will pull this in for Linux 5.8-rc2
Shuah - can you hold off on this until we discuss it?

Thanks,
 -- Tim


  reply	other threads:[~2020-06-10 17:43 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-10 15:44 Paolo Bonzini
2020-06-10 16:19 ` Shuah Khan
2020-06-10 17:43   ` Bird, Tim [this message]
2020-06-10 18:22     ` Paolo Bonzini
2020-06-15 19:28       ` Bird, Tim
2020-06-10 20:02     ` Shuah Khan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CY4PR13MB1175A17F29B281642DF05A6DFD830@CY4PR13MB1175.namprd13.prod.outlook.com \
    --to=tim.bird@sony.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=pbonzini@redhat.com \
    --cc=shuah@kernel.org \
    --cc=skhan@linuxfoundation.org \
    --subject='RE: [PATCH] kselftest: runner: fix TAP output for skipped tests' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).