nolibc: KTAP output and test reports

nolibc: KTAP output and test reports

[Thomas Weißschuh]

Hi Willy, Zhangjin,

after your recent discussions about the test output and report I
wondered if it would make sense to switch nolibc-test to KTAP output
format [0].

With this it would be possible to have a wrapper script run each
architecture test as its own test subcomponent.
A (K)TAP parser/runner could then directly recognize and report failing
testcases, making it easier to validate.

Also maybe we can hook it up into the regular kselftests setup and have
the bots run it as part of that.

The kernel even includes a header-only library to implement the format [1].
It also should be fairly easy to emit the format without a library.

Thomas

[0] Documentation/dev-tools/ktap.rst
[1] Documentation/dev-tools/kselftest.rst (Test harness)

nolibc: KTAP output and test reports

[Zhangjin Wu]

Hi, Thomas

> Hi Willy, Zhangjin,
> 
> after your recent discussions about the test output and report I
> wondered if it would make sense to switch nolibc-test to KTAP output
> format [0].

Just read the doc, it looks really good.

> 
> With this it would be possible to have a wrapper script run each
> architecture test as its own test subcomponent.
> A (K)TAP parser/runner could then directly recognize and report failing
> testcases, making it easier to validate.
>

Yeah, this is what can we benefit from the standard format.

> Also maybe we can hook it up into the regular kselftests setup and have
> the bots run it as part of that.
>

I did take a look at the other kselftests cases, seems lots of cases use
qemu to run the tests, perhaps we can share some of them. Not sure if
there are some libraries work on qemu test support, therefore, we can
reuse them.

> The kernel even includes a header-only library to implement the format [1].
> It also should be fairly easy to emit the format without a library.
>

Perhaps we can learn and discuss on how to use them at first, I'm a newbie to
both of them, but I'm really interested in running nolibc in the kselftest
framework ;-)

Thanks,
Zhangjin

> Thomas
> 
> [0] Documentation/dev-tools/ktap.rst
> [1] Documentation/dev-tools/kselftest.rst (Test harness)

Re: nolibc: KTAP output and test reports

[Thomas Weißschuh]

Hi Willy,

On 2023-06-08 00:15:27+0200, Thomas Weißschuh wrote:
> after your recent discussions about the test output and report I
> wondered if it would make sense to switch nolibc-test to KTAP output
> format [0].
> 
> With this it would be possible to have a wrapper script run each
> architecture test as its own test subcomponent.
> A (K)TAP parser/runner could then directly recognize and report failing
> testcases, making it easier to validate.
> 
> Also maybe we can hook it up into the regular kselftests setup and have
> the bots run it as part of that.
> 
> The kernel even includes a header-only library to implement the format [1].
> It also should be fairly easy to emit the format without a library.

Did you have a chance to look at this?

If you are not categorically opposed I would create a proof of concept
for further discussion.

> [0] Documentation/dev-tools/ktap.rst
> [1] Documentation/dev-tools/kselftest.rst (Test harness)

Re: nolibc: KTAP output and test reports

[Willy Tarreau]

Hi Thomas,

On Tue, Jul 18, 2023 at 09:31:06AM +0200, Thomas Weißschuh wrote:
> Hi Willy,
> 
> On 2023-06-08 00:15:27+0200, Thomas Weißschuh wrote:
> > after your recent discussions about the test output and report I
> > wondered if it would make sense to switch nolibc-test to KTAP output
> > format [0].
> > 
> > With this it would be possible to have a wrapper script run each
> > architecture test as its own test subcomponent.
> > A (K)TAP parser/runner could then directly recognize and report failing
> > testcases, making it easier to validate.
> > 
> > Also maybe we can hook it up into the regular kselftests setup and have
> > the bots run it as part of that.
> > 
> > The kernel even includes a header-only library to implement the format [1].
> > It also should be fairly easy to emit the format without a library.
> 
> Did you have a chance to look at this?
>
> If you are not categorically opposed I would create a proof of concept
> for further discussion.

I remember I had a quick look but had no opinion about it: it's not very
clear to me how this will be consumed because I don't know the tools
around and cannot invest time learning them. If you think it can bring
some value without complicating the usage, maintenance and contribution
of what we currently have, maybe let's give it a try. But I would like
to be sure we are careful about preserving the current ease of use, as
I'd hate to have to go through the makefile to figure how to get back
the simple output format that we can trivially read or grep without any
extra tools. That might for example mean that we'd need an option to
change the output format (and I think it's possible because most of the
outputs are done inside wrappers).

But in any case, feel free to explore, experiment and share your findings.
Hoping this helps,
Willy