Re: [Qemu-devel] [PATCH 2/2] test: replace gtester with a TAP driver

[Paolo Bonzini <pbonzini@redhat.com>]

On 30/11/18 16:50, Daniel P. Berrangé wrote:
> IIUC, this output is only seen when you with 'make check' passing V=1,
> so most people won't see it.

V=1 should be enabled by default on Patchew, so it will be visible in
the CI logs.  But anyway that's minor, there are more reasons why a
custom beautifier is better than prove.

First, at the very least, failures should be included in the output even
if V=1 is not provided.  Therefore, for failing tests we'd have:

  $ scripts/tap-driver.pl < f.tap
  FAIL 2 bbb
  XPASS 3 ccc # TODO
  FAIL 6 fff # SKIP cannot find frobnicator

And the output of prove in that case would be inferior for developers,
unless you use "V=1":

  $ prove -e cat f.tap
  f.tap .. Failed 2/6 subtests
	(less 2 skipped subtests: 2 okay)
	(1 TODO test unexpectedly succeeded)

  Test Summary Report
  -------------------
  f.tap (Wstat: 0 Tests: 6 Failed: 2)
    Failed tests:  2, 6
    TODO passed:   3
  Files=1, Tests=6,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
  Result: FAIL

where you don't really know what tests 2 and 6 are without looking at
the TAP output.  The part before the summary provides a nice progress
report, as you point out, but it adds a bunch of noise with respect to
skipped subtests (skipped tests will be more common once patch 1 is
merged; qgraph also adds some) that I personally dislike.

You can hide it with -Q, but then you cannot invoke anymore more than
one test with a single invocation of prove, and the Makefile output
becomes much bigger:

   TEST    check-foo

vs.

   TEST    check-foo
  All tests successful.
  Files=1, Tests=3,  0 wallclock secs ( 0.01 usr +  0.00 sys =  0.01 CPU)
  Result: PASS

There are two other smaller "philosophical" differences.  First, prove
treats XPASS as passes, while tap-driver.pl treats them as failures.
Second, skipped tests really stand out in the "prove -v" output, with
white-on-blue text.

I'd rather avoid getting into the business of beautifying the output of
prove.

Paolo