All of lore.kernel.org
 help / color / mirror / Atom feed
From: "SZEDER Gábor" <szeder.dev@gmail.com>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Cc: git@vger.kernel.org, Junio C Hamano <gitster@pobox.com>,
	Jeff King <peff@peff.net>,
	Johannes Schindelin <Johannes.Schindelin@gmx.de>
Subject: Re: [PATCH 0/6 + 1] test-lib: make --verbose output valid TAP
Date: Tue, 9 Mar 2021 23:07:17 +0100	[thread overview]
Message-ID: <20210309220717.GC1016223@szeder.dev> (raw)
In-Reply-To: <87h7lkhwed.fsf@evledraar.gmail.com>

On Tue, Mar 09, 2021 at 10:03:54PM +0100, Ævar Arnfjörð Bjarmason wrote:
> 
> On Tue, Mar 09 2021, SZEDER Gábor wrote:
> 
> > On Tue, Mar 09, 2021 at 05:02:12PM +0100, Ævar Arnfjörð Bjarmason wrote:
> >> The eventual goal not included in this series is to have multiple
> >> output targets, and e.g. convert ci/print-test-failures.sh to use a
> >> TAP parser.
> >> 
> >> Machine-readable "TAP --verbose -x" output can bring us a lot of nice
> >> things down the line, I have some local WIP code that's a smarter
> >> version of ci/print-test-failures.sh that knows how to spew out only
> >> the output relevant to the failing test(s).
> >
> > I wonder what you mean by the word "relevant" here, as I can't imagine
> > how you could possibly identify what is relevant for a failing test
> > and what isn't.  If you didn't at all meant "relevant", but that it
> > will show only the output of the failing test(s), then this is a bad
> > example.  Our test cases depend too much on previous test cases, and a
> > failure of one test can be the result of a change in a previous
> > successful tests.
> >
> > Therefore, any such change to 'ci/print-test-failures.sh' will have my
> > firm NACK.
> 
> On e.g. the github CI every step in the run is an collapsable button, so
> we could have our cake and eat it too here.

Well, one of the things I didn't like in GitHub CI is that I had to
click a lot to get to the information I wanted...

> It seems to me like a sane default would be to have an equivalent to
> "print-test-failures.sh" that only prints the --verbose output for the
> failing tests, and a "print-all-output-for-failing-tests.sh" or whatever
> which gave you the full output.
> 
> Even then, there seem to me to be some low hanging fruit for
> abbreviating even that output. E.g. if you we have 100 tests and we
> failed only on the 5th, isn't the --verbose -x output up to and
> including the 5th going to be enough, or do we need it for the other 95?

I have a test fix waiting to be sent out, where a failing git command
in the 'test_when_finished' block of test #21 went unnoticed because
of the broken && chain, and ultimately caused the failure of test
#91.

In my opinion your proposed changes to 'ci/print-test-failures.sh'
would make things worse.


  reply	other threads:[~2021-03-09 22:07 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-10-21  4:14 Prove "Tests out of sequence" Error Lars Schneider
2016-10-21  6:10 ` Stefan Beller
2016-10-21  8:20   ` Jeff King
2016-10-21  8:43     ` Jeff King
2016-10-21 10:41       ` [PATCH 0/3] fix travis TAP/--verbose conflict Jeff King
2016-10-21 10:42         ` [PATCH 1/3] test-lib: handle TEST_OUTPUT_DIRECTORY with spaces Jeff King
2016-10-21 10:48         ` [PATCH 2/3] test-lib: add --verbose-log option Jeff King
2016-10-21 17:12           ` Junio C Hamano
2016-10-21 21:46             ` Jeff King
2021-02-28 20:25           ` [PATCH/RFC] test-lib: make --verbose work under prove Ævar Arnfjörð Bjarmason
2021-03-01  9:51             ` Jeff King
2021-03-01 13:54               ` Ævar Arnfjörð Bjarmason
2021-03-09 16:02                 ` [PATCH 0/6 + 1] test-lib: make --verbose output valid TAP Ævar Arnfjörð Bjarmason
2021-03-09 17:52                   ` SZEDER Gábor
2021-03-09 21:03                     ` Ævar Arnfjörð Bjarmason
2021-03-09 22:07                       ` SZEDER Gábor [this message]
2021-03-09 16:02                 ` [PATCH 1/7] test-lib: remove test_external Ævar Arnfjörð Bjarmason
2021-03-10  1:04                   ` Junio C Hamano
2021-03-10  2:22                     ` Ævar Arnfjörð Bjarmason
2021-03-09 16:02                 ` [PATCH 2/7] test-lib: add say_color_tap helper to emit TAP format Ævar Arnfjörð Bjarmason
2021-03-10  0:39                   ` Junio C Hamano
2021-03-09 16:02                 ` [PATCH 3/7] test-lib: color "ok" TAP directives green under --verbose (or -x) Ævar Arnfjörð Bjarmason
2021-03-09 16:02                 ` [PATCH 4/7] test-lib: add tee with TAP support to test-tool Ævar Arnfjörð Bjarmason
2021-03-09 16:02                 ` [PATCH 5/7] test-lib: indent and format GIT_TEST_TEE_OUTPUT_FILE code Ævar Arnfjörð Bjarmason
2021-03-09 16:02                 ` [PATCH 6/7] test-lib: make --verbose output valid TAP Ævar Arnfjörð Bjarmason
2021-03-09 18:59                   ` SZEDER Gábor
2021-03-09 20:57                     ` Ævar Arnfjörð Bjarmason
2021-03-09 21:31                       ` SZEDER Gábor
2021-03-10  2:35                         ` Ævar Arnfjörð Bjarmason
2021-03-16  9:10                           ` Ævar Arnfjörð Bjarmason
2021-03-09 19:12                   ` SZEDER Gábor
2021-03-10  1:11                   ` Junio C Hamano
2021-03-10  7:42                   ` Chris Torek
2021-03-09 16:02                 ` [RFC/PATCH 7/7] test-lib: generate JUnit output via TAP Ævar Arnfjörð Bjarmason
2021-03-19 14:14                   ` Johannes Schindelin
2021-03-21  0:28                     ` Ævar Arnfjörð Bjarmason
2021-03-22 13:46                       ` Johannes Schindelin
2016-10-21 10:48         ` [PATCH 3/3] travis: use --verbose-log test option Jeff King
2016-10-21 17:19         ` [PATCH 0/3] fix travis TAP/--verbose conflict Stefan Beller
2016-10-24 18:06         ` Lars Schneider
2016-10-21 15:29       ` Prove "Tests out of sequence" Error Jacob Keller
2016-10-21 15:35         ` Jeff King
2016-10-21 15:42           ` Jacob Keller
2016-10-21 15:48             ` Jeff King
2016-10-21 16:15               ` Jacob Keller
2016-10-22  4:45                 ` [PATCH 4/3] test-lib: bail out when "-v" used under "prove" Jeff King
2016-10-22  5:25                   ` Jacob Keller

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210309220717.GC1016223@szeder.dev \
    --to=szeder.dev@gmail.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.