git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Derrick Stolee <derrickstolee@github.com>
To: Junio C Hamano <gitster@pobox.com>,
	Phillip Wood <phillip.wood123@gmail.com>
Cc: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	git@vger.kernel.org, "John Cai" <johncai86@gmail.com>,
	"Elijah Newren" <newren@gmail.com>,
	"Derrick Stolee" <stolee@gmail.com>
Subject: Re: [PATCH 1/7] test-lib: add a "test_expect_todo", similar to "test_expect_failure"
Date: Tue, 22 Mar 2022 10:43:23 -0400	[thread overview]
Message-ID: <a9478c1b-4441-0c79-332d-a7c5592dbc70@github.com> (raw)
In-Reply-To: <xmqqy2144hwa.fsf@gitster.g>

On 3/20/22 2:07 PM, Junio C Hamano wrote:
> Phillip Wood <phillip.wood123@gmail.com> writes:
> 
>> .... I had a look
>> at some of the conversions with your test_todo --want/--expect/--reset
>> and found the result really hard to follow. Junio's suggestions chimed
>> with some things I've been thinking about so I had a go at
>> implementing it and doing some sample conversions (see below). Marking
>> the individual commands that should fail is a big step forward and the
>> failing commands are checked to make sure they don't segfault etc.
> 
> ;-)
> 
> Another small detail in my suggestion that will make a huge
> difference in the end is not to introduce test_expect_todo as a
> separate top-level construct, and instead teach test_expect_success
> to pay attention to the use of test_todo "unfortunately this does
> not work yet" mark in it.  It allows us to use test_todo in a shared
> test helper function and call them in test_expect_success, and when
> the step the test helper has trouble with gets fixed, the "unmark"
> step will be an isolated change.
> 
> Your sample change seems to already have it, which is good.

After reading this thread, I agree with many of the ideas that were
generated in response to this topic.

The thing I'm hoping to see from a final version is that a top-level
helper like test_expect_todo will expect at least one test_todo
helper to be executed inside of the test (perhaps communicated by
setting a special GIT_TEST_* environment variable?) and if any of
the test_todo lines change from fail to pass, then that is
communicated as a _failure_ from CI's perspective. Let us discover
if we have accidentally "fixed" any of these test cases and update
the tests accordingly.

I can predict writing a test case with multiple test_todo lines
that need to be updated to drop the test_todo helpers one-by-one
as a change is being introduced.

Thanks,
-Stolee

  reply	other threads:[~2022-03-22 14:44 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-03-18  0:33 [PATCH 0/7] test-lib-functions: a better "test_expect_failure" Ævar Arnfjörð Bjarmason
2022-03-18  0:33 ` [PATCH 1/7] test-lib: add a "test_expect_todo", similar to "test_expect_failure" Ævar Arnfjörð Bjarmason
2022-03-18 18:59   ` Junio C Hamano
2022-03-18 20:50     ` Junio C Hamano
2022-03-18 23:07       ` Ævar Arnfjörð Bjarmason
2022-03-19  7:12         ` Junio C Hamano
2022-03-19 11:11           ` Ævar Arnfjörð Bjarmason
2022-03-20 15:13             ` Phillip Wood
2022-03-20 18:07               ` Junio C Hamano
2022-03-22 14:43                 ` Derrick Stolee [this message]
2022-03-23 22:13                   ` Junio C Hamano
2022-03-24 11:24                     ` Phillip Wood
2022-03-18  0:33 ` [PATCH 2/7] test-lib-functions: add and use a "test_todo" helper Ævar Arnfjörð Bjarmason
2022-03-18  0:33 ` [PATCH 3/7] tests: allow test_* in "test_must_fail_acceptable" for "test_todo" Ævar Arnfjörð Bjarmason
2022-03-18  0:33 ` [PATCH 4/7] test-lib-functions: add and use a "todo_test_cmp" helper Ævar Arnfjörð Bjarmason
2022-03-18  0:34 ` [PATCH 5/7] test-lib-functions: add and use a "todo_test_path" helper Ævar Arnfjörð Bjarmason
2022-03-18  0:34 ` [PATCH 6/7] test-lib-functions: make test_todo support a --reset Ævar Arnfjörð Bjarmason
2022-03-18  0:34 ` [PATCH 7/7] sparse tests: convert a TODO test to use "test_expect_todo" Ævar Arnfjörð Bjarmason

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a9478c1b-4441-0c79-332d-a7c5592dbc70@github.com \
    --to=derrickstolee@github.com \
    --cc=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=johncai86@gmail.com \
    --cc=newren@gmail.com \
    --cc=phillip.wood123@gmail.com \
    --cc=stolee@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).