Re: [PATCH 1/7] test-lib: add a "test_expect_todo", similar to "test_expect_failure"

From: Junio C Hamano <gitster@pobox.com>
To: Derrick Stolee <derrickstolee@github.com>
Cc: "Phillip Wood" <phillip.wood123@gmail.com>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	git@vger.kernel.org, "John Cai" <johncai86@gmail.com>,
	"Elijah Newren" <newren@gmail.com>,
	"Derrick Stolee" <stolee@gmail.com>
Subject: Re: [PATCH 1/7] test-lib: add a "test_expect_todo", similar to "test_expect_failure"
Date: Wed, 23 Mar 2022 15:13:06 -0700	[thread overview]
Message-ID: <xmqqy210pbbh.fsf@gitster.g> (raw)
In-Reply-To: <a9478c1b-4441-0c79-332d-a7c5592dbc70@github.com> (Derrick Stolee's message of "Tue, 22 Mar 2022 10:43:23 -0400")

Derrick Stolee <derrickstolee@github.com> writes:

> The thing I'm hoping to see from a final version is that a top-level
> helper like test_expect_todo will expect at least one test_todo
> helper to be executed inside of the test (perhaps communicated by
> setting a special GIT_TEST_* environment variable?) and if any of

I was hoping that we can do without test_expect_todo.
test_expect_success can turn itself into test_expect_todo when it
sees test_todo is invoked even once in it.  And Phillip's outline
actually implements the idea, if I am not mistaken.

> the test_todo lines change from fail to pass, then that is
> communicated as a _failure_ from CI's perspective. Let us discover
> if we have accidentally "fixed" any of these test cases and update
> the tests accordingly.

In other words, we do not want to lose the "TODO fixed" we have been
getting out of test_expect_failure, which I agree with.  I am not
sure if Phillip's outline had that feature.

> I can predict writing a test case with multiple test_todo lines
> that need to be updated to drop the test_todo helpers one-by-one
> as a change is being introduced.

Yes.