[PATCH] t7518: fix flaky grep invocation

[PATCH] t7518: fix flaky grep invocation

[Elijah Newren via GitGitGadget]

From: Elijah Newren <newren@gmail.com>

t7518.1 added in commit 862e80a413 ("ident: handle NULL email when
complaining of empty name", 2017-02-23), was trying to make sure that
the test with an empty ident did not segfault and did not result in
glibc quiety translating a NULL pointer into a name of "(null)".  It did
the latter by ensuring that a grep for "null" didn't appear in the
output, but on one automatic CI run I observed the following output:

fatal: empty ident name (for <runner@fv-az128-670.gcliasfzo2nullsdbrimjtbyhg.cx.internal.cloudapp.net>) not allowed

Note that 'null' appears as a substring of the domain name, found
within 'gcliasfzo2nullsdbrimjtbyhg'.  Tighten the test by searching for
"(null)" rather than "null".

Signed-off-by: Elijah Newren <newren@gmail.com>
---
    t7518: fix flaky grep invocation

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-884%2Fnewren%2Ffix-flaky-t7518-grep-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-884/newren/fix-flaky-t7518-grep-v1
Pull-Request: https://github.com/git/git/pull/884

 t/t7518-ident-corner-cases.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t7518-ident-corner-cases.sh b/t/t7518-ident-corner-cases.sh
index dc3e9c8c88..905957bd0a 100755
--- a/t/t7518-ident-corner-cases.sh
+++ b/t/t7518-ident-corner-cases.sh
@@ -13,7 +13,7 @@ test_expect_success 'empty name and missing email' '
 		sane_unset GIT_AUTHOR_EMAIL &&
 		GIT_AUTHOR_NAME= &&
 		test_must_fail git commit --allow-empty -m foo 2>err &&
-		test_i18ngrep ! null err
+		test_i18ngrep ! "(null)" err
 	)
 '
 

base-commit: d4a392452e292ff924e79ec8458611c0f679d6d4
-- 
gitgitgadget

Re: [PATCH] t7518: fix flaky grep invocation

[Junio C Hamano]

"Elijah Newren via GitGitGadget" <gitgitgadget@gmail.com> writes:

> output, but on one automatic CI run I observed the following output:
>
> fatal: empty ident name (for <runner@fv-az128-670.gcliasfzo2nullsdbrimjtbyhg.cx.internal.cloudapp.net>) not allowed
>
> Note that 'null' appears as a substring of the domain name, found
> within 'gcliasfzo2nullsdbrimjtbyhg'.  Tighten the test by searching for
> "(null)" rather than "null".

Why do we even need grep again?  If we were to segfault, wouldn't
test_must_fail catch it for us?

... ahh, OK, your libc may allow NULL left in email and that is what
we are trying to catch?

Honestly, I am not sure if it even makes sense to test it like this.
The code that the test tries to protect against future breakages
roughly look like this these days.

	if (!email) {
		... one way to assign non-NULL to email ...
	}
	if (!email) {
		... do other things to assign non-NULL to email or die ...
	}

	if (want_name) {
		... here we require email to be set because we show
		... it in an error message
	}

The original problem was that the code had "if (want_name)" part
first before email==NULL condition has been dealt with, and used
email==NULL in one of the error messages.  The fix was to move the
part that deals with email==NULL up, as it does not need any of the
effects that happen in "if (want_name)" block.

Now, what is this particular test protecting against?  We may again
move the "if (want_name)" block up and trigger the error message in
it that uses e-mail while having NULL in email again?  If so, would
it make more sense to do something along this line, lose the "grep"
and keep "test_must_fail"?  We expect and require at this point in
the code (the patch is inside that "if (want_name)" block) email
must have been set up already, so it would be a BUG() otherwise.

Hmm?

 ident.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git i/ident.c w/ident.c
index 6aba4b5cb6..9b71563d95 100644
--- i/ident.c
+++ w/ident.c
@@ -430,6 +430,8 @@ const char *fmt_ident(const char *name, const char *email,
 			if (strict) {
 				if (using_default)
 					ident_env_hint(whose_ident);
+				if (!email)
+					BUG("NULL email when we need to complain???");
 				die(_("empty ident name (for <%s>) not allowed"), email);
 			}
 			pw = xgetpwuid_self(NULL);

Re: [PATCH] t7518: fix flaky grep invocation

[Elijah Newren]

On Fri, Oct 16, 2020 at 5:02 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Elijah Newren via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > output, but on one automatic CI run I observed the following output:
> >
> > fatal: empty ident name (for <runner@fv-az128-670.gcliasfzo2nullsdbrimjtbyhg.cx.internal.cloudapp.net>) not allowed
> >
> > Note that 'null' appears as a substring of the domain name, found
> > within 'gcliasfzo2nullsdbrimjtbyhg'.  Tighten the test by searching for
> > "(null)" rather than "null".
>
> Why do we even need grep again?  If we were to segfault, wouldn't
> test_must_fail catch it for us?
>
> ... ahh, OK, your libc may allow NULL left in email and that is what
> we are trying to catch?

Yes, as mentioned in the commit message:

    ...was trying to make sure that
    the test with an empty ident did not segfault and did not result in
    glibc quiety translating a NULL pointer into a name of "(null)".

This was also taken from the comment in the file just above the code
in question (though it was just barely outside the context region).

> Honestly, I am not sure if it even makes sense to test it like this.
> The code that the test tries to protect against future breakages
> roughly look like this these days.
>
>         if (!email) {
>                 ... one way to assign non-NULL to email ...
>         }
>         if (!email) {
>                 ... do other things to assign non-NULL to email or die ...
>         }
>
>         if (want_name) {
>                 ... here we require email to be set because we show
>                 ... it in an error message
>         }
>
> The original problem was that the code had "if (want_name)" part
> first before email==NULL condition has been dealt with, and used
> email==NULL in one of the error messages.  The fix was to move the
> part that deals with email==NULL up, as it does not need any of the
> effects that happen in "if (want_name)" block.
>
> Now, what is this particular test protecting against?  We may again
> move the "if (want_name)" block up and trigger the error message in
> it that uses e-mail while having NULL in email again?  If so, would
> it make more sense to do something along this line, lose the "grep"
> and keep "test_must_fail"?  We expect and require at this point in
> the code (the patch is inside that "if (want_name)" block) email
> must have been set up already, so it would be a BUG() otherwise.
>
> Hmm?

It may come down to the angle you view it from; "the current code
doesn't have such a problem and can't within its newer design, so why
test it?" vs. "when you add a regression test at the time you make a
bugfix, the regression test should demonstrate the failure with the
old version of the code".  Without the grep, the test would not have
failed on the original old version of the code, at least not on some
platforms.
I'm guessing that's why Peff put it there, and still had a preference
to leave the grep in (see his comments on this issue at
https://lore.kernel.org/git/20201016200031.GA3355643@coredump.intra.peff.net/;
this is a case where it'd be nice if gitgitgadget accepted an
in-reply-to flag of some sort).

However, although I have a slight preference to fixing what grep is
searching for, my primary concern was with avoiding flakiness in the
test.  I'm happy if someone wants to make additional code changes or
tweak the testcase here so long as we avoid the flakiness.

>  ident.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git i/ident.c w/ident.c
> index 6aba4b5cb6..9b71563d95 100644
> --- i/ident.c
> +++ w/ident.c
> @@ -430,6 +430,8 @@ const char *fmt_ident(const char *name, const char *email,
>                         if (strict) {
>                                 if (using_default)
>                                         ident_env_hint(whose_ident);
> +                               if (!email)
> +                                       BUG("NULL email when we need to complain???");
>                                 die(_("empty ident name (for <%s>) not allowed"), email);
>                         }
>                         pw = xgetpwuid_self(NULL);

Re: [PATCH] t7518: fix flaky grep invocation

[Taylor Blau]

Hi Elijah,

On Fri, Oct 16, 2020 at 11:39:54PM +0000, Elijah Newren via GitGitGadget wrote:
> From: Elijah Newren <newren@gmail.com>
>
> t7518.1 added in commit 862e80a413 ("ident: handle NULL email when
> complaining of empty name", 2017-02-23), was trying to make sure that
> the test with an empty ident did not segfault and did not result in
> glibc quiety translating a NULL pointer into a name of "(null)".  It did
> the latter by ensuring that a grep for "null" didn't appear in the
> output, but on one automatic CI run I observed the following output:

:-). I have always been surprised by glibc's behavior here; I'd almost
always rather have a segfault than silently turning a NULL pointer into
"(null)" when formatted with "%s".

> fatal: empty ident name (for <runner@fv-az128-670.gcliasfzo2nullsdbrimjtbyhg.cx.internal.cloudapp.net>) not allowed
>
> Note that 'null' appears as a substring of the domain name, found
> within 'gcliasfzo2nullsdbrimjtbyhg'.  Tighten the test by searching for
> "(null)" rather than "null".

Yep; that sounds like a straightforward fix that will make this test
non-flaky.

  Reviewed-by: Taylor Blau <me@ttaylorr.com>

Thanks,
Taylor