[PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[ZheNing Hu via GitGitGadget]

From: ZheNing Hu <adlternative@gmail.com>

Johannes Schindelin seems to have introduced a bug in
cc72385f(for-each-ref: let upstream/push optionally
report the remote name), it use `atom->u.remote_ref.option`
which is a member of enumeration in the judgment statement.
When we use other members in the enumeration `used_atom.u`,
and it happened to fill in `remote_ref.push`, this judgment
may still be established and produces errors. So replace the
judgment statement with `starts_with(name, "push")` to fix
the error.

Signed-off-by: ZheNing Hu <adlternative@gmail.com>
---
    [GSOC] ref-filter: solve bugs caused by enumeration
    
    This is the problem I found in the implementation of git for-each-ref
    --format="%(notes)".
    
    I added a new option to the enum used_atom.u , but the program seems to
    output the value of "%(push)".
    
    After research, it should be the problem of incorrect use of enumeration
    values here.
    
    Thanks, may the force be with you!

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-949%2Fadlternative%2Fref-filter-enum-bug-fix-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-949/adlternative/ref-filter-enum-bug-fix-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/949

 ref-filter.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ref-filter.c b/ref-filter.c
index a0adb4551d87..f467f2fbbb73 100644
--- a/ref-filter.c
+++ b/ref-filter.c
@@ -1730,7 +1730,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
 			else
 				v->s = xstrdup("");
 			continue;
-		} else if (atom->u.remote_ref.push) {
+		} else if (starts_with(name, "push")) {
 			const char *branch_name;
 			v->s = xstrdup("");
 			if (!skip_prefix(ref->refname, "refs/heads/",

base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a
-- 
gitgitgadget

Re: [PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[Junio C Hamano]

"ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: ZheNing Hu <adlternative@gmail.com>
>
> Johannes Schindelin seems to have introduced a bug in
> cc72385f(for-each-ref: let upstream/push optionally
> report the remote name), it use `atom->u.remote_ref.option`
> which is a member of enumeration in the judgment statement.

Sorry but I am not sure if our readers would understand what "a
member of enumeration in the judgment statement" is (I certainly do
not), and even more importantly, "bugs caused by enumeration" on the
title does not hint much about what problem the patch is trying to
solve.

> When we use other members in the enumeration `used_atom.u`,
> and it happened to fill in `remote_ref.push`, this judgment
> may still be established and produces errors. So replace the
> judgment statement with `starts_with(name, "push")` to fix
> the error.

And this paragraph does not enlighten all that much, unfortunately.

Is it that a check refers to one member of a union without making
sure that member is the one in effect in the union?  I am most
puzzled by the mention of "enumeration" when there does not appear
to be any enum involved.

> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-949%2Fadlternative%2Fref-filter-enum-bug-fix-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-949/adlternative/ref-filter-enum-bug-fix-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/949
>
>  ref-filter.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/ref-filter.c b/ref-filter.c
> index a0adb4551d87..f467f2fbbb73 100644
> --- a/ref-filter.c
> +++ b/ref-filter.c
> @@ -1730,7 +1730,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
>  			else
>  				v->s = xstrdup("");
>  			continue;
> -		} else if (atom->u.remote_ref.push) {
> +		} else if (starts_with(name, "push")) {
>  			const char *branch_name;
>  			v->s = xstrdup("");
>  			if (!skip_prefix(ref->refname, "refs/heads/",
>
> base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a

Re: [PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[ZheNing Hu]

Junio C Hamano <gitster@pobox.com> 于2021年5月6日周四 上午9:53写道：
>
> "ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > From: ZheNing Hu <adlternative@gmail.com>
> >
> > Johannes Schindelin seems to have introduced a bug in
> > cc72385f(for-each-ref: let upstream/push optionally
> > report the remote name), it use `atom->u.remote_ref.option`
> > which is a member of enumeration in the judgment statement.
>
> Sorry but I am not sure if our readers would understand what "a
> member of enumeration in the judgment statement" is (I certainly do
> not), and even more importantly, "bugs caused by enumeration" on the
> title does not hint much about what problem the patch is trying to
> solve.
>
> > When we use other members in the enumeration `used_atom.u`,
> > and it happened to fill in `remote_ref.push`, this judgment
> > may still be established and produces errors. So replace the
> > judgment statement with `starts_with(name, "push")` to fix
> > the error.
>
> And this paragraph does not enlighten all that much, unfortunately.
>
> Is it that a check refers to one member of a union without making
> sure that member is the one in effect in the union?  I am most
> puzzled by the mention of "enumeration" when there does not appear
> to be any enum involved.
>

Sorry, I didn't make it clear. I re-describe the problem first, and then
modify the commit messages.

Suppose we are dealing with "%(notes)", the name member of our
`used_atom` item at this time is "notes", and its member union `u` uses
a `struct notes_option`, we fill some values in `used_atom.u.notes_option`,

When we traverse in `used_atom` array in `populate_value()` and previous
judgement like "if (starts_with(name, "refname"))" will failed, because we
are dealing with atom "notes", but in judgement "else if
(atom->u.remote_ref.push)",
The value we fill in `used_atom.u.notes_option` just makes
`used_atom.u.remote_ref.push` non-zero. This leads us into the wrong case.

Is this clearer?

Thanks.
--
ZheNing Hu

Re: [PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[Junio C Hamano]

ZheNing Hu <adlternative@gmail.com> writes:

>> Is it that a check refers to one member of a union without making
>> sure that member is the one in effect in the union?  I am most
>> puzzled by the mention of "enumeration" when there does not appear
>> to be any enum involved.
>
> Sorry, I didn't make it clear. I re-describe the problem first, and then
> modify the commit messages.
>
> Suppose we are dealing with "%(notes)", the name member of our
> `used_atom` item at this time is "notes", and its member union `u` uses
> a `struct notes_option`, we fill some values in `used_atom.u.notes_option`,
>
> When we traverse in `used_atom` array in `populate_value()` and previous
> judgement like "if (starts_with(name, "refname"))" will failed, because we
> are dealing with atom "notes", but in judgement "else if
> (atom->u.remote_ref.push)",
> The value we fill in `used_atom.u.notes_option` just makes
> `used_atom.u.remote_ref.push` non-zero. This leads us into the wrong case.
>
> Is this clearer?

This time you avoided the word enumeration, and it made it clearer.
The word commonly used is "condition" where you said "judgment", I
think, and wit that it would probably be even more clear.

used_atom.u is an union, and it has different members depending on
what atom the auxiliary data the union part of the "struct
used_atom" wants to record.  At most only one of the members can be
valid at any one time.  Since the code checks u.remote_ref without
even making sure if the atom is "push" or "push:" (which are only
two cases that u.remote_ref.push becomes valid), but u.remote_ref
shares the same storage for other members of the union, the check
was reading from an invalid member, which was the bug.

So the fix was to see what atom it is by checking its name member?
Is starts_with() a reliable test?  A fictitious atom "pushe" will be
different from "push" or "push:<something>", but will still pass
that test, so from the point of view of future-proofing the tests,
shouldn't it do the same check as the one at the beginning of
remote_ref_atom_parser()?

Thanks.

Re: [PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[ZheNing Hu]

> This time you avoided the word enumeration, and it made it clearer.
> The word commonly used is "condition" where you said "judgment", I
> think, and wit that it would probably be even more clear.
>

OK.

> used_atom.u is an union, and it has different members depending on
> what atom the auxiliary data the union part of the "struct
> used_atom" wants to record.  At most only one of the members can be
> valid at any one time.  Since the code checks u.remote_ref without
> even making sure if the atom is "push" or "push:" (which are only
> two cases that u.remote_ref.push becomes valid), but u.remote_ref
> shares the same storage for other members of the union, the check
> was reading from an invalid member, which was the bug.
>

Yes, that's what it means. I got it wrong before, of course used_atom.u
is not a enum, it's union.

> So the fix was to see what atom it is by checking its name member?
> Is starts_with() a reliable test?  A fictitious atom "pushe" will be
> different from "push" or "push:<something>", but will still pass
> that test, so from the point of view of future-proofing the tests,
> shouldn't it do the same check as the one at the beginning of
> remote_ref_atom_parser()?
>

I think it's not necesssary. Before we call `populate_value()`,
`parse_ref_filter_atom()` only allow user use atom which match
 "valid_atom" entry fully, something like "pushe" will be rejected
and process die with "fatal: unknown field name: pushe", so it didn't
pass this "starts_with()" test.

        /* Is the atom a valid one? */
        for (i = 0; i < ARRAY_SIZE(valid_atom); i++) {
                int len = strlen(valid_atom[i].name);
                if (len == atom_len && !memcmp(valid_atom[i].name, sp, len))
                        break;
        }

        if (ARRAY_SIZE(valid_atom) <= i)
                return strbuf_addf_ret(err, -1, _("unknown field name: %.*s"),
                                       (int)(ep-atom), atom);

> Thanks.

Thanks.
--
ZheNing Hu

Re: [PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[Junio C Hamano]

ZheNing Hu <adlternative@gmail.com> writes:

>> So the fix was to see what atom it is by checking its name member?
>> Is starts_with() a reliable test?  A fictitious atom "pushe" will be
>> different from "push" or "push:<something>", but will still pass
>> that test, so from the point of view of future-proofing the tests,
>> shouldn't it do the same check as the one at the beginning of
>> remote_ref_atom_parser()?
>>
>
> I think it's not necesssary. Before we call `populate_value()`, ...

I do not care if it is not needed with the "current" set of atoms we
accept.  The example "pushe", which obviously will not be accepted
by today's code, was all about future-proofing.

Re: [PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[ZheNing Hu]

Junio C Hamano <gitster@pobox.com> 于2021年5月6日周四 下午7:20写道：
>
> ZheNing Hu <adlternative@gmail.com> writes:
>
> >> So the fix was to see what atom it is by checking its name member?
> >> Is starts_with() a reliable test?  A fictitious atom "pushe" will be
> >> different from "push" or "push:<something>", but will still pass
> >> that test, so from the point of view of future-proofing the tests,
> >> shouldn't it do the same check as the one at the beginning of
> >> remote_ref_atom_parser()?
> >>
> >
> > I think it's not necesssary. Before we call `populate_value()`, ...
>
> I do not care if it is not needed with the "current" set of atoms we
> accept.  The example "pushe", which obviously will not be accepted
> by today's code, was all about future-proofing.

Well, what you said makes sense, now I think something like this will be
more secure. In the future, other people may be grateful for such a choice. :)

@@ -1730,7 +1759,7 @@ static int populate_value(struct ref_array_item
*ref, struct strbuf *err)
                        else
                                v->s = xstrdup("");
                        continue;
-               } else if (starts_with(name, "push")) {
+               } else if (starts_with(name, "push") &&
atom->u.remote_ref.push) {


--
ZheNing Hu

Re: [PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[Junio C Hamano]

ZheNing Hu <adlternative@gmail.com> writes:

> ... now I think something like this will be more secure. In the
> future, other people may be grateful for such a choice. :)
>
> @@ -1730,7 +1759,7 @@ static int populate_value(struct ref_array_item
> *ref, struct strbuf *err)
>                         else
>                                 v->s = xstrdup("");
>                         continue;
> -               } else if (starts_with(name, "push")) {
> +               } else if (starts_with(name, "push") &&
> atom->u.remote_ref.push) {

Hmph, I do no think that would be futureproof at all.  If a new atom
"pushe" gets added, it is not all that unlikely that it would add
its own member to the same union.  name here would be "pushe" and
starts with "push", and upon parsing "pushe", its own member in the
union, atom->u.X, would have been assigned to, but the code we see
here still accesses atom->u.remote_ref.*, so you still have the same
problem you started to solve, no?

The check we use in remote_ref_atom_parser() to see if the atom is
about pushing, i.e.

	if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:"))

is unlikely to be invalidated in future changes, as this is very
much end-user facing and changing the condition will break existing
scripts, so that is what I was expecting you to use instead.

Re: [PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[ZheNing Hu]

> Hmph, I do no think that would be futureproof at all.  If a new atom
> "pushe" gets added, it is not all that unlikely that it would add
> its own member to the same union.  name here would be "pushe" and
> starts with "push", and upon parsing "pushe", its own member in the
> union, atom->u.X, would have been assigned to, but the code we see
> here still accesses atom->u.remote_ref.*, so you still have the same
> problem you started to solve, no?
>

Well, this "pushe" has `atom->u.X` which is different from "push" and
"push:". This is indeed worth worrying about.

> The check we use in remote_ref_atom_parser() to see if the atom is
> about pushing, i.e.
>
>         if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:"))
>
> is unlikely to be invalidated in future changes, as this is very
> much end-user facing and changing the condition will break existing
> scripts, so that is what I was expecting you to use instead.
>

But I am afraid that the cost we paid for string matching here is too high,
Since the string matching here is to illustrate that we use one of the members
of the union, why can't we directly add a "union_type" member to used_atom?
if we have this, we can just switch the "union_type" flag and also make the
program correct. Perhaps this would be better than the large number of string
matches have done here.

>
>

Thanks.
--
ZheNing Hu

Re: [PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[Junio C Hamano]

ZheNing Hu <adlternative@gmail.com> writes:

> But I am afraid that the cost we paid for string matching here is too high,

If that is truly the concern (I do not know without measuring),
perhaps we should add a member next to the union to say which one of
the union members is valid, so that you can say

    if (atom->atom_type == ATOM_TYPE_REMOTE_REF &&
        atom->u.remote_ref.push)

(introduce an enum and define ATOM_TYPE_* after the member in the
union).

That would help futureproofing the code even further, as a new
synonym of "push" introduced laster [*] would not invalidate the check you are
adding there.


[Footnote]

* remote_ref_atom_parser() in the future may begin like so:

-	if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:"))
+	if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:") ||
+           !strcmp(atom->name, "a-synonym-for-push"))
		atom->u.remote_ref.push = 1;

Re: [PATCH] [GSOC] ref-filter: solve bugs caused by enumeration

[ZheNing Hu]

> If that is truly the concern (I do not know without measuring),
> perhaps we should add a member next to the union to say which one of
> the union members is valid, so that you can say
>
>     if (atom->atom_type == ATOM_TYPE_REMOTE_REF &&
>         atom->u.remote_ref.push)
>
> (introduce an enum and define ATOM_TYPE_* after the member in the
> union).
>

Yes, I think so. Since the content of this part needs to be modified for
 the parsing of all atoms, I will put it in a separate topic to complete.

> That would help futureproofing the code even further, as a new
> synonym of "push" introduced laster [*] would not invalidate the check you are
> adding there.
>

Yes, this enhances its generalization ability.

>
> [Footnote]
>
> * remote_ref_atom_parser() in the future may begin like so:
>
> -       if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:"))
> +       if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:") ||
> +           !strcmp(atom->name, "a-synonym-for-push"))
>                 atom->u.remote_ref.push = 1;
>

--
ZheNing Hu

[PATCH v2] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu via GitGitGadget]

From: ZheNing Hu <adlternative@gmail.com>

used_atom.u is an union, and it has different members depending on
what atom the auxiliary data the union part of the "struct
used_atom" wants to record.  At most only one of the members can be
valid at any one time.  Since the code checks u.remote_ref without
even making sure if the atom is "push" or "push:" (which are only
two cases that u.remote_ref.push becomes valid), but u.remote_ref
shares the same storage for other members of the union, the check
was reading from an invalid member, which was the bug.

Modify the condition here to first check whether the atom name
starts with "push", and then check u.remote_ref, to avoid reading
the value of invalid member of the union.

Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: ZheNing Hu <adlternative@gmail.com>
---
    [GSOC] ref-filter: fix read invalid union member bug
    
    Change from last version: Modified the documentation description with
    the help of Junio. And modify the processing method of the condition:
    check whether the name of the atom starts with "push" and whether
    u.remote_ref is non-zero.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-949%2Fadlternative%2Fref-filter-enum-bug-fix-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-949/adlternative/ref-filter-enum-bug-fix-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/949

Range-diff vs v1:

 1:  e51ca176f76b ! 1:  0e1923c9d722 [GSOC] ref-filter: solve bugs caused by enumeration
     @@ Metadata
      Author: ZheNing Hu <adlternative@gmail.com>
      
       ## Commit message ##
     -    [GSOC] ref-filter: solve bugs caused by enumeration
     +    [GSOC] ref-filter: fix read invalid union member bug
      
     -    Johannes Schindelin seems to have introduced a bug in
     -    cc72385f(for-each-ref: let upstream/push optionally
     -    report the remote name), it use `atom->u.remote_ref.option`
     -    which is a member of enumeration in the judgment statement.
     -    When we use other members in the enumeration `used_atom.u`,
     -    and it happened to fill in `remote_ref.push`, this judgment
     -    may still be established and produces errors. So replace the
     -    judgment statement with `starts_with(name, "push")` to fix
     -    the error.
     +    used_atom.u is an union, and it has different members depending on
     +    what atom the auxiliary data the union part of the "struct
     +    used_atom" wants to record.  At most only one of the members can be
     +    valid at any one time.  Since the code checks u.remote_ref without
     +    even making sure if the atom is "push" or "push:" (which are only
     +    two cases that u.remote_ref.push becomes valid), but u.remote_ref
     +    shares the same storage for other members of the union, the check
     +    was reading from an invalid member, which was the bug.
      
     +    Modify the condition here to first check whether the atom name
     +    starts with "push", and then check u.remote_ref, to avoid reading
     +    the value of invalid member of the union.
     +
     +    Helped-by: Junio C Hamano <gitster@pobox.com>
          Signed-off-by: ZheNing Hu <adlternative@gmail.com>
      
       ## ref-filter.c ##
     @@ ref-filter.c: static int populate_value(struct ref_array_item *ref, struct strbu
       				v->s = xstrdup("");
       			continue;
      -		} else if (atom->u.remote_ref.push) {
     -+		} else if (starts_with(name, "push")) {
     ++		} else if (starts_with(name, "push") && atom->u.remote_ref.push) {
       			const char *branch_name;
       			v->s = xstrdup("");
       			if (!skip_prefix(ref->refname, "refs/heads/",


 ref-filter.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ref-filter.c b/ref-filter.c
index a0adb4551d87..750b25914b82 100644
--- a/ref-filter.c
+++ b/ref-filter.c
@@ -1730,7 +1730,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
 			else
 				v->s = xstrdup("");
 			continue;
-		} else if (atom->u.remote_ref.push) {
+		} else if (starts_with(name, "push") && atom->u.remote_ref.push) {
 			const char *branch_name;
 			v->s = xstrdup("");
 			if (!skip_prefix(ref->refname, "refs/heads/",

base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a
-- 
gitgitgadget

[PATCH v3] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu via GitGitGadget]

From: ZheNing Hu <adlternative@gmail.com>

used_atom.u is an union, and it has different members depending on
what atom the auxiliary data the union part of the "struct
used_atom" wants to record. At most only one of the members can be
valid at any one time. Since the code checks u.remote_ref without
even making sure if the atom is "push" or "push:" (which are only
two cases that u.remote_ref.push becomes valid), but u.remote_ref
shares the same storage for other members of the union, the check
was reading from an invalid member, which was the bug.

Modify the condition here to check whether the atom name
equals to "push" or starts with "push:", to avoid reading the
value of invalid member of the union.

Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: ZheNing Hu <adlternative@gmail.com>
---
    [GSOC] ref-filter: fix read invalid union member bug
    
    Change from last version:
    Modify the processing method of the condition: check whether the name of
    the atom equals to "push" or starts with "pushs", which can enhanced
    security, although it may bring string match overhead.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-949%2Fadlternative%2Fref-filter-enum-bug-fix-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-949/adlternative/ref-filter-enum-bug-fix-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/949

Range-diff vs v2:

 1:  0e1923c9d722 ! 1:  21cf7a44e168 [GSOC] ref-filter: fix read invalid union member bug
     @@ Commit message
      
          used_atom.u is an union, and it has different members depending on
          what atom the auxiliary data the union part of the "struct
     -    used_atom" wants to record.  At most only one of the members can be
     -    valid at any one time.  Since the code checks u.remote_ref without
     +    used_atom" wants to record. At most only one of the members can be
     +    valid at any one time. Since the code checks u.remote_ref without
          even making sure if the atom is "push" or "push:" (which are only
          two cases that u.remote_ref.push becomes valid), but u.remote_ref
          shares the same storage for other members of the union, the check
          was reading from an invalid member, which was the bug.
      
     -    Modify the condition here to first check whether the atom name
     -    starts with "push", and then check u.remote_ref, to avoid reading
     -    the value of invalid member of the union.
     +    Modify the condition here to check whether the atom name
     +    equals to "push" or starts with "push:", to avoid reading the
     +    value of invalid member of the union.
      
          Helped-by: Junio C Hamano <gitster@pobox.com>
          Signed-off-by: ZheNing Hu <adlternative@gmail.com>
     @@ ref-filter.c: static int populate_value(struct ref_array_item *ref, struct strbu
       				v->s = xstrdup("");
       			continue;
      -		} else if (atom->u.remote_ref.push) {
     -+		} else if (starts_with(name, "push") && atom->u.remote_ref.push) {
     ++		} else if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:")) {
       			const char *branch_name;
       			v->s = xstrdup("");
       			if (!skip_prefix(ref->refname, "refs/heads/",


 ref-filter.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ref-filter.c b/ref-filter.c
index a0adb4551d87..213d3773ada3 100644
--- a/ref-filter.c
+++ b/ref-filter.c
@@ -1730,7 +1730,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
 			else
 				v->s = xstrdup("");
 			continue;
-		} else if (atom->u.remote_ref.push) {
+		} else if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:")) {
 			const char *branch_name;
 			v->s = xstrdup("");
 			if (!skip_prefix(ref->refname, "refs/heads/",

base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a
-- 
gitgitgadget

Re: [PATCH v3] [GSOC] ref-filter: fix read invalid union member bug

[Junio C Hamano]

"ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: ZheNing Hu <adlternative@gmail.com>
>
> used_atom.u is an union, and it has different members depending on
> what atom the auxiliary data the union part of the "struct
> used_atom" wants to record. At most only one of the members can be
> valid at any one time. Since the code checks u.remote_ref without
> even making sure if the atom is "push" or "push:" (which are only
> two cases that u.remote_ref.push becomes valid), but u.remote_ref
> shares the same storage for other members of the union, the check
> was reading from an invalid member, which was the bug.
>
> Modify the condition here to check whether the atom name
> equals to "push" or starts with "push:", to avoid reading the
> value of invalid member of the union.
>
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: ZheNing Hu <adlternative@gmail.com>
> ---
>     [GSOC] ref-filter: fix read invalid union member bug
>     
>     Change from last version:
>     Modify the processing method of the condition: check whether the name of
>     the atom equals to "push" or starts with "pushs", which can enhanced
>     security, although it may bring string match overhead.

I do not think this would have much security implication either
way.  What it buys us is the future-proofing.

I think it is OK to make this change without the enum thing to have
it graduate early as a fix to the existing code.  The enum thing can
come on top.

Will queue.  Thanks.

Re: [PATCH v3] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu]

Junio C Hamano <gitster@pobox.com> 于2021年5月10日周一 下午3:21写道：
>
> "ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > From: ZheNing Hu <adlternative@gmail.com>
> >
> > used_atom.u is an union, and it has different members depending on
> > what atom the auxiliary data the union part of the "struct
> > used_atom" wants to record. At most only one of the members can be
> > valid at any one time. Since the code checks u.remote_ref without
> > even making sure if the atom is "push" or "push:" (which are only
> > two cases that u.remote_ref.push becomes valid), but u.remote_ref
> > shares the same storage for other members of the union, the check
> > was reading from an invalid member, which was the bug.
> >
> > Modify the condition here to check whether the atom name
> > equals to "push" or starts with "push:", to avoid reading the
> > value of invalid member of the union.
> >
> > Helped-by: Junio C Hamano <gitster@pobox.com>
> > Signed-off-by: ZheNing Hu <adlternative@gmail.com>
> > ---
> >     [GSOC] ref-filter: fix read invalid union member bug
> >
> >     Change from last version:
> >     Modify the processing method of the condition: check whether the name of
> >     the atom equals to "push" or starts with "pushs", which can enhanced
> >     security, although it may bring string match overhead.
>
> I do not think this would have much security implication either
> way.  What it buys us is the future-proofing.
>

Ah, truely.

> I think it is OK to make this change without the enum thing to have
> it graduate early as a fix to the existing code.  The enum thing can
> come on top.
>

Indeed. "enum atom_type" is for ref-filter performance optimization and get
some other benefits like quick index. So I put it in another topic.

> Will queue.  Thanks.

Thanks.
--
ZheNing Hu

Re: [PATCH v3] [GSOC] ref-filter: fix read invalid union member bug

[Junio C Hamano]

"ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: ZheNing Hu <adlternative@gmail.com>
>
> used_atom.u is an union, and it has different members depending on
> what atom the auxiliary data the union part of the "struct
> used_atom" wants to record. At most only one of the members can be
> valid at any one time. Since the code checks u.remote_ref without
> even making sure if the atom is "push" or "push:" (which are only
> two cases that u.remote_ref.push becomes valid), but u.remote_ref
> shares the same storage for other members of the union, the check
> was reading from an invalid member, which was the bug.
>
> Modify the condition here to check whether the atom name
> equals to "push" or starts with "push:", to avoid reading the
> value of invalid member of the union.
>
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: ZheNing Hu <adlternative@gmail.com>
> ---

Just a final sanity check.  Is this a recent breakage or was the
code introduced at cc72385f (for-each-ref: let upstream/push
optionally report the remote name, 2017-10-05) broken from the
beginning?

I am wondering if it is easy to add a test to cover the codepath
that is affected by this change.

Thanks.

> diff --git a/ref-filter.c b/ref-filter.c
> index a0adb4551d87..213d3773ada3 100644
> --- a/ref-filter.c
> +++ b/ref-filter.c
> @@ -1730,7 +1730,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
>  			else
>  				v->s = xstrdup("");
>  			continue;
> -		} else if (atom->u.remote_ref.push) {
> +		} else if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:")) {
>  			const char *branch_name;
>  			v->s = xstrdup("");
>  			if (!skip_prefix(ref->refname, "refs/heads/",
>
> base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a

Re: [PATCH v3] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu]

> Just a final sanity check.  Is this a recent breakage or was the
> code introduced at cc72385f (for-each-ref: let upstream/push
> optionally report the remote name, 2017-10-05) broken from the
> beginning?
>

Well, The trigger condition is very special, but the bug was introduced
at that time. Let's see the "bug" example below.

> I am wondering if it is easy to add a test to cover the codepath
> that is affected by this change.
>
> Thanks.
>

Well, because this bug must require that the seventeenth bit of
`used_atom.u` is not 0, it took me a long time to find this bug.
in `used_atom.u`, only the member "color" and "contents" which
size is bigger than 17 bytes, but "%(contents:trailer:only)" only fill
the 16th byte of `used_atom.u`.

"Fortunately", I found it.

git for-each-ref --format='%(color:#aa22ac)'

I will add test for it!

Thanks!
--
ZheNing Hu

[PATCH v4] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu via GitGitGadget]

From: ZheNing Hu <adlternative@gmail.com>

used_atom.u is an union, and it has different members depending on
what atom the auxiliary data the union part of the "struct
used_atom" wants to record. At most only one of the members can be
valid at any one time. Since the code checks u.remote_ref without
even making sure if the atom is "push" or "push:" (which are only
two cases that u.remote_ref.push becomes valid), but u.remote_ref
shares the same storage for other members of the union, the check
was reading from an invalid member, which was the bug.

Modify the condition here to check whether the atom name
equals to "push" or starts with "push:", to avoid reading the
value of invalid member of the union.

Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: ZheNing Hu <adlternative@gmail.com>
---
    [GSOC] ref-filter: fix read invalid union member bug
    
    Change from last version:
    Prove that the bug may appear when using %(color) atom. And add
    corresponding test for it.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-949%2Fadlternative%2Fref-filter-enum-bug-fix-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-949/adlternative/ref-filter-enum-bug-fix-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/949

Range-diff vs v3:

 1:  21cf7a44e168 ! 1:  8c6c0368a590 [GSOC] ref-filter: fix read invalid union member bug
     @@ ref-filter.c: static int populate_value(struct ref_array_item *ref, struct strbu
       			const char *branch_name;
       			v->s = xstrdup("");
       			if (!skip_prefix(ref->refname, "refs/heads/",
     +
     + ## t/t6302-for-each-ref-filter.sh ##
     +@@ t/t6302-for-each-ref-filter.sh: test_expect_success '%(color) must fail' '
     + 	test_must_fail git for-each-ref --format="%(color)%(refname)"
     + '
     + 
     ++test_expect_success '%(color:#aa22ac) must success' '
     ++	cat >expect <<-\EOF &&
     ++	refs/heads/main
     ++	refs/heads/side
     ++	refs/odd/spot
     ++	refs/tags/annotated-tag
     ++	refs/tags/doubly-annotated-tag
     ++	refs/tags/doubly-signed-tag
     ++	refs/tags/four
     ++	refs/tags/one
     ++	refs/tags/signed-tag
     ++	refs/tags/three
     ++	refs/tags/two
     ++	EOF
     ++	git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
     ++	test_cmp expect actual
     ++'
     ++
     + test_expect_success 'left alignment is default' '
     + 	cat >expect <<-\EOF &&
     + 	refname is refs/heads/main    |refs/heads/main


 ref-filter.c                   |  2 +-
 t/t6302-for-each-ref-filter.sh | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/ref-filter.c b/ref-filter.c
index a0adb4551d87..213d3773ada3 100644
--- a/ref-filter.c
+++ b/ref-filter.c
@@ -1730,7 +1730,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
 			else
 				v->s = xstrdup("");
 			continue;
-		} else if (atom->u.remote_ref.push) {
+		} else if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:")) {
 			const char *branch_name;
 			v->s = xstrdup("");
 			if (!skip_prefix(ref->refname, "refs/heads/",
diff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh
index 9866b1b57368..38a7d83830aa 100755
--- a/t/t6302-for-each-ref-filter.sh
+++ b/t/t6302-for-each-ref-filter.sh
@@ -117,6 +117,24 @@ test_expect_success '%(color) must fail' '
 	test_must_fail git for-each-ref --format="%(color)%(refname)"
 '
 
+test_expect_success '%(color:#aa22ac) must success' '
+	cat >expect <<-\EOF &&
+	refs/heads/main
+	refs/heads/side
+	refs/odd/spot
+	refs/tags/annotated-tag
+	refs/tags/doubly-annotated-tag
+	refs/tags/doubly-signed-tag
+	refs/tags/four
+	refs/tags/one
+	refs/tags/signed-tag
+	refs/tags/three
+	refs/tags/two
+	EOF
+	git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'left alignment is default' '
 	cat >expect <<-\EOF &&
 	refname is refs/heads/main    |refs/heads/main

base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a
-- 
gitgitgadget

Re: [PATCH v4] [GSOC] ref-filter: fix read invalid union member bug

[Junio C Hamano]

"ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:

>     Change from last version:
>     Prove that the bug may appear when using %(color) atom. And add
>     corresponding test for it.

>      ++test_expect_success '%(color:#aa22ac) must success' '

s/success/succeed/

But more importantly, I am not sure how this is supposed to
demonstrate existing breakage around the %(push).  Did you mean to
use %(push) instead of %(refname) or something?


>      ++	cat >expect <<-\EOF &&
>      ++	refs/heads/main
>      ++	refs/heads/side
>      ++	refs/odd/spot
>      ++	refs/tags/annotated-tag
>      ++	refs/tags/doubly-annotated-tag
>      ++	refs/tags/doubly-signed-tag
>      ++	refs/tags/four
>      ++	refs/tags/one
>      ++	refs/tags/signed-tag
>      ++	refs/tags/three
>      ++	refs/tags/two
>      ++	EOF
>      ++	git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
>      ++	test_cmp expect actual
>      ++'
>      ++
>      + test_expect_success 'left alignment is default' '
>      + 	cat >expect <<-\EOF &&
>      + 	refname is refs/heads/main    |refs/heads/main
>
>
>  ref-filter.c                   |  2 +-
>  t/t6302-for-each-ref-filter.sh | 18 ++++++++++++++++++
>  2 files changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/ref-filter.c b/ref-filter.c
> index a0adb4551d87..213d3773ada3 100644
> --- a/ref-filter.c
> +++ b/ref-filter.c
> @@ -1730,7 +1730,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
>  			else
>  				v->s = xstrdup("");
>  			continue;
> -		} else if (atom->u.remote_ref.push) {
> +		} else if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:")) {
>  			const char *branch_name;
>  			v->s = xstrdup("");
>  			if (!skip_prefix(ref->refname, "refs/heads/",
> diff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh
> index 9866b1b57368..38a7d83830aa 100755
> --- a/t/t6302-for-each-ref-filter.sh
> +++ b/t/t6302-for-each-ref-filter.sh
> @@ -117,6 +117,24 @@ test_expect_success '%(color) must fail' '
>  	test_must_fail git for-each-ref --format="%(color)%(refname)"
>  '
>  
> +test_expect_success '%(color:#aa22ac) must success' '
> +	cat >expect <<-\EOF &&
> +	refs/heads/main
> +	refs/heads/side
> +	refs/odd/spot
> +	refs/tags/annotated-tag
> +	refs/tags/doubly-annotated-tag
> +	refs/tags/doubly-signed-tag
> +	refs/tags/four
> +	refs/tags/one
> +	refs/tags/signed-tag
> +	refs/tags/three
> +	refs/tags/two
> +	EOF
> +	git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
> +	test_cmp expect actual
> +'
> +
>  test_expect_success 'left alignment is default' '
>  	cat >expect <<-\EOF &&
>  	refname is refs/heads/main    |refs/heads/main
>
> base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a

Re: [PATCH v4] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu]

Junio C Hamano <gitster@pobox.com> 于2021年5月11日周二 上午10:29写道：
>
> "ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> >     Change from last version:
> >     Prove that the bug may appear when using %(color) atom. And add
> >     corresponding test for it.
>
> >      ++test_expect_success '%(color:#aa22ac) must success' '
>
> s/success/succeed/
>
> But more importantly, I am not sure how this is supposed to
> demonstrate existing breakage around the %(push).  Did you mean to
> use %(push) instead of %(refname) or something?
>

Ah, I don’t think this scene of damage is related to using %(push)
or %(refname).

We are just in the process of using `populate_value()`, if the atom we
specify meets the following conditions, then the condition of
atom->u.remote_ref.push!=0 will be established.

1. The atom that triggers the bug , its "if" condition order must after
"if (atom->u.remote_ref.push)", such as %(refname) or %(worktreepath),
they can be executed correctly because their order is before "push".

2. The member size in used_atom.u corresponding to the atom must
larger than 17 bytes, because the offset of "u.remote_ref.push" in
"u.remote_ref" is 17, the satisfied atoms are only "%(color)" and
 "%(contents)", their corresponding members are u.color and u.contents.

3. We happen to be able to fill in the 17th position of these structures,
 which makes atom->u.remote_ref.push not equal to 0 established.

So this kind of bug is not related to %(push), an atom that satisfies
the above conditions will make `if (atom->u.remote_ref.push)` be true.
then execute the program logic related to `%(push)`.

Now, we only have `%(color)` can trigger it "sometime",
It is unpredictable to fill in the 17th byte of used_atom.u.color,
so we cannot track all the atoms related to this bug.

git for-each-ref --format="%(color:#aa22ac)%(refname)"
git for-each-ref --format="%(color:#aa22ad)%(refname)"

will trigger the bug.

git for-each-ref --format="%(color:#aa22ae)%(refname)"
git for-each-ref --format="%(color:#aa22af)%(refname)"

will not trigger the bug.

In other words, we cannot use a perfect test set to cover all broken.
So now `%(color:#aa22ac)` is enough for explain the problem of this bug.

Thanks.
--
ZheNing Hu

Re: [PATCH v4] [GSOC] ref-filter: fix read invalid union member bug

[Junio C Hamano]

ZheNing Hu <adlternative@gmail.com> writes:

> We are just in the process of using `populate_value()`, if the atom we
> specify meets the following conditions, then the condition of
> atom->u.remote_ref.push!=0 will be established.
>
> 1. The atom that triggers the bug , its "if" condition order must after
> "if (atom->u.remote_ref.push)", such as %(refname) or %(worktreepath),
> they can be executed correctly because their order is before "push".
>
> 2. The member size in used_atom.u corresponding to the atom must
> larger than 17 bytes, because the offset of "u.remote_ref.push" in
> "u.remote_ref" is 17, the satisfied atoms are only "%(color)" and
>  "%(contents)", their corresponding members are u.color and u.contents.
>
> 3. We happen to be able to fill in the 17th position of these structures,
>  which makes atom->u.remote_ref.push not equal to 0 established.
>
> So this kind of bug is not related to %(push), an atom that satisfies
> the above conditions will make `if (atom->u.remote_ref.push)` be true.
> then execute the program logic related to `%(push)`.
>
> Now, we only have `%(color)` can trigger it "sometime",
> It is unpredictable to fill in the 17th byte of used_atom.u.color,
> so we cannot track all the atoms related to this bug.
>
> git for-each-ref --format="%(color:#aa22ac)%(refname)"
> git for-each-ref --format="%(color:#aa22ad)%(refname)"
>
> will trigger the bug.
>
> git for-each-ref --format="%(color:#aa22ae)%(refname)"
> git for-each-ref --format="%(color:#aa22af)%(refname)"
>
> will not trigger the bug.
>
> In other words, we cannot use a perfect test set to cover all broken.
> So now `%(color:#aa22ac)` is enough for explain the problem of this bug.

Well, the thing is,

    $ git checkout 49f38e2d ;# (The fifteenth batch, 2021-05-10)
    $ git am -s mbox
    $ git show --stat --oneline
    39509d100a (HEAD) ref-filter: fix read invalid union member bug
     ref-filter.c                   |  2 +-
     t/t6302-for-each-ref-filter.sh | 18 ++++++++++++++++++
     2 files changed, 19 insertions(+), 1 deletion(-)
    $ git show ref-filter.c | git apply -R ;# revert only the fix
    $ make -j32 && make -C t T=t6302-*.sh

does not seem to break anything.  Perhaps there is something more
than the "17th byte" thing (like structure padding that may vary
depending on the compiler and architecture)?

Re: [PATCH v4] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu]

> Well, the thing is,
>
>     $ git checkout 49f38e2d ;# (The fifteenth batch, 2021-05-10)
>     $ git am -s mbox
>     $ git show --stat --oneline
>     39509d100a (HEAD) ref-filter: fix read invalid union member bug
>      ref-filter.c                   |  2 +-
>      t/t6302-for-each-ref-filter.sh | 18 ++++++++++++++++++
>      2 files changed, 19 insertions(+), 1 deletion(-)
>     $ git show ref-filter.c | git apply -R ;# revert only the fix
>     $ make -j32 && make -C t T=t6302-*.sh
>
> does not seem to break anything.  Perhaps there is something more
> than the "17th byte" thing (like structure padding that may vary
> depending on the compiler and architecture)?

Fine, I guess the reason for this mystery is I "push" this branch to github
and you haven't done it. That may not be due to the platform. Because I
can see no this bug happening when I use a new git repo without "git push",
and I test in archlinux or deepin, this bug will happen in these environments.

I take back what I said before, maybe this is really related to push.

Let's see what happens:

#!/bin/sh
mkdir test
cd test
git init
echo 1>1
git add .
git branch -M main
git commit -m "test"
git remote add origin nowhere
git config branch.main.remote origin
git config branch.main.merge refs/heads/main
git for-each-ref --format="%(color:#aa22ac)%(objectname)"

These two "git config" is for simulating a push environment.

I guess you also saw this bug:

BUG: ref-filter.c:1544: unhandled RR_* enum

I will use it for the test.

Thanks.
--
ZheNing Hu

Re: [PATCH v4] [GSOC] ref-filter: fix read invalid union member bug

[Junio C Hamano]

ZheNing Hu <adlternative@gmail.com> writes:

>> Well, the thing is,
>>
>>     $ git checkout 49f38e2d ;# (The fifteenth batch, 2021-05-10)
>>     $ git am -s mbox
>>     $ git show --stat --oneline
>>     39509d100a (HEAD) ref-filter: fix read invalid union member bug
>>      ref-filter.c                   |  2 +-
>>      t/t6302-for-each-ref-filter.sh | 18 ++++++++++++++++++
>>      2 files changed, 19 insertions(+), 1 deletion(-)
>>     $ git show ref-filter.c | git apply -R ;# revert only the fix
>>     $ make -j32 && make -C t T=t6302-*.sh
>>
>> does not seem to break anything.  Perhaps there is something more
>> than the "17th byte" thing (like structure padding that may vary
>> depending on the compiler and architecture)?
>
> Fine, I guess the reason for this mystery is I "push" this branch to github
> and you haven't done it. That may not be due to the platform. Because I
> can see no this bug happening when I use a new git repo without "git push",
> and I test in archlinux or deepin, this bug will happen in these environments.

Sorry, you lost me.  I was talking about what happens in the new
test you added to t6302 not failing as designed, and there shouldn't
be "I've pushed but you haven't pushed to GitHub" distinction.  The
test is running in a brand-new repository just created for the sole
purpose of running the test after all.

> #!/bin/sh
> mkdir test
> cd test
> git init
> echo 1>1
> git add .
> git branch -M main
> git commit -m "test"
> git remote add origin nowhere
> git config branch.main.remote origin
> git config branch.main.merge refs/heads/main
> git for-each-ref --format="%(color:#aa22ac)%(objectname)"
>
> These two "git config" is for simulating a push environment.

So in short, the test script added to t6302 in the v4 patch was not
testing what it was supposed to be testing, as it didn't have the
configuration items related to %(push) atom necessary to trigger the
error?  

That I can believe.  I was starting to worry if there was something
more subtle going on, but I am glad that it was only an uncooked
patch submitted without checking.

> I guess you also saw this bug:
>
> BUG: ref-filter.c:1544: unhandled RR_* enum

No, I didn't.  I just tried to make sure the new test was truly
checking the existing breakage by partially reverting the code fix,
and saw that the new test did not fail.

Thanks.

Re: [PATCH v4] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu]

> > Fine, I guess the reason for this mystery is I "push" this branch to github
> > and you haven't done it. That may not be due to the platform. Because I
> > can see no this bug happening when I use a new git repo without "git push",
> > and I test in archlinux or deepin, this bug will happen in these environments.
>
> Sorry, you lost me.  I was talking about what happens in the new
> test you added to t6302 not failing as designed, and there shouldn't
> be "I've pushed but you haven't pushed to GitHub" distinction.  The
> test is running in a brand-new repository just created for the sole
> purpose of running the test after all.
>

Well, there are some expression problems here, I just want to express:
This bug is only triggered after I push and Git adds some config during
the push process. And then those config effort %(push) behavior.

> So in short, the test script added to t6302 in the v4 patch was not
> testing what it was supposed to be testing, as it didn't have the
> configuration items related to %(push) atom necessary to trigger the
> error?
>

Truly.

> That I can believe.  I was starting to worry if there was something
> more subtle going on, but I am glad that it was only an uncooked
> patch submitted without checking.
>

Me too, I want to complain a little bit: this bug is too difficult to locate.

> > I guess you also saw this bug:
> >
> > BUG: ref-filter.c:1544: unhandled RR_* enum
>
> No, I didn't.  I just tried to make sure the new test was truly
> checking the existing breakage by partially reverting the code fix,
> and saw that the new test did not fail.
>

I understand.

> Thanks.

Thanks.
--
ZheNing Hu

[PATCH v5] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu via GitGitGadget]

From: ZheNing Hu <adlternative@gmail.com>

used_atom.u is an union, and it has different members depending on
what atom the auxiliary data the union part of the "struct
used_atom" wants to record. At most only one of the members can be
valid at any one time. Since the code checks u.remote_ref without
even making sure if the atom is "push" or "push:" (which are only
two cases that u.remote_ref.push becomes valid), but u.remote_ref
shares the same storage for other members of the union, the check
was reading from an invalid member, which was the bug.

Modify the condition here to check whether the atom name
equals to "push" or starts with "push:", to avoid reading the
value of invalid member of the union.

Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: ZheNing Hu <adlternative@gmail.com>
---
    [GSOC] ref-filter: fix read invalid union member bug
    
    I and Junio discussed the situation that this bug might actually occur.
    
    The damage that can be found currently is using %(colors:#aa22ac) or
    some other %(colors) atoms. But Junio found that testing
    %(colors:#aa22ac) alone did not show the expected bug in the commit
    before the repair.
    
    So I conducted an experiment:
    
    When we use git push, Git will add some config, these configurations
    will affect the result of the execution process related to atom %(push)
    in populate_value().
    
    Change from last version: added a new test, which added two
    configurations:
    
    git config branch.main.remote origin git config branch.main.merge
    refs/heads/main
    
    used to simulate the configuration changes brought by git push.
    
    Finally, a test on the broken atom %(colors:#aa22ac). In the commit
    before the repair, breakage occurs. In the commit after the repair,
    breakage disappeared.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-949%2Fadlternative%2Fref-filter-enum-bug-fix-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-949/adlternative/ref-filter-enum-bug-fix-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/949

Range-diff vs v4:

 1:  8c6c0368a590 ! 1:  b546477e8c87 [GSOC] ref-filter: fix read invalid union member bug
     @@ t/t6302-for-each-ref-filter.sh: test_expect_success '%(color) must fail' '
       	test_must_fail git for-each-ref --format="%(color)%(refname)"
       '
       
     -+test_expect_success '%(color:#aa22ac) must success' '
     ++test_expect_success '%(color:#aa22ac) must successed' '
     ++	test_when_finished "cd .. && rm -rf ./test" &&
     ++	mkdir test &&
     ++	cd test &&
     ++	git init &&
      +	cat >expect <<-\EOF &&
      +	refs/heads/main
     -+	refs/heads/side
     -+	refs/odd/spot
     -+	refs/tags/annotated-tag
     -+	refs/tags/doubly-annotated-tag
     -+	refs/tags/doubly-signed-tag
     -+	refs/tags/four
     -+	refs/tags/one
     -+	refs/tags/signed-tag
     -+	refs/tags/three
     -+	refs/tags/two
      +	EOF
     ++	git add . &&
     ++	git branch -M main &&
     ++	git commit -m "test" &&
     ++	git remote add origin nowhere &&
     ++	git config branch.main.remote origin &&
     ++	git config branch.main.merge refs/heads/main &&
      +	git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
      +	test_cmp expect actual
      +'


 ref-filter.c                   |  2 +-
 t/t6302-for-each-ref-filter.sh | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/ref-filter.c b/ref-filter.c
index a0adb4551d87..213d3773ada3 100644
--- a/ref-filter.c
+++ b/ref-filter.c
@@ -1730,7 +1730,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
 			else
 				v->s = xstrdup("");
 			continue;
-		} else if (atom->u.remote_ref.push) {
+		} else if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:")) {
 			const char *branch_name;
 			v->s = xstrdup("");
 			if (!skip_prefix(ref->refname, "refs/heads/",
diff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh
index 9866b1b57368..309cf699506f 100755
--- a/t/t6302-for-each-ref-filter.sh
+++ b/t/t6302-for-each-ref-filter.sh
@@ -117,6 +117,24 @@ test_expect_success '%(color) must fail' '
 	test_must_fail git for-each-ref --format="%(color)%(refname)"
 '
 
+test_expect_success '%(color:#aa22ac) must successed' '
+	test_when_finished "cd .. && rm -rf ./test" &&
+	mkdir test &&
+	cd test &&
+	git init &&
+	cat >expect <<-\EOF &&
+	refs/heads/main
+	EOF
+	git add . &&
+	git branch -M main &&
+	git commit -m "test" &&
+	git remote add origin nowhere &&
+	git config branch.main.remote origin &&
+	git config branch.main.merge refs/heads/main &&
+	git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'left alignment is default' '
 	cat >expect <<-\EOF &&
 	refname is refs/heads/main    |refs/heads/main

base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a
-- 
gitgitgadget

Re: [PATCH v5] [GSOC] ref-filter: fix read invalid union member bug

[Junio C Hamano]

"ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: ZheNing Hu <adlternative@gmail.com>
>
> used_atom.u is an union, and it has different members depending on
> what atom the auxiliary data the union part of the "struct
> used_atom" wants to record. At most only one of the members can be
> valid at any one time. Since the code checks u.remote_ref without
> even making sure if the atom is "push" or "push:" (which are only
> two cases that u.remote_ref.push becomes valid), but u.remote_ref
> shares the same storage for other members of the union, the check
> was reading from an invalid member, which was the bug.
>
> Modify the condition here to check whether the atom name
> equals to "push" or starts with "push:", to avoid reading the
> value of invalid member of the union.
>
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: ZheNing Hu <adlternative@gmail.com>
> ---

Thanks.

> diff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh
> index 9866b1b57368..309cf699506f 100755
> --- a/t/t6302-for-each-ref-filter.sh
> +++ b/t/t6302-for-each-ref-filter.sh
> @@ -117,6 +117,24 @@ test_expect_success '%(color) must fail' '
>  	test_must_fail git for-each-ref --format="%(color)%(refname)"
>  '
>  
> +test_expect_success '%(color:#aa22ac) must successed' '

"succeed".

> +	test_when_finished "cd .. && rm -rf ./test" &&

Not a very good practice to chdir around, even if you have
when-finished clean-up.  Not worth risking "rm -rf" at random
places when for example somebody breaks when-finished.

Instead...

> +	mkdir test &&

Place everything below ...

> +	cd test &&
> +	git init &&
> +	cat >expect <<-\EOF &&
> +	refs/heads/main
> +	EOF
> +	git add . &&
> +	git branch -M main &&

This is in a freshly created repository without commit.  Does
"branch -M" work for such an unborn branch?  Perhaps start this
block like so:

	git init test &&
	(
		cd test &&
		test_commit initial &&
		git branch -M main &&
		cat >expect <<-\EOF &&
		refs/heads/main
                refs/tags/initial
		EOF
		git remote add origin nowhere &&
		...

> +	git commit -m "test" &&
> +	git remote add origin nowhere &&
> +	git config branch.main.remote origin &&
> +	git config branch.main.merge refs/heads/main &&
> +	git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
> +	test_cmp expect actual

... up to here inside a (subshell).

By the way, your use of "git branch -M" makes the test work whether
the default initial branch name is still 'master' or already 'main'
by forcing the branch used for testing to be 'main'.  Clever ;-).

Will queue with all of the above suggestions squashed in; please see
if the result is good when it is pushed out later today.

Thanks.

> +'
> +
>  test_expect_success 'left alignment is default' '
>  	cat >expect <<-\EOF &&
>  	refname is refs/heads/main    |refs/heads/main
>
> base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a

Re: [PATCH v5] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu]

> > +     test_when_finished "cd .. && rm -rf ./test" &&
>
> Not a very good practice to chdir around, even if you have
> when-finished clean-up.  Not worth risking "rm -rf" at random
> places when for example somebody breaks when-finished.
>

OK.

> Instead...
>
> > +     mkdir test &&
>
> Place everything below ...
>
> > +     cd test &&
> > +     git init &&
> > +     cat >expect <<-\EOF &&
> > +     refs/heads/main
> > +     EOF
> > +     git add . &&
> > +     git branch -M main &&
>
> This is in a freshly created repository without commit.  Does
> "branch -M" work for such an unborn branch?  Perhaps start this
> block like so:
>

I think "git branch -M" before "git commit" also will be fine.
Of course it's okay to put it after commit and before "test_cmp".

>         git init test &&
>         (
>                 cd test &&
>                 test_commit initial &&
>                 git branch -M main &&
>                 cat >expect <<-\EOF &&
>                 refs/heads/main
>                 refs/tags/initial
>                 EOF
>                 git remote add origin nowhere &&
>                 ...
>
> > +     git commit -m "test" &&
> > +     git remote add origin nowhere &&
> > +     git config branch.main.remote origin &&
> > +     git config branch.main.merge refs/heads/main &&
> > +     git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
> > +     test_cmp expect actual
>
> ... up to here inside a (subshell).
>

I get it now. By executing cd in the subshell, the "cd .." step can be
omitted.

> By the way, your use of "git branch -M" makes the test work whether
> the default initial branch name is still 'master' or already 'main'
> by forcing the branch used for testing to be 'main'.  Clever ;-).
>

Hh, real knowledge comes from practice. Yesterday, I tried it on the old
version of git on some classmates’ computers, and there was an error,
so using "git branch -M main" can avoid this error.

> Will queue with all of the above suggestions squashed in; please see
> if the result is good when it is pushed out later today.
>
> Thanks.
>

Thanks!
--
ZheNing Hu

[PATCH v6] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu via GitGitGadget]

From: ZheNing Hu <adlternative@gmail.com>

used_atom.u is an union, and it has different members depending on
what atom the auxiliary data the union part of the "struct
used_atom" wants to record. At most only one of the members can be
valid at any one time. Since the code checks u.remote_ref without
even making sure if the atom is "push" or "push:" (which are only
two cases that u.remote_ref.push becomes valid), but u.remote_ref
shares the same storage for other members of the union, the check
was reading from an invalid member, which was the bug.

Modify the condition here to check whether the atom name
equals to "push" or starts with "push:", to avoid reading the
value of invalid member of the union.

Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: ZheNing Hu <adlternative@gmail.com>
---
    [GSOC] ref-filter: fix read invalid union member bug
    
    Change from last version: Modify the test content to make the test more
    correct and complete.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-949%2Fadlternative%2Fref-filter-enum-bug-fix-v6
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-949/adlternative/ref-filter-enum-bug-fix-v6
Pull-Request: https://github.com/gitgitgadget/git/pull/949

Range-diff vs v5:

 1:  b546477e8c87 ! 1:  518cc41a78a4 [GSOC] ref-filter: fix read invalid union member bug
     @@ t/t6302-for-each-ref-filter.sh: test_expect_success '%(color) must fail' '
       	test_must_fail git for-each-ref --format="%(color)%(refname)"
       '
       
     -+test_expect_success '%(color:#aa22ac) must successed' '
     -+	test_when_finished "cd .. && rm -rf ./test" &&
     ++test_expect_success '%(color:#aa22ac) must succeed' '
      +	mkdir test &&
     -+	cd test &&
     -+	git init &&
     -+	cat >expect <<-\EOF &&
     -+	refs/heads/main
     -+	EOF
     -+	git add . &&
     -+	git branch -M main &&
     -+	git commit -m "test" &&
     -+	git remote add origin nowhere &&
     -+	git config branch.main.remote origin &&
     -+	git config branch.main.merge refs/heads/main &&
     -+	git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
     -+	test_cmp expect actual
     ++	git init test &&
     ++	(
     ++		cd test &&
     ++		test_commit initial &&
     ++		git branch -M main &&
     ++		cat >expect <<-\EOF &&
     ++		refs/heads/main
     ++		refs/tags/initial
     ++		EOF
     ++		git remote add origin nowhere &&
     ++		git config branch.main.remote origin &&
     ++		git config branch.main.merge refs/heads/main &&
     ++		git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
     ++		test_cmp expect actual
     ++	)
      +'
      +
       test_expect_success 'left alignment is default' '


 ref-filter.c                   |  2 +-
 t/t6302-for-each-ref-filter.sh | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/ref-filter.c b/ref-filter.c
index a0adb4551d87..213d3773ada3 100644
--- a/ref-filter.c
+++ b/ref-filter.c
@@ -1730,7 +1730,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
 			else
 				v->s = xstrdup("");
 			continue;
-		} else if (atom->u.remote_ref.push) {
+		} else if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:")) {
 			const char *branch_name;
 			v->s = xstrdup("");
 			if (!skip_prefix(ref->refname, "refs/heads/",
diff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh
index 9866b1b57368..b64abe4184b8 100755
--- a/t/t6302-for-each-ref-filter.sh
+++ b/t/t6302-for-each-ref-filter.sh
@@ -117,6 +117,25 @@ test_expect_success '%(color) must fail' '
 	test_must_fail git for-each-ref --format="%(color)%(refname)"
 '
 
+test_expect_success '%(color:#aa22ac) must succeed' '
+	mkdir test &&
+	git init test &&
+	(
+		cd test &&
+		test_commit initial &&
+		git branch -M main &&
+		cat >expect <<-\EOF &&
+		refs/heads/main
+		refs/tags/initial
+		EOF
+		git remote add origin nowhere &&
+		git config branch.main.remote origin &&
+		git config branch.main.merge refs/heads/main &&
+		git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
+		test_cmp expect actual
+	)
+'
+
 test_expect_success 'left alignment is default' '
 	cat >expect <<-\EOF &&
 	refname is refs/heads/main    |refs/heads/main

base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a
-- 
gitgitgadget

Re: [PATCH v6] [GSOC] ref-filter: fix read invalid union member bug

[Junio C Hamano]

"ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:

> +test_expect_success '%(color:#aa22ac) must succeed' '
> +	mkdir test &&

You do not need "mkdir test" here, as "git init test" would take
care of it.

On the other hand, you'd want to get 'test' directory removed after
this passes, so 

	test_when_finished rm -fr test &&

would be useful.

Other than that, looks good to me.

Thanks.

> +	git init test &&
> +	(
> +		cd test &&
> +		test_commit initial &&
> +		git branch -M main &&
> +		cat >expect <<-\EOF &&
> +		refs/heads/main
> +		refs/tags/initial
> +		EOF
> +		git remote add origin nowhere &&
> +		git config branch.main.remote origin &&
> +		git config branch.main.merge refs/heads/main &&
> +		git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
> +		test_cmp expect actual
> +	)
> +'
> +
>  test_expect_success 'left alignment is default' '
>  	cat >expect <<-\EOF &&
>  	refname is refs/heads/main    |refs/heads/main
>
> base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a

Re: [PATCH v6] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu]

Junio C Hamano <gitster@pobox.com> 于2021年5月13日周四 上午7:24写道：
>
> "ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > +test_expect_success '%(color:#aa22ac) must succeed' '
> > +     mkdir test &&
>
> You do not need "mkdir test" here, as "git init test" would take
> care of it.
>

Make sence.

> On the other hand, you'd want to get 'test' directory removed after
> this passes, so
>
>         test_when_finished rm -fr test &&
>
> would be useful.
>

Well, deleting or not deleting the directory "test" is not a big problem here.
Delete it is ok.

> Other than that, looks good to me.
>
> Thanks.
>

Thanks.
--
ZheNing Hu

[PATCH v7] [GSOC] ref-filter: fix read invalid union member bug

[ZheNing Hu via GitGitGadget]

From: ZheNing Hu <adlternative@gmail.com>

used_atom.u is an union, and it has different members depending on
what atom the auxiliary data the union part of the "struct
used_atom" wants to record. At most only one of the members can be
valid at any one time. Since the code checks u.remote_ref without
even making sure if the atom is "push" or "push:" (which are only
two cases that u.remote_ref.push becomes valid), but u.remote_ref
shares the same storage for other members of the union, the check
was reading from an invalid member, which was the bug.

Modify the condition here to check whether the atom name
equals to "push" or starts with "push:", to avoid reading the
value of invalid member of the union.

Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: ZheNing Hu <adlternative@gmail.com>
---
    [GSOC] ref-filter: fix read invalid union member bug
    
    Change from last version: Modify the test content, reduce additional
    'mkdir' operation and remove temp-dir "test" when finished.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-949%2Fadlternative%2Fref-filter-enum-bug-fix-v7
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-949/adlternative/ref-filter-enum-bug-fix-v7
Pull-Request: https://github.com/gitgitgadget/git/pull/949

Range-diff vs v6:

 1:  518cc41a78a4 ! 1:  50bef8439a85 [GSOC] ref-filter: fix read invalid union member bug
     @@ t/t6302-for-each-ref-filter.sh: test_expect_success '%(color) must fail' '
       '
       
      +test_expect_success '%(color:#aa22ac) must succeed' '
     -+	mkdir test &&
     ++	test_when_finished rm -fr test &&
      +	git init test &&
      +	(
      +		cd test &&


 ref-filter.c                   |  2 +-
 t/t6302-for-each-ref-filter.sh | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/ref-filter.c b/ref-filter.c
index a0adb4551d87..213d3773ada3 100644
--- a/ref-filter.c
+++ b/ref-filter.c
@@ -1730,7 +1730,7 @@ static int populate_value(struct ref_array_item *ref, struct strbuf *err)
 			else
 				v->s = xstrdup("");
 			continue;
-		} else if (atom->u.remote_ref.push) {
+		} else if (!strcmp(atom->name, "push") || starts_with(atom->name, "push:")) {
 			const char *branch_name;
 			v->s = xstrdup("");
 			if (!skip_prefix(ref->refname, "refs/heads/",
diff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh
index 9866b1b57368..5070d0cf1e80 100755
--- a/t/t6302-for-each-ref-filter.sh
+++ b/t/t6302-for-each-ref-filter.sh
@@ -117,6 +117,25 @@ test_expect_success '%(color) must fail' '
 	test_must_fail git for-each-ref --format="%(color)%(refname)"
 '
 
+test_expect_success '%(color:#aa22ac) must succeed' '
+	test_when_finished rm -fr test &&
+	git init test &&
+	(
+		cd test &&
+		test_commit initial &&
+		git branch -M main &&
+		cat >expect <<-\EOF &&
+		refs/heads/main
+		refs/tags/initial
+		EOF
+		git remote add origin nowhere &&
+		git config branch.main.remote origin &&
+		git config branch.main.merge refs/heads/main &&
+		git for-each-ref --format="%(color:#aa22ac)%(refname)" >actual &&
+		test_cmp expect actual
+	)
+'
+
 test_expect_success 'left alignment is default' '
 	cat >expect <<-\EOF &&
 	refname is refs/heads/main    |refs/heads/main

base-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a
-- 
gitgitgadget