[PATCH v2 1/2] Fix duplicated test name

[PATCH v2 1/2] Fix duplicated test name

[Charles Bailey]

Signed-off-by: Charles Bailey <charles@hashpling.org>
---

Spotted while testing t7810-grep and grep "i-t-a" fixes.

 t/t7810-grep.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t7810-grep.sh b/t/t7810-grep.sh
index 1e72971..c4302ed 100755
--- a/t/t7810-grep.sh
+++ b/t/t7810-grep.sh
@@ -353,7 +353,7 @@ test_expect_success 'grep -l -C' '
 cat >expected <<EOF
 file:5
 EOF
-test_expect_success 'grep -l -C' '
+test_expect_success 'grep -c -C' '
 	git grep -c -C1 foo >actual &&
 	test_cmp expected actual
 '
-- 
2.8.2.311.gee88674

[PATCH v2 2/2] grep: fix grepping for "intent to add" files

[Charles Bailey]

From: Charles Bailey <cbailey32@bloomberg.net>

This reverts commit 4d552005323034c1d6311796ac1074e9a4b4b57e and adds an
alternative fix to maintain the -L --cached behavior.

4d5520053 caused 'git grep' to no longer find matches in new files in
the working tree where the corresponding index entry had the "intent to
add" bit set, despite the fact that these files are tracked.

The content in the index of a file for which the "intent to add" bit is
set is considered indeterminate and not empty. For most grep queries we
want these to behave the same, however for -L --cached (files without a
match) we don't want to respond positively for "intent to add" files as
their contents are indeterminate. This is in contrast to files with
empty contents in the index (no lines implies no matches for any grep
query expression) which should be reported in the output of a grep -L
--cached invocation.

Add tests to cover this case and a few related cases which previously
lacked coverage.

Helped-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Charles Bailey <cbailey32@bloomberg.net>
---

Is "Helped-by" an appropriate attribution in this case?

Personally, I could have gone either way on the -L --cached
functionality but I know that Duy has put a lot more thought into
"intent-to-add" entries so I trust his judgement.

 builtin/grep.c  |  4 ++--
 t/t7810-grep.sh | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/builtin/grep.c b/builtin/grep.c
index 462e607..ae73831 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -386,7 +386,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int
 
 	for (nr = 0; nr < active_nr; nr++) {
 		const struct cache_entry *ce = active_cache[nr];
-		if (!S_ISREG(ce->ce_mode) || ce_intent_to_add(ce))
+		if (!S_ISREG(ce->ce_mode))
 			continue;
 		if (!ce_path_match(ce, pathspec, NULL))
 			continue;
@@ -396,7 +396,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int
 		 * cache version instead
 		 */
 		if (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {
-			if (ce_stage(ce))
+			if (ce_stage(ce) || ce_intent_to_add(ce))
 				continue;
 			hit |= grep_sha1(opt, ce->sha1, ce->name, 0, ce->name);
 		}
diff --git a/t/t7810-grep.sh b/t/t7810-grep.sh
index c4302ed..6c7ccb3 100755
--- a/t/t7810-grep.sh
+++ b/t/t7810-grep.sh
@@ -1364,4 +1364,42 @@ test_expect_success 'grep --color -e A --and -e B -p with context' '
 	test_cmp expected actual
 '
 
+test_expect_success 'grep can find things only in the work tree' '
+	touch work-tree-only &&
+	git add work-tree-only &&
+	echo "find in work tree" >work-tree-only &&
+	git grep --quiet "find in work tree" &&
+	test_must_fail git grep --quiet --cached "find in work tree" &&
+	test_must_fail git grep --quiet "find in work tree" HEAD &&
+	git rm -f work-tree-only
+'
+
+test_expect_success 'grep can find things only in the work tree (i-t-a)' '
+	echo "intend to add this" >intend-to-add &&
+	git add -N intend-to-add &&
+	git grep --quiet "intend to add this" &&
+	test_must_fail git grep --quiet --cached "intend to add this" &&
+	test_must_fail git grep --quiet "intend to add this" HEAD &&
+	git rm -f intend-to-add
+'
+
+test_expect_success 'grep can find things only in the index' '
+	echo "only in the index" >cache-this &&
+	git add cache-this &&
+	rm cache-this &&
+	test_must_fail git grep --quiet "only in the index" &&
+	git grep --quiet --cached "only in the index" &&
+	test_must_fail git grep --quiet "only in the index" HEAD &&
+	git rm --cached cache-this
+'
+
+test_expect_success 'grep does not report i-t-a with -L --cached' '
+	echo "intend to add this" >intend-to-add &&
+	git add -N intend-to-add &&
+	git ls-files | grep -v "^intend-to-add\$" >expected &&
+	git grep -L --cached "nonexistent_string" >actual &&
+	test_cmp expected actual &&
+	git rm -f intend-to-add
+'
+
 test_done
-- 
2.8.2.311.gee88674

Re: [PATCH v2 2/2] grep: fix grepping for "intent to add" files

[Junio C Hamano]

Charles Bailey <charles@hashpling.org> writes:

> Is "Helped-by" an appropriate attribution in this case?

Sure.

> diff --git a/builtin/grep.c b/builtin/grep.c
> index 462e607..ae73831 100644
> --- a/builtin/grep.c
> +++ b/builtin/grep.c
> @@ -386,7 +386,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int
>  
>  	for (nr = 0; nr < active_nr; nr++) {
>  		const struct cache_entry *ce = active_cache[nr];
> -		if (!S_ISREG(ce->ce_mode) || ce_intent_to_add(ce))
> +		if (!S_ISREG(ce->ce_mode))
>  			continue;
>  		if (!ce_path_match(ce, pathspec, NULL))
>  			continue;
> @@ -396,7 +396,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int
>  		 * cache version instead
>  		 */
>  		if (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {
> -			if (ce_stage(ce))
> +			if (ce_stage(ce) || ce_intent_to_add(ce))
>  				continue;
>  			hit |= grep_sha1(opt, ce->sha1, ce->name, 0, ce->name);
>  		}

OK, so this function handles searching in either the index or the
working tree.

The first hunk used to unconditionally discard paths marked as
i-t-a, even when we are looking at the working tree, which is
clearly useless, and we stop rejecting i-t-a paths too early, which
is good.

The second hunk is for "grep --cached" but also covers two other
cases.  What are these?

CE_VALID is used by "Assume unchanged".  Because the user promised
that s/he will take responsibility of keeping the working tree
contents in sync with what is in the index by not modifying it, even
when we are not doing "grep --cached", we pick up the contents from
the index and look for the string in there, instead of going to the
working tree.  In other words, even though at the mechanical level
we are looking into the index, logically we are searching in the
working tree.  Is it sensible to skip i-t-a entries in that case?

I think the same discussion would apply to CE_SKIP_WORKTREE (see
"Skip-worktree bit" in Documentation/git-update-index.txt).

So I wonder if a better change would be more like

	for (...) {
        	if (!S_ISREG(ce->ce_mode))
                	continue; /* not a regular file */
		if (!ce_path_match(ce, pathspec, NULL)
                	continue; /* uninteresting */
+		if (cached && ce_intent_to_add(ce))
+			continue; /* path not yet in the index */
		        
		if (cached || ...)
                	UNCHANGED FROM THE ORIGINAL

perhaps?

I actually think (ce->ce_flags & CE_VALID) case should go to working
tree for performance, but that is a separate topic.

> +test_expect_success 'grep can find things only in the work tree' '
> +	touch work-tree-only &&

Please do not use "touch" if the reason to use the command is *not*
to muck with the timestamp, e.g. to create an empty file.

> +	git add work-tree-only &&
> +	echo "find in work tree" >work-tree-only &&
> +	git grep --quiet "find in work tree" &&
> +	test_must_fail git grep --quiet --cached "find in work tree" &&
> +	test_must_fail git grep --quiet "find in work tree" HEAD &&
> +	git rm -f work-tree-only
> +'
> +
> +test_expect_success 'grep can find things only in the work tree (i-t-a)' '
> +	echo "intend to add this" >intend-to-add &&
> +	git add -N intend-to-add &&
> +	git grep --quiet "intend to add this" &&
> +	test_must_fail git grep --quiet --cached "intend to add this" &&
> +	test_must_fail git grep --quiet "intend to add this" HEAD &&
> +	git rm -f intend-to-add
> +'
> +
> +test_expect_success 'grep can find things only in the index' '
> +	echo "only in the index" >cache-this &&
> +	git add cache-this &&
> +	rm cache-this &&
> +	test_must_fail git grep --quiet "only in the index" &&
> +	git grep --quiet --cached "only in the index" &&
> +	test_must_fail git grep --quiet "only in the index" HEAD &&
> +	git rm --cached cache-this
> +'
> +
> +test_expect_success 'grep does not report i-t-a with -L --cached' '
> +	echo "intend to add this" >intend-to-add &&
> +	git add -N intend-to-add &&
> +	git ls-files | grep -v "^intend-to-add\$" >expected &&
> +	git grep -L --cached "nonexistent_string" >actual &&
> +	test_cmp expected actual &&
> +	git rm -f intend-to-add
> +'
> +
>  test_done

Re: [PATCH v2 2/2] grep: fix grepping for "intent to add" files

[Eric Sunshine]

On Tue, Jun 21, 2016 at 5:14 PM, Charles Bailey <charles@hashpling.org> wrote:
> From: Charles Bailey <cbailey32@bloomberg.net>
>
> This reverts commit 4d552005323034c1d6311796ac1074e9a4b4b57e and adds an
> alternative fix to maintain the -L --cached behavior.

It is common to provide some context along with the (shortened) commit
ID. For instance:

    This reverts 4d55200 (grep: make it clear i-t-a entries are
    ignored, 2015-12-27) and adds ...

> 4d5520053 caused 'git grep' to no longer find matches in new files in
> the working tree where the corresponding index entry had the "intent to
> add" bit set, despite the fact that these files are tracked.
> [...]
> Helped-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> Signed-off-by: Charles Bailey <cbailey32@bloomberg.net>
> ---
>
> Is "Helped-by" an appropriate attribution in this case?

Very much so.

> diff --git a/t/t7810-grep.sh b/t/t7810-grep.sh
> @@ -1364,4 +1364,42 @@ test_expect_success 'grep --color -e A --and -e B -p with context' '
> +test_expect_success 'grep can find things only in the work tree' '
> +       touch work-tree-only &&

Avoid 'touch' if the timestamp of the file has no significance. Use '>' instead:

    >work-tree-only &&

> +       git add work-tree-only &&
> +       echo "find in work tree" >work-tree-only &&
> +       git grep --quiet "find in work tree" &&
> +       test_must_fail git grep --quiet --cached "find in work tree" &&
> +       test_must_fail git grep --quiet "find in work tree" HEAD &&
> +       git rm -f work-tree-only

If any statement before this cleanup code fails, then the cleanup will
never take place (due to the &&-chain). To ensure cleanup regardless
of test outcome, instead use test_when_finished() at the beginning of
the test:

    test_when_finished "git rm -f work-tree-only" &&

Same applies to other added tests.

> +'
> +
> +test_expect_success 'grep can find things only in the work tree (i-t-a)' '
> +       echo "intend to add this" >intend-to-add &&
> +       git add -N intend-to-add &&
> +       git grep --quiet "intend to add this" &&
> +       test_must_fail git grep --quiet --cached "intend to add this" &&
> +       test_must_fail git grep --quiet "intend to add this" HEAD &&
> +       git rm -f intend-to-add
> +'
> +
> +test_expect_success 'grep can find things only in the index' '
> +       echo "only in the index" >cache-this &&
> +       git add cache-this &&
> +       rm cache-this &&
> +       test_must_fail git grep --quiet "only in the index" &&
> +       git grep --quiet --cached "only in the index" &&
> +       test_must_fail git grep --quiet "only in the index" HEAD &&
> +       git rm --cached cache-this
> +'
> +
> +test_expect_success 'grep does not report i-t-a with -L --cached' '
> +       echo "intend to add this" >intend-to-add &&
> +       git add -N intend-to-add &&
> +       git ls-files | grep -v "^intend-to-add\$" >expected &&
> +       git grep -L --cached "nonexistent_string" >actual &&
> +       test_cmp expected actual &&
> +       git rm -f intend-to-add
> +'
> +
>  test_done

Re: [PATCH v2 2/2] grep: fix grepping for "intent to add" files

[Duy Nguyen]

On Wed, Jun 22, 2016 at 3:13 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
> On Tue, Jun 21, 2016 at 5:14 PM, Charles Bailey <charles@hashpling.org> wrote:
>> From: Charles Bailey <cbailey32@bloomberg.net>
>>
>> This reverts commit 4d552005323034c1d6311796ac1074e9a4b4b57e and adds an
>> alternative fix to maintain the -L --cached behavior.
>
> It is common to provide some context along with the (shortened) commit
> ID. For instance:
>
>     This reverts 4d55200 (grep: make it clear i-t-a entries are
>     ignored, 2015-12-27) and adds ...

And that could be produced with some git alias like

git config alias.one 'show -s --date=short --pretty='format:%h (%s - %ad)'

No point in manually copy/pasting the context.
-- 
Duy

Re: [PATCH v2 2/2] grep: fix grepping for "intent to add" files

[Duy Nguyen]

On Wed, Jun 22, 2016 at 12:49 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> @@ -396,7 +396,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int
>>                * cache version instead
>>                */
>>               if (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {
>> -                     if (ce_stage(ce))
>> +                     if (ce_stage(ce) || ce_intent_to_add(ce))
>>                               continue;
>>                       hit |= grep_sha1(opt, ce->sha1, ce->name, 0, ce->name);
>>               }
>
> OK, so this function handles searching in either the index or the
> working tree.
>
> The first hunk used to unconditionally discard paths marked as
> i-t-a, even when we are looking at the working tree, which is
> clearly useless, and we stop rejecting i-t-a paths too early, which
> is good.
>
> The second hunk is for "grep --cached" but also covers two other
> cases.  What are these?
>
> CE_VALID is used by "Assume unchanged".  Because the user promised
> that s/he will take responsibility of keeping the working tree
> contents in sync with what is in the index by not modifying it, even
> when we are not doing "grep --cached", we pick up the contents from
> the index and look for the string in there, instead of going to the
> working tree.  In other words, even though at the mechanical level
> we are looking into the index, logically we are searching in the
> working tree.  Is it sensible to skip i-t-a entries in that case?
>
> I think the same discussion would apply to CE_SKIP_WORKTREE (see
> "Skip-worktree bit" in Documentation/git-update-index.txt).
>
> So I wonder if a better change would be more like
>
>         for (...) {
>                 if (!S_ISREG(ce->ce_mode))
>                         continue; /* not a regular file */
>                 if (!ce_path_match(ce, pathspec, NULL)
>                         continue; /* uninteresting */
> +               if (cached && ce_intent_to_add(ce))
> +                       continue; /* path not yet in the index */
>
>                 if (cached || ...)
>                         UNCHANGED FROM THE ORIGINAL
>
> perhaps?

I did wonder a bit about these cases. But, can i-t-a really be
combined with CE_VALID or CE_SKIP_WORKTREE? CE_SKIP_... is
automatically set and should not cover i-t-a entries imo (I didn't
check the implementation). CE_VALID is about real entries, yes you
could do "git update-index --assume-unchanged <ita-path>" but it does
not feel right to me.

If cached is false and ce_ita() is true and either CE_VALID or
CE_SKIP_WORKTREE is set, we would continue to grep an _empty_ SHA-1.
But I think we should grep_file() instead, at least for CE_VALID.
-- 
Duy

Re: [PATCH v2 2/2] grep: fix grepping for "intent to add" files

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

>> So I wonder if a better change would be more like
>>
>>         for (...) {
>>                 if (!S_ISREG(ce->ce_mode))
>>                         continue; /* not a regular file */
>>                 if (!ce_path_match(ce, pathspec, NULL)
>>                         continue; /* uninteresting */
>> +               if (cached && ce_intent_to_add(ce))
>> +                       continue; /* path not yet in the index */
>>
>>                 if (cached || ...)
>>                         UNCHANGED FROM THE ORIGINAL
>>
>> perhaps?
>
> I did wonder a bit about these cases. But, can i-t-a really be
> combined with CE_VALID or CE_SKIP_WORKTREE? CE_SKIP_... is
> automatically set and should not cover i-t-a entries imo (I didn't
> check the implementation). CE_VALID is about real entries, yes you
> could do "git update-index --assume-unchanged <ita-path>" but it does
> not feel right to me.

Yeah but we know people are stupid^W^Wdo unexpected things ;-)

> If cached is false and ce_ita() is true and either CE_VALID or
> CE_SKIP_WORKTREE is set, we would continue to grep an _empty_ SHA-1.
> But I think we should grep_file() instead, at least for CE_VALID.

Yes, that is the breakage I noticed in the patch under discussion
and that I wanted to fix in the "I wonder if a better change would
be..." version.

Re: [PATCH v2 2/2] grep: fix grepping for "intent to add" files

[Duy Nguyen]

On Wed, Jun 22, 2016 at 8:00 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Duy Nguyen <pclouds@gmail.com> writes:
>
>>> So I wonder if a better change would be more like
>>>
>>>         for (...) {
>>>                 if (!S_ISREG(ce->ce_mode))
>>>                         continue; /* not a regular file */
>>>                 if (!ce_path_match(ce, pathspec, NULL)
>>>                         continue; /* uninteresting */
>>> +               if (cached && ce_intent_to_add(ce))
>>> +                       continue; /* path not yet in the index */
>>>
>>>                 if (cached || ...)
>>>                         UNCHANGED FROM THE ORIGINAL
>>>
>>> perhaps?
>>
>> I did wonder a bit about these cases. But, can i-t-a really be
>> combined with CE_VALID or CE_SKIP_WORKTREE? CE_SKIP_... is
>> automatically set and should not cover i-t-a entries imo (I didn't
>> check the implementation). CE_VALID is about real entries, yes you
>> could do "git update-index --assume-unchanged <ita-path>" but it does
>> not feel right to me.
>
> Yeah but we know people are stupid^W^Wdo unexpected things ;-)
>
>> If cached is false and ce_ita() is true and either CE_VALID or
>> CE_SKIP_WORKTREE is set, we would continue to grep an _empty_ SHA-1.
>> But I think we should grep_file() instead, at least for CE_VALID.
>
> Yes, that is the breakage I noticed in the patch under discussion
> and that I wanted to fix in the "I wonder if a better change would
> be..." version.

Heh.. I did guess that. Since neither solution is complete, I'm in
favor of Charles's and assume that i-t-a forces to ignore CE_SKIP and
CE_SKIP_WORKTREE. I could wait for people to come back complaining,
then we know there are real users in very obscure cases and will fix
it then.
-- 
Duy

Re: [PATCH v2 2/2] grep: fix grepping for "intent to add" files

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

>>> If cached is false and ce_ita() is true and either CE_VALID or
>>> CE_SKIP_WORKTREE is set, we would continue to grep an _empty_ SHA-1.
>>> But I think we should grep_file() instead, at least for CE_VALID.
>>
>> Yes, that is the breakage I noticed in the patch under discussion
>> and that I wanted to fix in the "I wonder if a better change would
>> be..." version.
>
> Heh.. I did guess that. Since neither solution is complete, I'm in
> favor of Charles's and assume that i-t-a forces to ignore CE_SKIP and
> CE_SKIP_WORKTREE. I could wait for people to come back complaining,
> then we know there are real users in very obscure cases and will fix
> it then.

I said something that can be misunderstood.  I meant "I wonder if ..."
version is correct.  Charles's has the bugs you mentioned and I
wanted to fix them by sending the "I wonder if..." version out.

But you seem to have misread my statement as "A bug is in my version
and I want to fix that bug in my version".  That is not what I
meant.