[PATCH] grep: fix grepping for "intent to add" files

[PATCH] grep: fix grepping for "intent to add" files

[Charles Bailey]

From: Charles Bailey <cbailey32@bloomberg.net>

This reverts commit 4d552005323034c1d6311796ac1074e9a4b4b57e.

This commit caused 'git grep' to no longer find matches in new files in
the working tree where the corresponding index entry had the "intent to
add" bit set.

Add tests to cover this case and a few related cases which previously
lacked coverage.

Signed-off-by: Charles Bailey <cbailey32@bloomberg.net>
---

Originally discussed:

http://thread.gmane.org/gmane.comp.version-control.git/272363/focus=276358

http://thread.gmane.org/gmane.comp.version-control.git/283001/focus=283002

Unless I've misunderstood the conversation and commit message, the
referenced commit was supposed to be a "code as a comment" commit with
no change in observable behavior however a user was surprised that 'git
grep' couldn't find something that regular grep could, despite the file
being tracked - albeit new and "intended to add".

 builtin/grep.c  |  2 +-
 t/t7810-grep.sh | 29 +++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/builtin/grep.c b/builtin/grep.c
index 462e607..d5aacba 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -386,7 +386,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int
 
 	for (nr = 0; nr < active_nr; nr++) {
 		const struct cache_entry *ce = active_cache[nr];
-		if (!S_ISREG(ce->ce_mode) || ce_intent_to_add(ce))
+		if (!S_ISREG(ce->ce_mode))
 			continue;
 		if (!ce_path_match(ce, pathspec, NULL))
 			continue;
diff --git a/t/t7810-grep.sh b/t/t7810-grep.sh
index 1e72971..eae731a 100755
--- a/t/t7810-grep.sh
+++ b/t/t7810-grep.sh
@@ -1364,4 +1364,33 @@ test_expect_success 'grep --color -e A --and -e B -p with context' '
 	test_cmp expected actual
 '
 
+test_expect_success 'grep can find things only in the work tree' '
+	touch work-tree-only &&
+	git add work-tree-only &&
+	echo "find in work tree" >work-tree-only &&
+	git grep --quiet "find in work tree" &&
+	test_must_fail git grep --quiet --cached "find in work tree" &&
+	test_must_fail git grep --quiet "find in work tree" HEAD &&
+	git rm -f work-tree-only
+'
+
+test_expect_success 'grep can find things only in the work tree (i-t-a)' '
+	echo "intend to add this" >intend-to-add &&
+	git add -N intend-to-add &&
+	git grep --quiet "intend to add this" &&
+	test_must_fail git grep --quiet --cached "intend to add this" &&
+	test_must_fail git grep --quiet "intend to add this" HEAD &&
+	git rm -f intend-to-add
+'
+
+test_expect_success 'grep can find things only in the index' '
+	echo "only in the index" >cache-this &&
+	git add cache-this &&
+	rm cache-this &&
+	test_must_fail git grep --quiet "only in the index" &&
+	git grep --quiet --cached "only in the index" &&
+	test_must_fail git grep --quiet "only in the index" HEAD &&
+	git rm --cached cache-this
+'
+
 test_done
-- 
2.8.2.311.gee88674

Re: [PATCH] grep: fix grepping for "intent to add" files

[Duy Nguyen]

On Thu, Jun 16, 2016 at 07:53:24AM +0100, Charles Bailey wrote:
> From: Charles Bailey <cbailey32@bloomberg.net>
> 
> This reverts commit 4d552005323034c1d6311796ac1074e9a4b4b57e.
> 
> This commit caused 'git grep' to no longer find matches in new files in
> the working tree where the corresponding index entry had the "intent to
> add" bit set.

I don't think revert is right. It rather needs a re-fix like below.
Basically we want grep_file() to run as normal, but grep_sha1()
(i.e. git grep --cached) should ignore i-t-a entries, because empty
SHA-1 is not the right content to grep. It does not matter in positive
matching, sure, but it may in -v cache.

-- 8< --
diff --git a/builtin/grep.c b/builtin/grep.c
index 462e607..ae73831 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -386,7 +386,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int
 
 	for (nr = 0; nr < active_nr; nr++) {
 		const struct cache_entry *ce = active_cache[nr];
-		if (!S_ISREG(ce->ce_mode) || ce_intent_to_add(ce))
+		if (!S_ISREG(ce->ce_mode))
 			continue;
 		if (!ce_path_match(ce, pathspec, NULL))
 			continue;
@@ -396,7 +396,7 @@ static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int
 		 * cache version instead
 		 */
 		if (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {
-			if (ce_stage(ce))
+			if (ce_stage(ce) || ce_intent_to_add(ce))
 				continue;
 			hit |= grep_sha1(opt, ce->sha1, ce->name, 0, ce->name);
 		}
-- 8< --
--
Duy

Re: [PATCH] grep: fix grepping for "intent to add" files

[Charles Bailey]

On Thu, Jun 16, 2016 at 02:47:09PM +0700, Duy Nguyen wrote:
> I don't think revert is right. It rather needs a re-fix like below.
> Basically we want grep_file() to run as normal, but grep_sha1()
> (i.e. git grep --cached) should ignore i-t-a entries, because empty
> SHA-1 is not the right content to grep. It does not matter in positive
> matching, sure, but it may in -v cache.

You don't think the revert is correct or you don't think the revert is
sufficient? (I wasn't able to find a test case which proved that the
change to line 399 was necessary, so perhaps I don't understand.)

I would have thought that grepping the empty SHA-1 would be correct for
with or without -v. An "intent to add" file has no content in the index
so I would expect it to have zero matching and zero non-matching lines
for any grep --cached query?

Or is this an efficiency and not a correctness concern?

Charles.

Re: [PATCH] grep: fix grepping for "intent to add" files

[Duy Nguyen]

On Thu, Jun 16, 2016 at 4:47 PM, Charles Bailey <charles@hashpling.org> wrote:
> On Thu, Jun 16, 2016 at 02:47:09PM +0700, Duy Nguyen wrote:
>> I don't think revert is right. It rather needs a re-fix like below.
>> Basically we want grep_file() to run as normal, but grep_sha1()
>> (i.e. git grep --cached) should ignore i-t-a entries, because empty
>> SHA-1 is not the right content to grep. It does not matter in positive
>> matching, sure, but it may in -v cache.
>
> You don't think the revert is correct or you don't think the revert is
> sufficient? (I wasn't able to find a test case which proved that the
> change to line 399 was necessary, so perhaps I don't understand.)

OK insufficient.

> I would have thought that grepping the empty SHA-1 would be correct for
> with or without -v. An "intent to add" file has no content in the index
> so I would expect it to have zero matching and zero non-matching lines
> for any grep --cached query?
>
> Or is this an efficiency and not a correctness concern?

"git grep --cached" searches file content that will be committed by
"git commit" (no -a). An i-t-a entry will not be committed (you would
need "git add" first, or do "git commit -a"). So if I say "search
among the to-be-committed file content, list files that do not match
abc" (git grep -l -v --cached abc), the i-t-a entry will show up
because its fake content is empty (i.e. not contain "abc"), even
though it's not in the "to-be-committed" list. So yeah, correctness
issue.
-- 
Duy

Re: [PATCH] grep: fix grepping for "intent to add" files

[Charles Bailey]

On Thu, Jun 16, 2016 at 05:57:18PM +0700, Duy Nguyen wrote:
> 
> "git grep --cached" searches file content that will be committed by
> "git commit" (no -a). An i-t-a entry will not be committed (you would
> need "git add" first, or do "git commit -a"). So if I say "search
> among the to-be-committed file content, list files that do not match
> abc" (git grep -l -v --cached abc), the i-t-a entry will show up
> because its fake content is empty (i.e. not contain "abc"), even
> though it's not in the "to-be-committed" list. So yeah, correctness
> issue.

OK, I think there is an issue there but it's not with "-l -v --cached"
but rather with "-L --cached". If my understanding is correct, "-l -v"
means "has a line that doesn't match" whereas "-L" means "has no line
that matches".

Does this sound correct? I'll try adding a new test.

Re: [PATCH] grep: fix grepping for "intent to add" files

[Duy Nguyen]

On Thu, Jun 16, 2016 at 6:44 PM, Charles Bailey <charles@hashpling.org> wrote:
> On Thu, Jun 16, 2016 at 05:57:18PM +0700, Duy Nguyen wrote:
>>
>> "git grep --cached" searches file content that will be committed by
>> "git commit" (no -a). An i-t-a entry will not be committed (you would
>> need "git add" first, or do "git commit -a"). So if I say "search
>> among the to-be-committed file content, list files that do not match
>> abc" (git grep -l -v --cached abc), the i-t-a entry will show up
>> because its fake content is empty (i.e. not contain "abc"), even
>> though it's not in the "to-be-committed" list. So yeah, correctness
>> issue.
>
> OK, I think there is an issue there but it's not with "-l -v --cached"
> but rather with "-L --cached". If my understanding is correct, "-l -v"
> means "has a line that doesn't match" whereas "-L" means "has no line
> that matches".
>
> Does this sound correct? I'll try adding a new test.

Yeah "-L --cached" should work the same, I think.
-- 
Duy

Re: [PATCH] grep: fix grepping for "intent to add" files

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

>> You don't think the revert is correct or you don't think the revert is
>> sufficient? (I wasn't able to find a test case which proved that the
>> change to line 399 was necessary, so perhaps I don't understand.)
>
> OK insufficient.
>
>> I would have thought that grepping the empty SHA-1 would be correct for
>> with or without -v. An "intent to add" file has no content in the index
>> so I would expect it to have zero matching and zero non-matching lines
>> for any grep --cached query?
>>
>> Or is this an efficiency and not a correctness concern?
>
> "git grep --cached" searches file content that will be committed by
> "git commit" (no -a). An i-t-a entry will not be committed (you would
> need "git add" first, or do "git commit -a"). So if I say "search
> among the to-be-committed file content, list files that do not match
> abc" (git grep -l -v --cached abc), the i-t-a entry will show up
> because its fake content is empty (i.e. not contain "abc"), even
> though it's not in the "to-be-committed" list. So yeah, correctness
> issue.

OK, sounds like a good start for a proper log message for the fix.
Thanks for hashing it all out before I got to the end of the thread
;-)

Re: [PATCH] grep: fix grepping for "intent to add" files

[Junio C Hamano]

Charles Bailey <charles@hashpling.org> writes:

> http://thread.gmane.org/gmane.comp.version-control.git/272363/focus=276358
>
> http://thread.gmane.org/gmane.comp.version-control.git/283001/focus=283002
>
> Unless I've misunderstood the conversation and commit message, the
> referenced commit was supposed to be a "code as a comment" commit with
> no change in observable behavior

Thanks for a pointer; 276358 claims that ce->ce_mode would be zero
for path added with "git add -N path", but I do not think it is
correct.

The updated behaviour is more understandable.  With "add -N", the
user said "Just keep an eye on this path, I cannot decide what the
contents for this path in the index should be at this moment".
grep_cache() that checks the contents in the index cannot say what
is in the index, because the contents is not yet there.