[PATCH v2] commit.c: don't persist substituted parents when unshallowing

[Taylor Blau <me@ttaylorr.com>]

Since 37b9dcabfc (shallow.c: use '{commit,rollback}_shallow_file',
2020-04-22), Git knows how to reset stat-validity checks for the
$GIT_DIR/shallow file, allowing it to change between a shallow and
non-shallow state in the same process (e.g., in the case of 'git fetch
--unshallow').

However, when $GIT_DIR/shallow changes, Git does not alter or remove any
grafts (nor substituted parents) in memory.

This comes up in a "git fetch --unshallow" with fetch.writeCommitGraph
set to true. Ordinarily in a shallow repository (and before 37b9dcabfc,
even in this case), commit_graph_compatible() would return false,
indicating that the repository should not be used to write a
commit-graphs (since commit-graph files cannot represent a shallow
history). But since 37b9dcabfc, in an --unshallow operation that check
succeeds.

Thus even though the repository isn't shallow any longer (that is, we
have all of the objects), the in-core representation of those objects
still has munged parents at the shallow boundaries.  When the
commit-graph write proceeds, we use the incorrect parentage, producing
wrong results.

There are two ways for a user to work around this: either (1) set
'fetch.writeCommitGraph' to 'false', or (2) drop the commit-graph after
unshallowing.

One way to fix this would be to reset the parsed object pool entirely
(flushing the cache and thus preventing subsequent reads from modifying
their parents) after unshallowing. That would produce a problem when
callers have a now-stale reference to the old pool, and so this patch
implements a different approach. Instead, attach a new bit to the pool,
'substituted_parent', which indicates if the repository *ever* stored a
commit which had its parents modified (i.e., the shallow boundary
prior to unshallowing).

This bit needs to be sticky because all reads subsequent to modifying a
commit's parents are unreliable when unshallowing. Modify the check in
'commit_graph_compatible' to take this bit into account, and correctly
avoid generating commit-graphs in this case, thus solving the bug.

Helped-by: Derrick Stolee <dstolee@microsoft.com>
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Reported-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
The patch contents are unchanged from v1, but the patch's message is
wordsmithed with jrn's help. This should be ready for -rc0 now.

Range-diff below for easier review:

Range-diff against v1:
1:  82939831ad ! 1:  1f769bbbb4 commit.c: don't persist substituted parents when unshallowing
    @@ Metadata
      ## Commit message ##
         commit.c: don't persist substituted parents when unshallowing

    -    In 37b9dcabfc (shallow.c: use '{commit,rollback}_shallow_file',
    -    2020-04-22), Git learned how to reset stat-validity checks for the
    -    '$GIT_DIR/shallow' file, allowing it to change between a shallow and
    +    Since 37b9dcabfc (shallow.c: use '{commit,rollback}_shallow_file',
    +    2020-04-22), Git knows how to reset stat-validity checks for the
    +    $GIT_DIR/shallow file, allowing it to change between a shallow and
         non-shallow state in the same process (e.g., in the case of 'git fetch
         --unshallow').

    -    However, 37b9dcabfc does not alter or remove any grafts nor substituted
    -    parents. This produces a problem when unshallowing if another part of
    -    the code relies on the un-grafted and/or non-substituted parentage for
    -    commits after, say, fetching.
    +    However, when $GIT_DIR/shallow changes, Git does not alter or remove any
    +    grafts (nor substituted parents) in memory.

    -    This can arise 'fetch.writeCommitGraph' is true. Ordinarily (and
    -    certainly previous to 37b9dcabfc), 'commit_graph_compatible()' would
    -    return 0, indicating that the repository should not generate
    -    commit-graphs (since at one point in the same process it was shallow).
    -    But with 37b9dcabfc, that check succeeds.
    +    This comes up in a "git fetch --unshallow" with fetch.writeCommitGraph
    +    set to true. Ordinarily in a shallow repository (and before 37b9dcabfc,
    +    even in this case), commit_graph_compatible() would return false,
    +    indicating that the repository should not be used to write a
    +    commit-graphs (since commit-graph files cannot represent a shallow
    +    history). But since 37b9dcabfc, in an --unshallow operation that check
    +    succeeds.

    -    This is where the bug occurs: even though the repository isn't shallow
    -    any longer (that is, we have all of the objects), the in-core
    -    representation of those objects still has munged parents at the shallow
    -    boundaries. If a commit-graph write proceeds, we will use the incorrect
    -    parentage, producing wrong results.
    +    Thus even though the repository isn't shallow any longer (that is, we
    +    have all of the objects), the in-core representation of those objects
    +    still has munged parents at the shallow boundaries.  When the
    +    commit-graph write proceeds, we use the incorrect parentage, producing
    +    wrong results.

    -    (Prior to this patch, there were two ways of fixing this: either (1)
    -    set 'fetch.writeCommitGraph' to 'false', or (2) drop the commit-graph
    -    after unshallowing).
    +    There are two ways for a user to work around this: either (1) set
    +    'fetch.writeCommitGraph' to 'false', or (2) drop the commit-graph after
    +    unshallowing.

         One way to fix this would be to reset the parsed object pool entirely
         (flushing the cache and thus preventing subsequent reads from modifying
    -    their parents) after unshallowing. This can produce a problem when
    +    their parents) after unshallowing. That would produce a problem when
         callers have a now-stale reference to the old pool, and so this patch
    -    implements a different approach. Instead, we attach a new bit to the
    -    pool, 'substituted_parent' which indicates if the repository *ever*
    -    stored a commit which had its parents modified (i.e., the shallow
    -    boundary *before* unshallowing).
    +    implements a different approach. Instead, attach a new bit to the pool,
    +    'substituted_parent', which indicates if the repository *ever* stored a
    +    commit which had its parents modified (i.e., the shallow boundary
    +    prior to unshallowing).

    -    This bit is sticky, since all subsequent reads after modifying a
    -    commit's parent are unreliable when unshallowing. This patch modifies
    -    the check in 'commit_graph_compatible' to take this bit into account,
    -    and correctly avoid generating commit-graphs in this case.
    +    This bit needs to be sticky because all reads subsequent to modifying a
    +    commit's parents are unreliable when unshallowing. Modify the check in
    +    'commit_graph_compatible' to take this bit into account, and correctly
    +    avoid generating commit-graphs in this case, thus solving the bug.

         Helped-by: Derrick Stolee <dstolee@microsoft.com>
         Helped-by: Jonathan Nieder <jrnieder@gmail.com>
         Reported-by: Jay Conrod <jayconrod@google.com>
    +    Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
         Signed-off-by: Taylor Blau <me@ttaylorr.com>

      ## commit-graph.c ##

 commit-graph.c           |  3 ++-
 commit.c                 |  2 ++
 object.h                 |  1 +
 t/t5537-fetch-shallow.sh | 14 ++++++++++++++
 4 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/commit-graph.c b/commit-graph.c
index fdd1c4fa7c..328ab06fd4 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -203,7 +203,8 @@ static int commit_graph_compatible(struct repository *r)
 	}

 	prepare_commit_graft(r);
-	if (r->parsed_objects && r->parsed_objects->grafts_nr)
+	if (r->parsed_objects &&
+	    (r->parsed_objects->grafts_nr || r->parsed_objects->substituted_parent))
 		return 0;
 	if (is_repository_shallow(r))
 		return 0;
diff --git a/commit.c b/commit.c
index 43d29a800d..7128895c3a 100644
--- a/commit.c
+++ b/commit.c
@@ -423,6 +423,8 @@ int parse_commit_buffer(struct repository *r, struct commit *item, const void *b
 	pptr = &item->parents;

 	graft = lookup_commit_graft(r, &item->object.oid);
+	if (graft)
+		r->parsed_objects->substituted_parent = 1;
 	while (bufptr + parent_entry_len < tail && !memcmp(bufptr, "parent ", 7)) {
 		struct commit *new_parent;

diff --git a/object.h b/object.h
index 38dc2d5a6c..96a2105859 100644
--- a/object.h
+++ b/object.h
@@ -25,6 +25,7 @@ struct parsed_object_pool {
 	char *alternate_shallow_file;

 	int commit_graft_prepared;
+	int substituted_parent;

 	struct buffer_slab *buffer_slab;
 };
diff --git a/t/t5537-fetch-shallow.sh b/t/t5537-fetch-shallow.sh
index d427a2d7f7..a55202d2d3 100755
--- a/t/t5537-fetch-shallow.sh
+++ b/t/t5537-fetch-shallow.sh
@@ -81,6 +81,20 @@ test_expect_success 'fetch --unshallow from shallow clone' '
 	)
 '

+test_expect_success 'fetch --unshallow from a full clone' '
+	git clone --no-local --depth=2 .git shallow3 &&
+	(
+	cd shallow3 &&
+	git log --format=%s >actual &&
+	test_write_lines 4 3 >expect &&
+	test_cmp expect actual &&
+	git -c fetch.writeCommitGraph fetch --unshallow &&
+	git log origin/master --format=%s >actual &&
+	test_write_lines 4 3 2 1 >expect &&
+	test_cmp expect actual
+	)
+'
+
 test_expect_success 'fetch something upstream has but hidden by clients shallow boundaries' '
 	# the blob "1" is available in .git but hidden by the
 	# shallow2/.git/shallow and it should be resent
--
2.27.0.225.g9fa765a71d