reflogs and worktrees?

reflogs and worktrees?

[Johannes Schindelin]

Hi Duy,

ever since I started working extensively with worktrees, I end up with
these funny gc problems, like broken links and stale reflogs.

Is it maybe possible that the reflogs (which are per-worktree now) are not
traversed completely when gc'ing (which is *not* per-worktree)?

Ciao,
Dscho

Re: reflogs and worktrees?

[Duy Nguyen]

On Tue, Jul 5, 2016 at 5:07 PM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Hi Duy,
>
> ever since I started working extensively with worktrees, I end up with
> these funny gc problems, like broken links and stale reflogs.

Yeah we have problem with gc not traversing all worktree refs. I had
something that could fix it temporarily [1] but the decision was to
wait for refs code to support new "worktree storage" so that we can
walk worktree refs properly, then fix gc [2]. We should get pretty
close to that point after [3] lands.

[1] http://thread.gmane.org/gmane.comp.version-control.git/295961/focus=296076
[2] http://thread.gmane.org/gmane.comp.version-control.git/295961/focus=296219
[3] http://thread.gmane.org/gmane.comp.version-control.git/296409

> Is it maybe possible that the reflogs (which are per-worktree now) are not
> traversed completely when gc'ing (which is *not* per-worktree)?
-- 
Duy

Re: reflogs and worktrees?

[Johannes Schindelin]

Hi Duy,

On Tue, 5 Jul 2016, Duy Nguyen wrote:

> On Tue, Jul 5, 2016 at 5:07 PM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> > Hi Duy,
> >
> > ever since I started working extensively with worktrees, I end up with
> > these funny gc problems, like broken links and stale reflogs.
> 
> Yeah we have problem with gc not traversing all worktree refs.

That is a bit of an understatement.

I get tons of errors like this one, just from the regular auto gc:

error: Could not read 296ee31af712b02469c4bb606fbf2fac229bcca6
fatal: Failed to traverse parents of commit 3c22e84e2ccdec5d8243344fc4ec68942b87a393
error: failed to run repack

I still have to address all of them, in particular because we do not (yet)
have tools to identify how certain objects are expected to be reachable
when they are missing (see my `gc-worktree` branch for my current progress
of teaching git-fsck to describe the broken links better).

For the moment, I am working around this problem with the following hook
(wrapped into a unit test to verify that it does what I expect it to do):

-- snipsnap --
Subject: [PATCH] Demonstrate a workaround for the worktree/gc problem

When gc --auto is called in the presence of worktrees, pretty much all of
the reflogs go to hell.

In the --auto case, we can install a hook that runs before-hand,
accumulates all the worktree-specific refs and reflogs and installs them
into a very special reflog in the common refs namespace. The only purpose
of this stunt is to let gc pick up on those refs and reflogs, of course,
and *not* ignore them.

Unfortunately, this still does not address the "git gc" case, but
hopefully a real fix will be here some time soon.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 t/t6500-gc.sh | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index 5d7d414..518e809 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -43,4 +43,67 @@ test_expect_success 'gc is not aborted due to a stale symref' '
 	)
 '
 
+test_expect_success 'install pre-auto-gc hook for worktrees' '
+	mkdir -p .git/hooks &&
+	write_script .git/hooks/pre-auto-gc <<-\EOF
+	echo "Preserving refs/reflogs of worktrees" >&2 &&
+	dir="$(git rev-parse --git-common-dir)" &&
+	refsdir="$dir/logs/refs" &&
+	rm -f "$refsdir"/preserve &&
+	ident="$(GIT_COMMITTER_DATE= git var GIT_COMMITTER_IDENT)" &&
+	(
+		find "$dir"/logs "$dir"/worktrees/*/logs \
+			-type f -exec cat {} \; |
+		cut -d" " -f1
+		find "$dir"/HEAD "$dir"/worktrees/*/HEAD "$dir"/refs \
+			"$dir"/worktrees/*/refs -type f -exec cat {} \; |
+		grep -v "/^ref:/"
+	) 2>/dev/null |
+	sort |
+	uniq |
+	sed "s/.*/& & $ident	dummy/" >"$dir"/preserve &&
+	mkdir -p "$refsdir" &&
+	mv "$dir"/preserve "$refsdir"/
+	EOF
+'
+
+trigger_auto_gc () {
+	# This is unfortunately very, very ugly
+	gdir="$(git rev-parse --git-common-dir)" &&
+	mkdir -p "$gdir"/objects/17 &&
+	touch "$gdir"/objects/17/17171717171717171717171717171717171717 &&
+	touch "$gdir"/objects/17/17171717171717171717171717171717171718 &&
+	git -c gc.auto=1 -c gc.pruneexpire=now -c gc.autodetach=0 gc --auto
+}
+
+test_expect_success 'gc respects refs/reflogs in all worktrees' '
+	test_commit something &&
+	git worktree add worktree &&
+	(
+		cd worktree &&
+		git checkout --detach &&
+		echo 1 >something.t &&
+		test_tick &&
+		git commit -m worktree-reflog something.t &&
+		git rev-parse --verify HEAD >../commit-reflog &&
+		echo 2 >something.t &&
+		test_tick &&
+		git commit -m worktree-ref something.t &&
+		git rev-parse --verify HEAD >../commit-ref
+	) &&
+	trigger_auto_gc &&
+	git rev-parse --verify $(cat commit-ref)^{commit} &&
+	git rev-parse --verify $(cat commit-reflog)^{commit} &&
+
+	# Now, add a reflog in the top-level dir and verify that `git gc` in
+	# the worktree does not purge that
+	git checkout --detach &&
+	echo 3 >something.t &&
+	test_tick &&
+	git commit -m commondir-reflog something.t &&
+	git rev-parse --verify HEAD >commondir-reflog &&
+	(cd worktree && trigger_auto_gc) &&
+	git rev-parse --verify $(cat commondir-reflog)^{commit}
+'
+
 test_done
-- 
2.9.0.278.g1caae67