Re: [PATCH v3] rev-list: exclude promisor objects at walk time

From: Josh Steadmon <steadmon@google.com>
To: "SZEDER Gábor" <szeder.dev@gmail.com>
Cc: git@vger.kernel.org, peff@peff.net, jonathantanmy@google.com,
	jrnieder@gmail.com
Subject: Re: [PATCH v3] rev-list: exclude promisor objects at walk time
Date: Tue, 9 Apr 2019 16:42:55 -0700	[thread overview]
Message-ID: <20190409234255.GM60888@google.com> (raw)
In-Reply-To: <20190409180418.GH8796@szeder.dev>

On 2019.04.09 20:04, SZEDER Gábor wrote:
> On Mon, Apr 08, 2019 at 02:06:04PM -0700, Josh Steadmon wrote:
> > For large repositories, enumerating the list of all promisor objects (in
> > order to exclude them from a rev-list walk) can take a significant
> > amount of time).
> > 
> > When --exclude-promisor-objects is passed to rev-list, don't enumerate
> > the promisor objects. Instead, filter them (and any children objects)
> > during the actual graph walk.
> > 
> > Remove the mark_uninteresting() function as it's not used anywhere else.
> > 
> > When testing against a large repo [1], this patch reduces the
> > connectivity check runtime from 3 minutes to ~7 seconds.
> 
> This patch breaks test 'repack -d does not irreversibly delete
> promisor objects' in 't0410-partial-clone.sh' when run with
> GIT_TEST_COMMIT_GRAPH=1.
> 
>   +rm -rf repo
>   +test_create_repo repo
>   +test 1 = 1
>   +repo=repo
>   +mkdir -p repo
>   +cd repo
>   +/home/travis/build/git/git/t/../git init --template=/home/travis/build/git/git/t/../templates/blt/
>   Initialized empty Git repository in /home/travis/build/git/git/t/trash directory.t0410-partial-clone/repo/.git/
>   +mv .git/hooks .git/hooks-disabled
>   +git -C repo config core.repositoryformatversion 1
>   +git -C repo config extensions.partialclone arbitrary string
>   +git -C repo commit --allow-empty -m one
>   [master (root-commit) 71905df] one
>    Author: A U Thor <author@example.com>
>   +git -C repo commit --allow-empty -m two
>   [master 202c4a3] two
>    Author: A U Thor <author@example.com>
>   +git -C repo commit --allow-empty -m three
>   [master 4737577] three
>    Author: A U Thor <author@example.com>
>   +git -C repo commit --allow-empty -m four
>   [master d6ba7e0] four
>    Author: A U Thor <author@example.com>
>   +git -C repo rev-parse HEAD^^^
>   +ONE=71905dfcd543b7cbb0b4b66fbd20379e67220557
>   +git -C repo rev-parse HEAD^^
>   +TWO=202c4a3dd9a2dac927f056abb747cce9ea2eb67b
>   +git -C repo rev-parse HEAD^
>   +THREE=47375779ebcca4b422e3afdd14aa37a358081297
>   +pack_as_from_promisor
>   +printf 202c4a3dd9a2dac927f056abb747cce9ea2eb67b\n
>   +git -C repo pack-objects .git/objects/pack/pack
>   +HASH=2e675cd706e508d6c52a21d28cfcddde5ec02a06
>   +
>   +echo 2e675cd706e508d6c52a21d28cfcddde5ec02a06
>   2e675cd706e508d6c52a21d28cfcddde5ec02a06
>   +printf 47375779ebcca4b422e3afdd14aa37a358081297\n
>   +pack_as_from_promisor
>   +git -C repo pack-objects .git/objects/pack/pack
>   +HASH=31f7d2797549ab9b1c425a9e60eb2030481170e5
>   +
>   +echo 31f7d2797549ab9b1c425a9e60eb2030481170e5
>   31f7d2797549ab9b1c425a9e60eb2030481170e5
>   +delete_object repo 71905dfcd543b7cbb0b4b66fbd20379e67220557
>   +sed -e s|^..|&/|
>   +echo 71905dfcd543b7cbb0b4b66fbd20379e67220557
>   +rm repo/.git/objects/71/905dfcd543b7cbb0b4b66fbd20379e67220557
>   +repack_and_check -a 202c4a3dd9a2dac927f056abb747cce9ea2eb67b 47375779ebcca4b422e3afdd14aa37a358081297
>   +rm -rf repo2
>   +cp -r repo repo2
>   +git -C repo2 repack -a -d
>   warning: reflog of 'HEAD' references pruned commits
>   warning: reflog of 'refs/heads/master' references pruned commits
>   fatal: unable to read 71905dfcd543b7cbb0b4b66fbd20379e67220557
>   error: last command exited with $?=128
>   not ok 23 - repack -d does not irreversibly delete promisor objects
> 
> 
>   https://travis-ci.org/git/git/jobs/517874310#L5822
> 

Thank you for catching this. I haven't yet figured out the cause. I will
look into this more tomorrow and then send a V4 once I've fixed it.