Re: [PATCH v3] rev-list: exclude promisor objects at walk time

From: Jeff King <peff@peff.net>
To: "Josh Steadmon" <steadmon@google.com>,
	"SZEDER Gábor" <szeder.dev@gmail.com>,
	git@vger.kernel.org, jonathantanmy@google.com,
	jrnieder@gmail.com
Subject: Re: [PATCH v3] rev-list: exclude promisor objects at walk time
Date: Sat, 13 Apr 2019 01:34:35 -0400	[thread overview]
Message-ID: <20190413053434.GA31884@sigill.intra.peff.net> (raw)
In-Reply-To: <20190412223847.GO60888@google.com>

On Fri, Apr 12, 2019 at 03:38:47PM -0700, Josh Steadmon wrote:

> > > Thank you for catching this. I haven't yet figured out the cause. I will
> > > look into this more tomorrow and then send a V4 once I've fixed it.
> > 
> > I'm concerned that this is a sign that the approach I suggested does not
> > actually work everywhere. I.e., could this be a case where we have some
> > non-promisor object that points to a sub-object that is reachable from
> > the promisor pack, but not a direct tip? Before your patch we'd consider
> > that sub-object a promisor (because we enumerate all of the graph that
> > we do have and mark each such object), but afterwards we would not.
> > 
> > And I wonder if that confuses pack-objects. Though I think it would
> > confuse it in the _opposite_ direction. I.e., using
> > --exclude-promisor-objects would count such an object as not-a-promisor
> > and would be more inclined to include it in the new pack.
> > 
> > It is curious that this only turns up with GIT_TEST_COMMIT_GRAPH=1, too.
> > It seems like any such problem ought to be independent of that.
> > 
> > Puzzling...
> 
> Do you think this justifies going back to the V1 approach (only checking
> presence of objects pointed to by refs when doing a partial clone)?

Yes, I think it might. Especially coupled with your other report that
the V1 approach is 500ms compared to several seconds for this one. Which
I'd guess is probably because we actually parse the ref tip objects in
rev-list, whereas your V1 just skipped that step entirely (which is
perfectly fine for a clone, as we'd have just hashed the objects via
index-pack anyway).

It might be interesting to know if the problem is indeed insurmountable
with the V3 approach here, or if it's simply another bug. But diving
into it is going to be rather tricky, and I am not volunteering to do
it. :) So if you want to punt and go back to the more clearly correct V1
approach, I can live with that. We can always revisit this approach
later (it wouldn't be necessary for the clone case after your V1, but in
theory it could be helping other cases, too).

-Peff