Re: Git is not scalable with too many refs/*

[Michael Haggerty <mhagger@alum.mit.edu>]

I have answered some of my own questions:

On 09/09/2011 03:50 PM, Michael Haggerty wrote:
> 3. Try using the --no-replace-objects option (I assume that it can be
> used like "git --no-replace-objects fetch ...").  In my case this option
> made a dramatic improvement in the runtimes.

This does not seem to help much.

> 4. Try a test using a repository generated something like the test
> script that I posted in [1].  If it also gives pathologically bad
> performance, then it can serve as a test case to use while we debug the
> problem.

Yes, a simple test repo like that created by the script is enough to
reproduce the problem.  The slowdown becomes very obvious after only a
few hundred references.

Curiously, "git clone" is very fast under the same circumstances that
"git fetch" is excruciatingly slow.

According to strace, git seems to be repopulating the ref cache after
each new ref is created (it walks through the whole refs subdirectory
and reads every file).  Apparently the ref cache is being discarded
completely whenever a ref is added (which can and should be fixed) and
then being reloaded for some reason (though single refs can be inspected
much faster without reading the cache).  This situation should be
improved by the hierarchical refcache changes that I'm working on plus
smarter updating (rather than discarding) of the cache when a new
reference is created.

Some earlier speculation in this thread was that that slowdowns might be
caused by "pessimal" ordering of revisions in the walker queue.  But my
test repository shards the references in such a way that the lexical
order of the refnames does not correspond to the topological order of
the commits.  So that can't be the whole story.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/