Re: [PU PATCH] Fix git fetch for very large ref counts

[Julian Phillips <julian@quantumfyre.co.uk>]

On Mon, 12 Feb 2007, Junio C Hamano wrote:

> "Julian Phillips" <jp3@quantumfyre.co.uk> writes:
>
>> The updated git fetch in pu is vastly improved on repositories with very
>> large numbers of refs.  The time taken for a no-op fetch over ~9000 refs
>> drops from ~48m to ~0.5m.
>>
>> However, before git fetch will actually run on a repository with ~9000
>> refs the calling interface between fetch and fetch--tool needs to be
>> changed.  The existing version passes the entire reflist on the command
>> line, which means that it is subject to the maxiumum environment size
>> passed to a child process by execve.
>>
>> The following patches add a stdin based interface to fetch--tool allowing
>> the ~9000 refs to be passed without exceeding the environment limit.
>
> Thanks.
>
> But the ones in 'pu' were done primarily as demonstration of
> where the bottlenecks are, and not meant for real-world
> consumption.  I think the final shaving of 0.5m down to a few
> seconds needs to move the ls_remote_result string currently kept
> as a shell variable to a list of strings represented in a
> git-fetch largely rewritten in C, and at that point the
> interface from outside fetch--tool to throw 9000 refs at it
> would be an internal function call and the code you fixed along
> with new function you introduced would probably need to be
> discarded.

And there I was thinking 0.5m was fast ... given how long I've been 
reading this list I really should have know better. ;)

I only really made the changes so I could try your improvements to fetch 
out - if they aren't necessary because you're making it even faster then I 
really don't have much cause to complain.

-- 
Julian

  ---
To be a kind of moral Unix, he touched the hem of Nature's shift.
 		-- Shelley