Re: Performance issue: initial git clone causes massive repack

[Jon Smirl <jonsmirl@gmail.com>]

2009/4/7 Jakub Narebski <jnareb@gmail.com>:
> On Tue, 7 Apr 2009, Nicolas Pitre wrote:
>> On Tue, 7 Apr 2009, Jakub Narebski wrote:
>>> Björn Steinbrink <B.Steinbrink@gmx.de> writes:
>
> [...]
>>>> It's not primarily pack-objects, I think. It's the rev-list that's run
>>>> by upload-pack.  Running "git rev-list --objects --all" on that repo
>>>> eats about 2G RSS, easily killing the system's cache on a small box,
>>>> leading to swapping and a painful time reading the packfile contents
>>>> afterwards to send them to the client.
>>>
>>> Than I think that "packfile caching" GSoC project (which is IIRC
>>> "object enumeration caching", or at least includes it) should help
>>> here.
>>
>> NO!
>>
>> Please people stop being so creative with all sort of ways to simply
>> avoid the real issue and focussing on a real fix.  Git has not become
>> what it is today by the accumulation of workarounds and ignorance of
>> fundamental issues.
>>
>> Having git-rev-list consume about 2G RSS for the enumeration of 4M
>> objects is simply inacceptable, period.  This is the equivalent of 500
>> bytes per object pinned in memory on average, just for listing object,
>> which is completely silly. We ought to do better than that.
>
> I have thought that the large amount of memory consumed by git-rev-list
> was caused by not-so-sequential access to very large packfile (1.5GB+ if
> I remember correctly), which I thought causes the whole packfile to be
> mmapped and not only window, plus large amount of objects in 300MB+ mem
> range or something; those both would account for around 2GB.

I don't know all of the finer details of chasing revision lists, but
would it help if pack files recorded the root IDs of their object
trees at creation time and stored it in the front of the pack?


>
> Besides even if git-rev-list wouldn't take so much memory, object
> enumeration caching would still help with CPU load... admittedly less.
>
> --
> Jakub Narebski
> Poland
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Jon Smirl
jonsmirl@gmail.com