Re: git clone submodules recursive and reference

[Phil Hord <phil.hord@gmail.com>]

On Fri, Apr 20, 2012 at 2:59 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 20.04.2012 17:12, schrieb Samuel Maftoul:
>> Hello,
>>
>> I'm using git clone --reference, it works like a charm !
>>
>> Now, I have submodules, so I call git clone with both --recursive and
>> --reference, works only for the repo itself, submodules are being
>> cloned without the "--reference" option.
>>
>> With GIT_TRACE=1 I can see this for the initial repo:
>>
>> trace: built-in: git 'clone' '--recursive' '--reference' [...]
>>
>> And this for the submodules:
>>
>> trace: built-in: git 'clone' '-n' [...]
>>
>> for every submodules.
>>
>> Is this an intended behavior ?
>
> Hmm, to me it looks like passing the --reference option to the clone
> run in the submodules doesn't make much sense, as that would make
> all submodules and the superproject use the same alternates. And as
> far as I know sharing objects between different repositories is not
> supported.

Suppose we had an option for submodules which would use the _relative_
submodule URL to augment the --reference path.  Would that be tenable,
or do we need some extra option?

On my Jenkins server, I have a local mirror of my repos on my Gerrit
server.  I intentionally set the mirror paths up to match the layout
of the Gerrit forest of repositories.  Thus, my relative URLs work for
local clones from this mirror as well.

Currently I am doing something like this:

myrepo=/tmp/test-mirror1
mirror=/var/lib/jenkins/mirror/superproject.git
remote=gerrit:superproject.git

#-- Clone fresh from local mirrors
git clone --recursive ${mirror} ${myrepo}
cd ${myrepo}

#-- Switch to the remote server URL
git config remote.origin.url ${remote}
git submodule sync

#-- Checkout remote updates
git pull --ff-only --recurse-submodules origin
git submodule update


In my tests, this is about twice as fast[*1*] doing it the normal way:

git clone --recursive ${remote} ${myrepo}


But I would like to just do it like this:

git clone --reference=${mirror} --recursive ${remote}  ${myrepo}

It would be silly for all the submodules to use that reference as-is,
except in the weird case where you've pull the remote objects from
several repositories into one bare container.  I would argue we don't
need to support the weird case at all, or only with some --weirdo
switch added on.  Except there is already precedent for doing this
"the weird way":

git submodule update --recursive --reference=/var/mirror/foo

There's nothing there that stops the the absolute reference being used
for each submodule, and that's glaringly discongruent to my proposed
addition to git-clone.  So I can think of two ways to move forward
with the relative-reference idea.

1. Use another switch to turn this behavior on
    git clone --reference=${mirror} \
              --submodule-relative-references \
              --recursive ${remote} ${myrepo}

2. Use a different switch name for 'reference':
    git clone --reference-forest=${mirror} \
              --recursive ${remote} ${myrepo}

Does someone have an opinion to guide me?

>> How can I force the clones for submodules to be executed with the
>> --reference option ?
>
> You'd have to use "git clone" without the --recursive option and
> then do a "git submodule update --init --reference ...".

Or the way I did it, but my method has its flaws.

Phil

[*1*] While it is twice as fast, it is subject to some fatal errors if
my mirrored repos are out of sync. That is, if a mirrored submodule
does not include the commit which is gitlinked from the
super-project's HEAD, then the original recursive clone will fail;
then I lose my speedup and possibly error out as well.  --references
does not suffer this same malady.