recursive cloning with --reference-if-able + --dissociate

recursive cloning with --reference-if-able + --dissociate

[Rasmus Villemoes]

At $dayjob, most of our projects are organized as a rather small
superproject, with a number of git submodules. Usually, one of those
submodules is a linux kernel git tree.

For CI testing, we create a fresh docker container which starts by doing
a "git clone --recursive $superproject", which takes quite a while due
to the linux repo. So we'd like to speed that up by having a reasonable
up-to-date linux repo on each build slave, mapping that into the docker
container, and then telling git "when you get to cloning the src/linux
submodule, look over here for objects", i.e. having some way to have
--reference-if-able and --disassociate in effect for submodules.

If this is already possible, I can't find it in the documentation. And
if it doesn't exist, I can't really think of a good way to define the UI
for giving those options (other than, perhaps, having a top-level
--reference-if-able pass down to the submodules with the meaning 'when
initializing the submodule at some/path, try to use the submodule, if
any, at some/path in the referenced top-level repo'".

Rasmus

Re: recursive cloning with --reference-if-able + --dissociate

[Junio C Hamano]

Rasmus Villemoes <rv@rasmusvillemoes.dk> writes:

> At $dayjob, most of our projects are organized as a rather small
> superproject, with a number of git submodules. Usually, one of those
> submodules is a linux kernel git tree.
>
> For CI testing, we create a fresh docker container which starts by doing
> a "git clone --recursive $superproject", which takes quite a while due
> to the linux repo. So we'd like to speed that up by having a reasonable
> up-to-date linux repo on each build slave, mapping that into the docker
> container, and then telling git "when you get to cloning the src/linux
> submodule, look over here for objects", i.e. having some way to have
> --reference-if-able and --disassociate in effect for submodules.
>
> If this is already possible, I can't find it in the documentation. And
> if it doesn't exist, I can't really think of a good way to define the UI
> for giving those options (other than, perhaps, having a top-level
> --reference-if-able pass down to the submodules with the meaning 'when
> initializing the submodule at some/path, try to use the submodule, if
> any, at some/path in the referenced top-level repo'".

Sorry for not giving a full solution but I suspect that discussing
this topic will lead to a useful new feature, similar in spirit to
the url rewriting (i.e. "url.$LOCAL.insteadOf = $REMOTE") to tell
Git to use a local mirror $LOCAL instead of the remote authentic
repository when interacting with $REMOTE.

Perhaps "url.$LOCALREF.autoReference = $REMOTE" to specify a local
mirror that does not have to be completely up-to-date, that will
altomatically be used as if "--reference-if-able $LOCALREF" is
passed when running "git clone $REMOTE", or something like that.

The actual design has to get a lot more involved, including
deciding:

 - if autoReference is orthogonal to --disssociate (I think it
   should be but others may differ).

 - how the way --dissociate is passed down from the top-level
   superproject to cloning of submodules is controlled (passing
   unconditionally would probably be too coarse, but I dunno).

etc. etc.