Make "git checkout" automatically update submodules?

Make "git checkout" automatically update submodules?

[Kannan Goundan]

Git submodules seem to be a great fit for one of our repos.  The biggest
pain point is that it's too easy to forget to update submodules.

1. I often forget since most repos don't need it.
2. Infrequent users of our repo almost never know to update submodules and
end up coming to us with strange build errors.
3. Existing scripts that work with Git repos are usually not built to handle
submodules.

In the common case of the submodule content having no local edits, it would
be nice if "git checkout" automatically updated submodules [1].  If there
are local edits, it could error out (maybe override with
"--ignore-modified-submodules" or something).

I'm not a Git expert, though.  Is there a reason something like this isn't
already implemented?  Maybe there's an existing write-up or mailing list
thread I can read to get some background information?

Thanks!

[1] Our post-checkout procedure is:

    git submodule sync
    git submodule update --init
    git submodule foreach --recursive \
      'git submodule sync ; git submodule update --init'

(Not sure if this is correct.  Different articles/blogs suggest a slightly
different set of commands.)

Re: Make "git checkout" automatically update submodules?

[Junio C Hamano]

Kannan Goundan <kannan@cakoose.com> writes:

> Git submodules seem to be a great fit for one of our repos.  The biggest
> pain point is that it's too easy to forget to update submodules.
> ...
> In the common case of the submodule content having no local edits, it would
> be nice if "git checkout" automatically updated submodules [1].  If there
> are local edits, it could error out (maybe override with
> "--ignore-modified-submodules" or something).
>
> I'm not a Git expert, though.  Is there a reason something like this isn't
> already implemented?  Maybe there's an existing write-up or mailing list
> thread I can read to get some background information?

I think it is mostly because the area has a lot of corner cases and
different workflows.  For example ...

> [1] Our post-checkout procedure is:
>
>     git submodule sync
>     git submodule update --init
>     git submodule foreach --recursive \
>       'git submodule sync ; git submodule update --init'

... this will clearly not work well for everybody, as you do not
want to instantiate _all_ the submodules _unconditionally_.  Well,
"you" (Kannan) and your project may want to, but not necessarily
other large projects (e.g. imagine Android using submodules).

So you can view the current status being "nothing is instantiated by
default", which _is_ far from satisfactory than the ideal case where
perhaps the project can specify in .gitmodules which submodules are
to be instantiated by default, add labels to modules in .gitmodules
so that "git clone" of a top-level project with submodules can be
told "git clone --init-submodules=<label>" to instantiate the modules
that match the given label after the top-level is cloned, etc. etc.

But such a way to allow these more complicated situations to be
handled nicely has not been designed by anybody, so for now "nothing
instantiated by default" stands as the safest default.

Re: Make

[Kannan Goundan]

Thanks for the explanation.  I didn't realize some projects don't want to
initialize all their submodules, but the explicit opt-in idea you described
sounds nice.

I've seen cases where people will financially "sponsor" feature development
in open source projects.  Is there any precedent for this in the Git
project?  Is it ok to use this mailing list to ask about such things?

Re: Make

[Junio C Hamano]

Kannan Goundan <kannan@cakoose.com> writes:

> Thanks for the explanation.  I didn't realize some projects don't want to
> initialize all their submodules, but the explicit opt-in idea you described
> sounds nice.
>
> I've seen cases where people will financially "sponsor" feature development
> in open source projects.  Is there any precedent for this in the Git
> project?  Is it ok to use this mailing list to ask about such things?

We are unfortunately not set up to handle money well.  For a
background explanation, please go read [*1*], which I wrote my take
on "money" some time ago.  Note that it is an explanation and not a
justification.  It explains why we are not set up to handle money
well and what the issues around money that are troublesome for the
project are.  It does not mean to say that it is a good thing that
it is hard to buy feature with money from our project [*2*].

I do not see (and back then in the discussion I do not think anybody
saw) how we can make "We, a sponsoring company, pay this money to
the project to fund effort Y." work well.  But that of course does
not mean it is impossible to make it work--somebody with a fresh
perspective may come up a way to do so, and that would be a very
welcome development.


[Footnote/Reference]

*1* http://thread.gmane.org/gmane.comp.version-control.git/264993/focus=265215

*2* Just like my message that you are responding to was an
    explanation of the reason why we do not recurse and and
    initialize all submodules by default.

Re: Make "git checkout" automatically update submodules?

[Kannan Goundan]

Junio C Hamano <gitster <at> pobox.com> writes:

> We are unfortunately not set up to handle money well.  For a
> background explanation, please go read [*1*], which I wrote my take
> on "money" some time ago.  Note that it is an explanation and not a
> justification.  It explains why we are not set up to handle money
> well and what the issues around money that are troublesome for the
> project are.  It does not mean to say that it is a good thing that
> it is hard to buy feature with money from our project [*2*].

I think the way I described it ("sponsoring a feature") doesn't really
reflect how I was imagining it.  In my head, it looked like this:

1. Figure out whether the Git community and maintainers seem ok with the
overall feature idea.  If not, give up.
2. Come up with a plan for the UI/UX; see if the Git community and
maintainers seem ok with it.  If not, iterate or give up.
3. Implement it, then go through the regular process of getting it merged
upstream.  If it doesn't go well, might have to iterate or give up.

I could try doing that myself, but someone familiar with the Git
codebase/community/history would be better at it (and probably be easier for
you guys to work with :-)

I guess I'm just wondering if there are people who meet those qualifications
and are interested in going through those steps for pay.  Or maybe there's a
company that does this, like the old Cygnus Solutions?

In particular, I don't expect anything to change about the project's
development process.

(This part is not relevant to the Git project, but I understand that it's
hard for anyone to guarantee a feature will make it into an open source
project.  I imagine these kinds of contracts are set up so that you're
primarily paying for the effort, not the outcome.  If it ends up not working
out, you don't get your money back.)

Re: Make "git checkout" automatically update submodules?

[Christian Couder]

On Fri, Oct 23, 2015 at 5:46 AM, Kannan Goundan <kannan@cakoose.com> wrote:
> Junio C Hamano <gitster <at> pobox.com> writes:
>
>> We are unfortunately not set up to handle money well.  For a
>> background explanation, please go read [*1*], which I wrote my take
>> on "money" some time ago.  Note that it is an explanation and not a
>> justification.  It explains why we are not set up to handle money
>> well and what the issues around money that are troublesome for the
>> project are.  It does not mean to say that it is a good thing that
>> it is hard to buy feature with money from our project [*2*].
>
> I think the way I described it ("sponsoring a feature") doesn't really
> reflect how I was imagining it.  In my head, it looked like this:
>
> 1. Figure out whether the Git community and maintainers seem ok with the
> overall feature idea.  If not, give up.
> 2. Come up with a plan for the UI/UX; see if the Git community and
> maintainers seem ok with it.  If not, iterate or give up.
> 3. Implement it, then go through the regular process of getting it merged
> upstream.  If it doesn't go well, might have to iterate or give up.
>
> I could try doing that myself, but someone familiar with the Git
> codebase/community/history would be better at it (and probably be easier for
> you guys to work with :-)
>
> I guess I'm just wondering if there are people who meet those qualifications
> and are interested in going through those steps for pay.  Or maybe there's a
> company that does this, like the old Cygnus Solutions?

Well I am starting to do that for Booking.com. Not sure if it will
also be possible for me to work for you as I also work on IPFS
(http://ipfs.io) for the company that develops it, but we can discuss
it privately.

There was David Kastrup (dak at gnu.org) who previously said he could
be interested in such jobs. We wrote a very short article about it in
the first edition of Git Rev News last March:

http://git.github.io/rev_news/2015/03/25/edition-1/

We also wrote a very short article "Job Offer" article about
Booking.com looking for Git developers in Git Rev News in the third
edition last May:

http://git.github.io/rev_news/2015/05/13/edition-3/

so if you want we can write a similar "Job Offer" article in the next
Git Rev News edition.

You can even propose such an article yourself by editing the draft of
the next edition here:

https://github.com/git/git.github.io/blob/master/rev_news/drafts/edition-9.md

and then creating a pull request.

> In particular, I don't expect anything to change about the project's
> development process.
>
> (This part is not relevant to the Git project, but I understand that it's
> hard for anyone to guarantee a feature will make it into an open source
> project.  I imagine these kinds of contracts are set up so that you're
> primarily paying for the effort, not the outcome.  If it ends up not working
> out, you don't get your money back.)

Re: Make "git checkout" automatically update submodules?

[Junio C Hamano]

Kannan Goundan <kannan@cakoose.com> writes:

> I think the way I described it ("sponsoring a feature") doesn't really
> reflect how I was imagining it.  In my head, it looked like this:
> ...
> I could try doing that myself, but someone familiar with the Git
> codebase/community/history would be better at it (and probably be easier for
> you guys to work with :-)
>
> I guess I'm just wondering if there are people who meet those qualifications
> and are interested in going through those steps for pay.  Or maybe there's a
> company that does this, like the old Cygnus Solutions?
>
> In particular, I don't expect anything to change about the project's
> development process.

In other words, the sponsoring entity is paying for effort and not
for result--money does not buy inclusion.

That may be workable from the project's point of view; I however
wonder if that is workable from the sponsor's point of view.  Things
that may be problematic inside the sponsoring entity (i.e. between
those with money and those who interact with the hired person)
include:

 - Does the hired person really meet the right qualifications?

 - Did the hired person make a good faith effort?

 - Was it the right time to start the topic, or was the codebase too
   much in flux at the moment to accept work in that area?

But these are problems between the sponsor and the hired person and
do not concern us ;-)

Re: Make "git checkout" automatically update submodules?

[Stefan Beller]

On Thu, Oct 15, 2015 at 3:50 PM, Kannan Goundan <kannan@cakoose.com> wrote:
> Git submodules seem to be a great fit for one of our repos.  The biggest
> pain point is that it's too easy to forget to update submodules.
>
> 1. I often forget since most repos don't need it.
> 2. Infrequent users of our repo almost never know to update submodules and
> end up coming to us with strange build errors.
> 3. Existing scripts that work with Git repos are usually not built to handle
> submodules.
>
> In the common case of the submodule content having no local edits, it would
> be nice if "git checkout" automatically updated submodules [1].  If there
> are local edits, it could error out (maybe override with
> "--ignore-modified-submodules" or something).
>
> I'm not a Git expert, though.  Is there a reason something like this isn't
> already implemented?  Maybe there's an existing write-up or mailing list
> thread I can read to get some background information?
>
> Thanks!
>
> [1] Our post-checkout procedure is:
>
>     git submodule sync
>     git submodule update --init
>     git submodule foreach --recursive \
>       'git submodule sync ; git submodule update --init'
>
> (Not sure if this is correct.  Different articles/blogs suggest a slightly
> different set of commands.)
>

Checkout [1]. There are lots of good patches, but hard to find.
(Including, but not limited to a recursive git checkout enhancement!)

That said I've recently started working on submodules, too.
I am trying to push my work upstream as fast as possible
as that works best for us.

[1] https://github.com/jlehmann/git-submod-enhancements/wiki

Re: Make "git checkout" automatically update submodules?

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> Checkout [1]. There are lots of good patches, but hard to find.
> (Including, but not limited to a recursive git checkout enhancement!)
> ...
> [1] https://github.com/jlehmann/git-submod-enhancements/wiki

Yes, Jens is not just one of the people who have been working on
harder, and thinking longer about, submodules than anybody else, but
also has demonstrated that he has good taste and balanced view on
the design of the subsystem over time, whose technical judgment we
can trust.

Not all the changes listed on the page may necessarily be good as-is
(e.g. some may help only a subset of users while hurting others,
like the "recursively check-out everything unconditionally" that
trigerred this thread), but the page has a good collection to remind
anybody, who designs a coherent whole, of issues that need to be
taken into account.

Thanks for a pointer to an excellent starting page.

Re: Make "git checkout" automatically update submodules?

[Kannan Goundan]

Stefan Beller <sbeller <at> google.com> writes:

> [1] https://github.com/jlehmann/git-submod-enhancements/wiki

Oh wow, Christmas came early!  I'll give this code a try.