[PATCH v2 0/3] Teach fetch and pull to recursively fetch submodules too

[Jens Lehmann <Jens.Lehmann@web.de>]

So here is the next iteration of the recursive fetching for submodules.

Changes to the first patchset and its fixes:

* Squashed all fixes into the first patch
* Added the new config setting 'fetch.recursive' to set the default
  for all submodules

IMO this patchset should replace the one currently in pu.


Still unresolved issues:

1) Performance

   Kevin reported the fetch time went up from 1.5s to 20s for him
   because of the recursion. Kevin, could you please test the branch
   "parallel-submodule-fetch" from my github repository:

      http://github.com/jlehmann/git-submod-enhancements.git

   It has these three patches based on next plus a preliminary
   commit fetching submodules in parallel (but note that a limit
   of 128 submodules is hardcoded and the output might be mixed
   between the fetch threads, I'll fix that when you confirm the
   performance benefit I expect).

   (It took me some time to get that working, the DNS server in my
   DSL router seems to silently drop IPv6 requests. This lead to a
   15 second timeout in getaddrinfo() when forking fetch commands
   in quick succession, turning the intended speedup into a
   massive slowdown. If you happen to run into the same issue,
   please try the "broken-dns-server-hack" branch which contains a
   workaround for that issue).


2) The proper default

   Kevin argues that submodule fetching should only happen when new
   commits for the submodule have been fetched (which is kind of
   similar to what "git submodule update" does now), while I argue
   that submodules should always be fetched by default, no matter
   if new commits from submodules are committed inside the
   superproject or not.

   Ok, lets look at the basic ways you can run the fetch command and
   look at possible defaults:

   a) "git fetch --all"

      The user wanted to fetch new commits from all remotes. I think
      git should also fetch all submodules then, no matter if new
      commits from them are fetched in the superproject, as the user
      explicitly said he wants everything. Objections?

   b) "git fetch [<repository>]"

      The user wants to fetch from the default [or a single repo]. I
      think all submodules should be fetched too, Kevin thinks this
      should happen only when it is necessary (at least for his 'H'
      repository). While I think fetching all submodules too is
      consistent with the fact that you get all branches in the
      superproject too, whether you need them or not, we can't seem
      to agree on this one (also see my proposal below).

   c) "git fetch [<repository>] <refspec>"

      If the user wants to fetch only commits from a certain refspec,
      I think that it is sane to let git fetch only those submodules
      where new commits were fetched in the superproject as he was
      explicitly asking only for a subset of available commits.

   Assuming we agree on a) and - the still to be implemented - c), It
   looks like we need a new config setting for 'fetch.recursive' and
   'submodule.<name>.fetch', say 'changed'? This would allow Kevin to
   set 'submodule.H.fetch' to 'changed' in the .gitmodules describing
   it. Then "git fetch" would only then recurse further when new
   commits from 'H' have been used in the commits fetched in its
   superproject. Would that be an acceptable solution?


Jens Lehmann (3):
  fetch/pull: Recursively fetch populated submodules
  Submodules: Add the new "fetch" config option for fetch and pull
  Add the 'fetch.recursive' config setting

 Documentation/config.txt        |   10 ++
 Documentation/fetch-options.txt |   15 +++
 Documentation/gitmodules.txt    |    8 ++
 builtin/fetch.c                 |   58 ++++++++---
 git-pull.sh                     |   10 ++-
 submodule.c                     |   91 +++++++++++++++++-
 submodule.h                     |    3 +
 t/t5526-fetch-submodules.sh     |  209 +++++++++++++++++++++++++++++++++++++++
 t/t7403-submodule-sync.sh       |    2 +-
 9 files changed, 388 insertions(+), 18 deletions(-)
 create mode 100755 t/t5526-fetch-submodules.sh

-- 
1.7.3.1.108.gb6303