[PATCH 00/10] decoupling a submodule's existence and its url

[PATCH 00/10] decoupling a submodule's existence and its url

[Brandon Williams]

There are two motivations for decoupling a submodule's existence from the url
that is stored in .git/config.

1. Worktrees can't really be used with submodules since the existence is
   checked based on the shared .git/config.  This means that two different
   worktress have to have the same initialized submodules.  By decoupling a
   submodule's existence from the url, two different work trees can be
   configured to have different submodules checked out.
2. Easily configure gorups of submodules that a user is interested in.  In a
   repository with hundreds of submodules it would be difficult to easily tell git
   which modules to worry about without having to individually init all of
   them.  Instead, a pathspec can be used to more easily select or deselect
   groups of submodules.

This patch series works to do this decoupling and instead allows a user to
configure submodule.active with a pathspec to use to check if a submodule is
initialized.

Brandon Williams (10):
  submodule: decouple url and submodule existence
  submodule update: add `--init-active` switch
  clone: add --submodule-spec=<pathspec> switch
  completion: clone can initialize specific submodules
  submodule--helper: add is_active command
  submodule add: respect submodule.active
  submodule status: use submodule--helper is-active
  submodule deinit: use most reliable url
  submodule sync: use submodule--helper is-active
  submodule--helper clone: check for configured submodules using helper

 Documentation/git-clone.txt            |  23 ++++---
 Documentation/git-submodule.txt        |  11 +++-
 builtin/clone.c                        |  36 ++++++++++-
 builtin/submodule--helper.c            |  22 ++++---
 contrib/completion/git-completion.bash |   1 +
 git-submodule.sh                       |  39 +++++++++---
 submodule.c                            |  25 +++++++-
 t/t7400-submodule-basic.sh             | 109 +++++++++++++++++++++++++++++++++
 t/t7413-submodule-is-active.sh         |  85 +++++++++++++++++++++++++
 9 files changed, 323 insertions(+), 28 deletions(-)
 create mode 100755 t/t7413-submodule-is-active.sh

-- 
2.11.0.483.g087da7b7c-goog

[PATCH 01/10] submodule: decouple url and submodule existence

[Brandon Williams]

Currently the submodule.<name>.url config option is used to determine
if a given submodule exists and is interesting to the user.  This
however doesn't work very well because the URL is a config option for
the scope of a repository, whereas the existence of a submodule is an
option scoped to the working tree.

In a future with worktree support for submodules, there will be multiple
working trees, each of which may only need a subset of the submodules
checked out.  The URL (which is where the submodule repository can be
obtained) should not differ between different working trees.

It may also be convenient for users to more easily specify groups of
submodules they are interested in as apposed to running "git submodule
init <path>" on each submodule they want checked out in their working
tree.

To this end, the config option submodule.active is introduced which
holds a pathspec that specifies which submodules should exist in the
working tree.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 submodule.c | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)

diff --git a/submodule.c b/submodule.c
index 4c4f033e8..991066ddf 100644
--- a/submodule.c
+++ b/submodule.c
@@ -217,11 +217,16 @@ void gitmodules_config_sha1(const unsigned char *commit_sha1)
 int is_submodule_initialized(const char *path)
 {
 	int ret = 0;
-	const struct submodule *module = NULL;
+	const struct string_list *sl;
+	const struct submodule *module = submodule_from_path(null_sha1, path);
 
-	module = submodule_from_path(null_sha1, path);
+	/* early return if there isn't a path->module mapping */
+	if (!module)
+		return 0;
+
+	sl = git_config_get_value_multi("submodule.active");
 
-	if (module) {
+	if (!sl) {
 		char *key = xstrfmt("submodule.%s.url", module->name);
 		char *value = NULL;
 
@@ -229,6 +234,20 @@ int is_submodule_initialized(const char *path)
 
 		free(value);
 		free(key);
+	} else {
+		struct pathspec ps;
+		struct argv_array args = ARGV_ARRAY_INIT;
+		const struct string_list_item *item;
+
+		for_each_string_list_item(item, sl) {
+			argv_array_push(&args, item->string);
+		}
+
+		parse_pathspec(&ps, 0, 0, 0, args.argv);
+		ret = match_pathspec(&ps, path, strlen(path), 0, NULL, 1);
+
+		argv_array_clear(&args);
+		clear_pathspec(&ps);
 	}
 
 	return ret;
-- 
2.11.0.483.g087da7b7c-goog

Re: [PATCH 01/10] submodule: decouple url and submodule existence

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> Currently the submodule.<name>.url config option is used to determine
> if a given submodule exists and is interesting to the user.  This
> however doesn't work very well because the URL is a config option for
> the scope of a repository, whereas the existence of a submodule is an
> option scoped to the working tree.
>
> In a future with worktree support for submodules, there will be multiple
> working trees, each of which may only need a subset of the submodules
> checked out.  The URL (which is where the submodule repository can be
> obtained) should not differ between different working trees.
>
> It may also be convenient for users to more easily specify groups of
> submodules they are interested in as apposed to running "git submodule
> init <path>" on each submodule they want checked out in their working
> tree.
>
> To this end, the config option submodule.active is introduced which
> holds a pathspec that specifies which submodules should exist in the
> working tree.

Hmph.  submodule.active in .git/config would be shared the same way
submodule.<name>.url in .git/config is shared across the worktrees
that stems from the same primary repository, no?

Perhaps there are some other uses of this submodule.active idea, but
I do not see how it is relevant to solving "multiple worktrees"
issue.  Per-worktree config would solve it with the current
submodule.<name>.url without submodule.active list, I would think [*1*].

Also as a grouping measure, submodule.active that lists submodule
paths feels hard to use.  When switching between two branches in the
superproject that have the same submodule bound at two different
paths, who is responsible for updating submodule.active in
superproject's config?  If it were a list of submodule names, this
objection does not apply, though.



[Footnote]

*1* At the conceptual level, I agree that .url that also means "we
    are interested in this one" feels like somewhat an unclean
    design, but that is not what you are "fixing", is it?

Re: [PATCH 01/10] submodule: decouple url and submodule existence

[Brandon Williams]

On 02/24, Junio C Hamano wrote:
> Brandon Williams <bmwill@google.com> writes:
> 
> > Currently the submodule.<name>.url config option is used to determine
> > if a given submodule exists and is interesting to the user.  This
> > however doesn't work very well because the URL is a config option for
> > the scope of a repository, whereas the existence of a submodule is an
> > option scoped to the working tree.
> >
> > In a future with worktree support for submodules, there will be multiple
> > working trees, each of which may only need a subset of the submodules
> > checked out.  The URL (which is where the submodule repository can be
> > obtained) should not differ between different working trees.
> >
> > It may also be convenient for users to more easily specify groups of
> > submodules they are interested in as apposed to running "git submodule
> > init <path>" on each submodule they want checked out in their working
> > tree.
> >
> > To this end, the config option submodule.active is introduced which
> > holds a pathspec that specifies which submodules should exist in the
> > working tree.
> 
> Hmph.  submodule.active in .git/config would be shared the same way
> submodule.<name>.url in .git/config is shared across the worktrees
> that stems from the same primary repository, no?
> 
> Perhaps there are some other uses of this submodule.active idea, but
> I do not see how it is relevant to solving "multiple worktrees"
> issue.  Per-worktree config would solve it with the current
> submodule.<name>.url without submodule.active list, I would think [*1*].

Correct, I should update the language to indicate this allows the URL to
be shared between worktrees, but a per-worktree config must exist before
submodule.active can actually be used to select different groups of
submodules per-worktree.  The idea is that if submodule.active is set
then you no longer look at the URLs to see what is interesting but
rather at the paths.

> Also as a grouping measure, submodule.active that lists submodule
> paths feels hard to use.  When switching between two branches in the
> superproject that have the same submodule bound at two different
> paths, who is responsible for updating submodule.active in
> superproject's config?  If it were a list of submodule names, this
> objection does not apply, though.

I agree that if you are listing every submodule path by hand then this
may not be the best approach and would be difficult to use.  The idea is
that this would allow a user to set a general pathspec to identify a
group of modules they are interested in.  Perhaps once attributes can be
used in pathspecs a user could group submodules by setting a particular
attribute and then submodule.active would have a value like
":(attr:foo)" to indicate I'm interested in all submodules with the
"foo" attribute.

> 
> 
> 
> [Footnote]
> 
> *1* At the conceptual level, I agree that .url that also means "we
>     are interested in this one" feels like somewhat an unclean
>     design, but that is not what you are "fixing", is it?
> 

-- 
Brandon Williams

Re: [PATCH 01/10] submodule: decouple url and submodule existence

[Stefan Beller]

IIRC most of the series is actually refactoring, i.e.
providing a central function that answers
"is this submodule active/initialized?" and then making use of this
function.

Maybe it would be easier to reroll these refactorings first without adding
in the change of behavior.

Off the list we talked about different models how to indicate that
a submodule is active.

Current situation:
submodule.<name>.URL is used as a boolean flag

Possible future A:
1) If submodule.active exists use that
2) otherwise go be individual .URL configs

submodule.active is a config that contains a pathspec,
i.e. specifies the group of submodules instead of marking
each submodule individually.

How to get to future A:
* apply this patch series

Possible future B:
1) the boolean submodule.<name>.exists is used
    if existent
2) If that boolean is missing we'd respect a grouping
   feature such as submodule.active
3) fallback to the .URL as a boolean indicator.

How to get to future B:
1) possibly take the refactoring part from this series
2) implement the .exists flag (that can solve the worktree
  problem, as then we don't have to abuse the .URL
  as a boolean indicator any more.)
3) implement the grouping feature that takes precedence
  over .URL, but yields precedence to the .exists flag.
  (This would solve the grouping problem)

By having two different series (2) and (3) we'd solve
just one thing at a time with each series, such that
this seems easier to understand and review.

The question is if this future B is also easier to use for
users and I think it would be, despite being technically more
complex.

Most users only have a few submodules. Also the assumption
is to have either interest in very few specific submodules
or all of them. For both of these cases the .exists is a good idea
as it is very explicit, but verbose.

The grouping feature would help in the case of lots of
submodules, e.g. to select all of them you would just give
an easy pathspec "." and that would help going forward
as by such a submodule spec all new submodules would
be "auto-on".

Possible future C:
Similar to B, but with branch specific configuration.
submodule.<branchname>.active would work as a
grouping feature. I wonder how it would work with
the .exists flag.

Stefan

Re: [PATCH 01/10] submodule: decouple url and submodule existence

[Brandon Williams]

On 03/01, Stefan Beller wrote:

Sorry I've been slow at rerolling this.  I'll send out a reroll today.

> IIRC most of the series is actually refactoring, i.e.
> providing a central function that answers
> "is this submodule active/initialized?" and then making use of this
> function.
> 
> Maybe it would be easier to reroll these refactorings first without adding
> in the change of behavior.

I agree, when I resend out the series I can drop the first patch so that
the series as a whole just moves to using a standardized function to
check that a submodule is active.  I can then resend out the first patch
in the series on its own so that we can more easily discuss the best
route to take.

> 
> Off the list we talked about different models how to indicate that
> a submodule is active.
> 
> Current situation:
> submodule.<name>.URL is used as a boolean flag
> 
> Possible future A:
> 1) If submodule.active exists use that
> 2) otherwise go be individual .URL configs
> 
> submodule.active is a config that contains a pathspec,
> i.e. specifies the group of submodules instead of marking
> each submodule individually.
> 
> How to get to future A:
> * apply this patch series
> 
> Possible future B:
> 1) the boolean submodule.<name>.exists is used
>     if existent
> 2) If that boolean is missing we'd respect a grouping
>    feature such as submodule.active
> 3) fallback to the .URL as a boolean indicator.

Thinking on this, and to maintain simplicity and uniformity, we could
have the per-submodule boolean flag to be "submodule.<name>.active"
while the general pathspec based config option would be
"submodule.active".

> 
> How to get to future B:
> 1) possibly take the refactoring part from this series
> 2) implement the .exists flag (that can solve the worktree
>   problem, as then we don't have to abuse the .URL
>   as a boolean indicator any more.)
> 3) implement the grouping feature that takes precedence
>   over .URL, but yields precedence to the .exists flag.
>   (This would solve the grouping problem)
> 
> By having two different series (2) and (3) we'd solve
> just one thing at a time with each series, such that
> this seems easier to understand and review.
> 
> The question is if this future B is also easier to use for
> users and I think it would be, despite being technically more
> complex.
> 
> Most users only have a few submodules. Also the assumption
> is to have either interest in very few specific submodules
> or all of them. For both of these cases the .exists is a good idea
> as it is very explicit, but verbose.
> 
> The grouping feature would help in the case of lots of
> submodules, e.g. to select all of them you would just give
> an easy pathspec "." and that would help going forward
> as by such a submodule spec all new submodules would
> be "auto-on".

And if you wanted all but one you could have submodule.active="." while
submodule.dontwant.active=false instead of having a multi value
submodule.active with an exclusion pathspec.  Both work but one may be
easier for a user to understand.

> 
> Possible future C:
> Similar to B, but with branch specific configuration.
> submodule.<branchname>.active would work as a
> grouping feature. I wonder how it would work with
> the .exists flag.
> 
> Stefan

-- 
Brandon Williams

Re: [PATCH 01/10] submodule: decouple url and submodule existence

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> On 02/24, Junio C Hamano wrote:
>
>> Also as a grouping measure, submodule.active that lists submodule
>> paths feels hard to use.  When switching between two branches in the
>> superproject that have the same submodule bound at two different
>> paths, who is responsible for updating submodule.active in
>> superproject's config?  If it were a list of submodule names, this
>> objection does not apply, though.
>
> I agree that if you are listing every submodule path by hand then this
> may not be the best approach and would be difficult to use.  The idea is
> that this would allow a user to set a general pathspec to identify a
> group of modules they are interested in.  Perhaps once attributes can be
> used in pathspecs a user could group submodules by setting a particular
> attribute and then submodule.active would have a value like
> ":(attr:foo)" to indicate I'm interested in all submodules with the
> "foo" attribute.

OK.  As .gitattributes is tracked just like .gitmodules in tree, the
project can adjust the path pattern that matches a renamed submodule
and gives it an attribute in .gitattributes in the same commit in
the superproject that moves the directory to which the submodule is
bound, just like it updates .gitmodules to adjust the name<->path
mapping.  So that plan solves my specific worry about using "path"
for grouping and not "name".

Thanks.

[PATCH 02/10] submodule update: add `--init-active` switch

[Brandon Williams]

The new switch `--init-active` initializes the submodules which are
configured in `submodule.active` instead of those given as
command line arguments before updating. In the first implementation this
is made incompatible with further command line arguments as it is
unclear what the user means by

    git submodule update --init --init-active <paths>

This new switch allows users to record more complex patterns as it saves
retyping them whenever you invoke update.

Based on a patch by Stefan Beller <sbeller@google.com>

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/git-submodule.txt | 11 ++++++++++-
 git-submodule.sh                | 21 ++++++++++++++++++---
 t/t7400-submodule-basic.sh      | 39 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 67 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 918bd1d1b..626b9760a 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -14,7 +14,7 @@ SYNOPSIS
 'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]
 'git submodule' [--quiet] init [--] [<path>...]
 'git submodule' [--quiet] deinit [-f|--force] (--all|[--] <path>...)
-'git submodule' [--quiet] update [--init] [--remote] [-N|--no-fetch]
+'git submodule' [--quiet] update [--init[-active]] [--remote] [-N|--no-fetch]
 	      [--[no-]recommend-shallow] [-f|--force] [--rebase|--merge]
 	      [--reference <repository>] [--depth <depth>] [--recursive]
 	      [--jobs <n>] [--] [<path>...]
@@ -195,6 +195,10 @@ If the submodule is not yet initialized, and you just want to use the
 setting as stored in .gitmodules, you can automatically initialize the
 submodule with the `--init` option.
 
+You can configure a set of submodules using pathspec syntax in
+submodule.active you can use `--init-active` to initialize
+those before updating.
+
 If `--recursive` is specified, this command will recurse into the
 registered submodules, and update any nested submodules within.
 --
@@ -378,6 +382,11 @@ the submodule itself.
 	Initialize all submodules for which "git submodule init" has not been
 	called so far before updating.
 
+--init-active::
+	This option is only valid for the update command.
+	Initialize all submodules configured in "`submodule.active`"
+	that have not been updated before.
+
 --name::
 	This option is only valid for the add command. It sets the submodule's
 	name to the given string instead of defaulting to its path. The name
diff --git a/git-submodule.sh b/git-submodule.sh
index 554bd1c49..e5b9a8920 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -9,7 +9,7 @@ USAGE="[--quiet] add [-b <branch>] [-f|--force] [--name <name>] [--reference <re
    or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]
    or: $dashless [--quiet] init [--] [<path>...]
    or: $dashless [--quiet] deinit [-f|--force] (--all| [--] <path>...)
-   or: $dashless [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference <repository>] [--recursive] [--] [<path>...]
+   or: $dashless [--quiet] update [--init[-active]] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference <repository>] [--recursive] [--] [<path>...]
    or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
    or: $dashless [--quiet] foreach [--recursive] <command>
    or: $dashless [--quiet] sync [--recursive] [--] [<path>...]"
@@ -499,7 +499,12 @@ cmd_update()
 			progress="--progress"
 			;;
 		-i|--init)
-			init=1
+			test -z $init || test $init = by_args || die "$(gettext "Only one of --init or --init-active may be used.")"
+			init=by_args
+			;;
+		--init-active)
+			test -z $init || test $init = by_config || die "$(gettext "Only one of --init or --init-active may be used.")"
+			init=by_config
 			;;
 		--remote)
 			remote=1
@@ -568,7 +573,17 @@ cmd_update()
 
 	if test -n "$init"
 	then
-		cmd_init "--" "$@" || return
+		if test "$init" = "by_config"
+		then
+			if test $# -gt 0
+			then
+				die "$(gettext "path arguments are incompatible with --init-active")"
+			fi
+			cmd_init "--" $(git config --get-all submodule.active) || return
+		else
+			cmd_init "--" "$@" || return
+		fi
+
 	fi
 
 	{
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index b77cce8e4..f5df0a5b4 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1116,5 +1116,44 @@ test_expect_success 'submodule helper list is not confused by common prefixes' '
 	test_cmp expect actual
 '
 
+test_expect_success 'setup superproject with submodules' '
+	git init sub1 &&
+	test_commit -C sub1 test &&
+	test_commit -C sub1 test2 &&
+	git init multisuper &&
+	git -C multisuper submodule add ../sub1 sub0 &&
+	git -C multisuper submodule add ../sub1 sub1 &&
+	git -C multisuper submodule add ../sub1 sub2 &&
+	git -C multisuper submodule add ../sub1 sub3 &&
+	git -C multisuper commit -m "add some submodules"
+'
+
+cat >expect <<-EOF
+-sub0
+ sub1 (test2)
+ sub2 (test2)
+ sub3 (test2)
+EOF
+
+test_expect_success 'submodule update --init with a specification' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone submodule update --init . ":(exclude)sub0" &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'submodule update --init-active' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone config submodule.active "." &&
+	git -C multisuper_clone config --add submodule.active ":(exclude)sub0" &&
+	git -C multisuper_clone submodule update --init-active &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_must_fail git -C multisuper_clone submodule update --init-active sub0 &&
+	test_cmp expect actual
+'
 
 test_done
-- 
2.11.0.483.g087da7b7c-goog

[PATCH 03/10] clone: add --submodule-spec=<pathspec> switch

[Brandon Williams]

The new switch passes the pathspec to `git submodule update
--init-active` which is called after the actual clone is done.

Additionally this configures the submodule.active option to
be the given pathspec, such that any future invocation of
`git submodule update --init-active` will keep up
with the pathspec.

Based on a patch by Stefan Beller <sbeller@google.com>

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/git-clone.txt | 23 ++++++++++-----
 builtin/clone.c             | 36 +++++++++++++++++++++--
 t/t7400-submodule-basic.sh  | 70 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 120 insertions(+), 9 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 35cc34b2f..9692eab30 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -15,7 +15,8 @@ SYNOPSIS
 	  [--dissociate] [--separate-git-dir <git dir>]
 	  [--depth <depth>] [--[no-]single-branch]
 	  [--recursive | --recurse-submodules] [--[no-]shallow-submodules]
-	  [--jobs <n>] [--] <repository> [<directory>]
+	  [--submodule-spec <pathspec>] [--jobs <n>] [--]
+	  <repository> [<directory>]
 
 DESCRIPTION
 -----------
@@ -217,12 +218,20 @@ objects from the source repository into a pack in the cloned repository.
 
 --recursive::
 --recurse-submodules::
-	After the clone is created, initialize all submodules within,
-	using their default settings. This is equivalent to running
-	`git submodule update --init --recursive` immediately after
-	the clone is finished. This option is ignored if the cloned
-	repository does not have a worktree/checkout (i.e. if any of
-	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
+	After the clone is created, initialize and clone all submodules
+	within, using their default settings. This is equivalent to
+	running `git submodule update --recursive --init` immediately
+	after the clone is finished. This option is ignored if the
+	cloned repository does not have a worktree/checkout (i.e.  if
+	any of `--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
+
+--submodule-spec::
+	After the clone is created, initialize and clone specified
+	submodules within, using their default settings. It is possible
+	to give multiple specifications by giving this argument multiple
+	times. This is equivalent to configuring `submodule.active`
+	and then running `git submodule update --init-active`
+	immediately after the clone is finished.
 
 --[no-]shallow-submodules::
 	All submodules which are cloned will be shallow with a depth of 1.
diff --git a/builtin/clone.c b/builtin/clone.c
index 5ef81927a..15a8d3ef0 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -56,6 +56,16 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;
 static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;
 static int option_dissociate;
 static int max_jobs = -1;
+static struct string_list submodule_spec;
+
+static int submodule_spec_cb(const struct option *opt, const char *arg, int unset)
+{
+	if (unset)
+		return -1;
+
+	string_list_append((struct string_list *)opt->value, arg);
+	return 0;
+}
 
 static struct option builtin_clone_options[] = {
 	OPT__VERBOSITY(&option_verbosity),
@@ -112,6 +122,9 @@ static struct option builtin_clone_options[] = {
 			TRANSPORT_FAMILY_IPV4),
 	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
 			TRANSPORT_FAMILY_IPV6),
+	OPT_CALLBACK(0, "submodule-spec", &submodule_spec, N_("<pathspec>"),
+			N_("clone specific submodules. Pass multiple times for complex pathspecs"),
+			submodule_spec_cb),
 	OPT_END()
 };
 
@@ -733,13 +746,21 @@ static int checkout(int submodule_progress)
 	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
 			   sha1_to_hex(sha1), "1", NULL);
 
-	if (!err && option_recursive) {
+	if (!err && (option_recursive || submodule_spec.nr > 0)) {
 		struct argv_array args = ARGV_ARRAY_INIT;
-		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
+		argv_array_pushl(&args, "submodule", "update", NULL);
+
+		if (submodule_spec.nr > 0)
+			argv_array_pushf(&args, "--init-active");
+		else
+			argv_array_pushf(&args, "--init");
 
 		if (option_shallow_submodules == 1)
 			argv_array_push(&args, "--depth=1");
 
+		if (option_recursive)
+			argv_array_pushf(&args, "--recursive");
+
 		if (max_jobs != -1)
 			argv_array_pushf(&args, "--jobs=%d", max_jobs);
 
@@ -887,6 +908,17 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		option_no_checkout = 1;
 	}
 
+	if (submodule_spec.nr > 0) {
+		struct string_list_item *item;
+		struct strbuf sb = STRBUF_INIT;
+		for_each_string_list_item(item, &submodule_spec) {
+			strbuf_addf(&sb, "submodule.active=%s",
+				    item->string);
+			string_list_append(&option_config,
+					   strbuf_detach(&sb, NULL));
+		}
+	}
+
 	if (!option_origin)
 		option_origin = "origin";
 
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index f5df0a5b4..39364ef68 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1156,4 +1156,74 @@ test_expect_success 'submodule update --init-active' '
 	test_cmp expect actual
 '
 
+test_expect_success 'clone --submodule-spec works' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat >expected <<-\EOF &&
+	 sub0 (test2)
+	-sub1
+	-sub2
+	-sub3
+	EOF
+
+	git clone --recurse-submodules --submodule-spec="sub0" multisuper multisuper_clone &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp actual expected
+'
+
+test_expect_success 'clone with multiple --submodule-spec options' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat >expect <<-\EOF &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	EOF
+
+	git clone --recurse-submodules \
+		  --submodule-spec="." \
+		  --submodule-spec ":(exclude)sub0" \
+		  --submodule-spec ":(exclude)sub2" \
+		  multisuper multisuper_clone &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'clone and subsequent updates correctly auto-initialize submodules' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat <<-\EOF >expect &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	EOF
+
+	cat <<-\EOF >expect2 &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	-sub4
+	 sub5 (test2)
+	EOF
+
+	git clone --recurse-submodules \
+		  --submodule-spec="." \
+		  --submodule-spec ":(exclude)sub0" \
+		  --submodule-spec ":(exclude)sub2" \
+		  --submodule-spec ":(exclude)sub4" \
+		  multisuper multisuper_clone &&
+
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect actual &&
+
+	git -C multisuper submodule add ../sub1 sub4 &&
+	git -C multisuper submodule add ../sub1 sub5 &&
+	git -C multisuper commit -m "add more submodules" &&
+	# obtain the new superproject
+	git -C multisuper_clone pull &&
+	git -C multisuper_clone submodule update --init-active &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect2 actual
+'
+
 test_done
-- 
2.11.0.483.g087da7b7c-goog

[PATCH 04/10] completion: clone can initialize specific submodules

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 contrib/completion/git-completion.bash | 1 +
 1 file changed, 1 insertion(+)

diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index 6721ff80f..4e473aa90 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -1138,6 +1138,7 @@ _git_clone ()
 			--single-branch
 			--branch
 			--recurse-submodules
+			--submodule-spec
 			"
 		return
 		;;
-- 
2.11.0.483.g087da7b7c-goog

[PATCH 05/10] submodule--helper: add is_active command

[Brandon Williams]

There are a lot of places where an explicit check for
submodule."<name>".url is done to see if a submodule exists.  In order
to more easily facilitate the use of the submodule.active config option
to indicate active submodules, add a helper which can be used to query
if a submodule is active or not.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c    | 11 ++++++++
 t/t7413-submodule-is-active.sh | 63 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 74 insertions(+)
 create mode 100755 t/t7413-submodule-is-active.sh

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index df0d9c166..dac02604d 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -1128,6 +1128,16 @@ static int absorb_git_dirs(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+static int is_active(int argc, const char **argv, const char *prefix)
+{
+	if (argc != 2)
+		die("submodule--helper is-active takes exactly 1 arguments");
+
+	gitmodules_config();
+
+	return !is_submodule_initialized(argv[1]);
+}
+
 #define SUPPORT_SUPER_PREFIX (1<<0)
 
 struct cmd_struct {
@@ -1147,6 +1157,7 @@ static struct cmd_struct commands[] = {
 	{"init", module_init, 0},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
+	{"is-active", is_active, 0},
 };
 
 int cmd_submodule__helper(int argc, const char **argv, const char *prefix)
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
new file mode 100755
index 000000000..683487020
--- /dev/null
+++ b/t/t7413-submodule-is-active.sh
@@ -0,0 +1,63 @@
+#!/bin/sh
+
+test_description='Test submodule--helper is-active
+
+This test verifies that `git submodue--helper is-active` correclty identifies
+submodules which are "active" and interesting to the user.
+'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	git init sub &&
+	test_commit -C sub initial &&
+	git init super &&
+	test_commit -C super initial &&
+	git -C super submodule add ../sub sub1 &&
+	git -C super submodule add ../sub sub2 &&
+	git -C super commit -a -m "add 2 submodules at sub{1,2}"
+'
+
+test_expect_success 'is-active works with urls' '
+	git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config --unset submodule.sub1.URL &&
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super config submodule.sub1.URL ../sub &&
+	git -C super submodule--helper is-active sub1
+'
+
+test_expect_success 'is-active works with basic submodule.active config' '
+	git -C super config --add submodule.active "." &&
+	git -C super config --unset submodule.sub1.URL &&
+	git -C super config --unset submodule.sub2.URL &&
+
+	git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config submodule.sub1.URL ../sub &&
+	git -C super config submodule.sub2.URL ../sub &&
+	git -C super config --unset-all submodule.active
+'
+
+test_expect_success 'is-active correctly works with paths that are not submodules' '
+	test_must_fail git -C super submodule--helper is-active not-a-submodule &&
+
+	git -C super config --add submodule.active "." &&
+	test_must_fail git -C super submodule--helper is-active not-a-submodule &&
+
+	git -C super config --unset-all submodule.active
+'
+
+test_expect_success 'is-active works with exclusions in submodule.active config' '
+	git -C super config --add submodule.active "." &&
+	git -C super config --add submodule.active ":(exclude)sub1" &&
+
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config --unset-all submodule.active
+'
+
+test_done
-- 
2.11.0.483.g087da7b7c-goog

Re: [PATCH 05/10] submodule--helper: add is_active command

[Stefan Beller]

On Thu, Feb 23, 2017 at 3:47 PM, Brandon Williams <bmwill@google.com> wrote:
> There are a lot of places where an explicit check for
> submodule."<name>".url is done to see if a submodule exists.  In order
> to more easily facilitate the use of the submodule.active config option
> to indicate active submodules, add a helper which can be used to query
> if a submodule is active or not.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  builtin/submodule--helper.c    | 11 ++++++++
>  t/t7413-submodule-is-active.sh | 63 ++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 74 insertions(+)
>  create mode 100755 t/t7413-submodule-is-active.sh
>
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index df0d9c166..dac02604d 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -1128,6 +1128,16 @@ static int absorb_git_dirs(int argc, const char **argv, const char *prefix)
>         return 0;
>  }
>
> +static int is_active(int argc, const char **argv, const char *prefix)
> +{
> +       if (argc != 2)
> +               die("submodule--helper is-active takes exactly 1 arguments");
> +
> +       gitmodules_config();
> +
> +       return !is_submodule_initialized(argv[1]);
> +}
> +
>  #define SUPPORT_SUPER_PREFIX (1<<0)
>
>  struct cmd_struct {
> @@ -1147,6 +1157,7 @@ static struct cmd_struct commands[] = {
>         {"init", module_init, 0},
>         {"remote-branch", resolve_remote_submodule_branch, 0},
>         {"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
> +       {"is-active", is_active, 0},
>  };
>
>  int cmd_submodule__helper(int argc, const char **argv, const char *prefix)
> diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
> new file mode 100755
> index 000000000..683487020
> --- /dev/null
> +++ b/t/t7413-submodule-is-active.sh
> @@ -0,0 +1,63 @@
> +#!/bin/sh
> +
> +test_description='Test submodule--helper is-active
> +
> +This test verifies that `git submodue--helper is-active` correclty identifies
> +submodules which are "active" and interesting to the user.
> +'
> +
> +. ./test-lib.sh
> +
> +test_expect_success 'setup' '
> +       git init sub &&
> +       test_commit -C sub initial &&
> +       git init super &&
> +       test_commit -C super initial &&
> +       git -C super submodule add ../sub sub1 &&
> +       git -C super submodule add ../sub sub2 &&
> +       git -C super commit -a -m "add 2 submodules at sub{1,2}"
> +'
> +
> +test_expect_success 'is-active works with urls' '
> +       git -C super submodule--helper is-active sub1 &&
> +       git -C super submodule--helper is-active sub2 &&
> +
> +       git -C super config --unset submodule.sub1.URL &&
> +       test_must_fail git -C super submodule--helper is-active sub1 &&
> +       git -C super config submodule.sub1.URL ../sub &&
> +       git -C super submodule--helper is-active sub1
> +'
> +
> +test_expect_success 'is-active works with basic submodule.active config' '
> +       git -C super config --add submodule.active "." &&
> +       git -C super config --unset submodule.sub1.URL &&
> +       git -C super config --unset submodule.sub2.URL &&

I think we'd want to unset only one of them here

> +
> +       git -C super submodule--helper is-active sub1 &&
> +       git -C super submodule--helper is-active sub2 &&

to test 2 different cases of one being active by config setting only and
the other having both.

I could not spot test for having the URL set but the config setting set, not
including the submodule, e.g.

    git -C super config  submodule.sub1.URL ../sub &&
    git -C super submodule.active  ":(exclude)sub1" &&

which would be expected to not be active, as once the configuration
is there it takes precedence over any (no)URL setting?

Re: [PATCH 05/10] submodule--helper: add is_active command

[Brandon Williams]

On 02/23, Stefan Beller wrote:
> On Thu, Feb 23, 2017 at 3:47 PM, Brandon Williams <bmwill@google.com> wrote:
> > There are a lot of places where an explicit check for
> > submodule."<name>".url is done to see if a submodule exists.  In order
> > to more easily facilitate the use of the submodule.active config option
> > to indicate active submodules, add a helper which can be used to query
> > if a submodule is active or not.
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> >  builtin/submodule--helper.c    | 11 ++++++++
> >  t/t7413-submodule-is-active.sh | 63 ++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 74 insertions(+)
> >  create mode 100755 t/t7413-submodule-is-active.sh
> >
> > diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> > index df0d9c166..dac02604d 100644
> > --- a/builtin/submodule--helper.c
> > +++ b/builtin/submodule--helper.c
> > @@ -1128,6 +1128,16 @@ static int absorb_git_dirs(int argc, const char **argv, const char *prefix)
> >         return 0;
> >  }
> >
> > +static int is_active(int argc, const char **argv, const char *prefix)
> > +{
> > +       if (argc != 2)
> > +               die("submodule--helper is-active takes exactly 1 arguments");
> > +
> > +       gitmodules_config();
> > +
> > +       return !is_submodule_initialized(argv[1]);
> > +}
> > +
> >  #define SUPPORT_SUPER_PREFIX (1<<0)
> >
> >  struct cmd_struct {
> > @@ -1147,6 +1157,7 @@ static struct cmd_struct commands[] = {
> >         {"init", module_init, 0},
> >         {"remote-branch", resolve_remote_submodule_branch, 0},
> >         {"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
> > +       {"is-active", is_active, 0},
> >  };
> >
> >  int cmd_submodule__helper(int argc, const char **argv, const char *prefix)
> > diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
> > new file mode 100755
> > index 000000000..683487020
> > --- /dev/null
> > +++ b/t/t7413-submodule-is-active.sh
> > @@ -0,0 +1,63 @@
> > +#!/bin/sh
> > +
> > +test_description='Test submodule--helper is-active
> > +
> > +This test verifies that `git submodue--helper is-active` correclty identifies
> > +submodules which are "active" and interesting to the user.
> > +'
> > +
> > +. ./test-lib.sh
> > +
> > +test_expect_success 'setup' '
> > +       git init sub &&
> > +       test_commit -C sub initial &&
> > +       git init super &&
> > +       test_commit -C super initial &&
> > +       git -C super submodule add ../sub sub1 &&
> > +       git -C super submodule add ../sub sub2 &&
> > +       git -C super commit -a -m "add 2 submodules at sub{1,2}"
> > +'
> > +
> > +test_expect_success 'is-active works with urls' '
> > +       git -C super submodule--helper is-active sub1 &&
> > +       git -C super submodule--helper is-active sub2 &&
> > +
> > +       git -C super config --unset submodule.sub1.URL &&
> > +       test_must_fail git -C super submodule--helper is-active sub1 &&
> > +       git -C super config submodule.sub1.URL ../sub &&
> > +       git -C super submodule--helper is-active sub1
> > +'
> > +
> > +test_expect_success 'is-active works with basic submodule.active config' '
> > +       git -C super config --add submodule.active "." &&
> > +       git -C super config --unset submodule.sub1.URL &&
> > +       git -C super config --unset submodule.sub2.URL &&
> 
> I think we'd want to unset only one of them here
> 
> > +
> > +       git -C super submodule--helper is-active sub1 &&
> > +       git -C super submodule--helper is-active sub2 &&
> 
> to test 2 different cases of one being active by config setting only and
> the other having both.

Will do.

> 
> I could not spot test for having the URL set but the config setting set, not
> including the submodule, e.g.
> 
>     git -C super config  submodule.sub1.URL ../sub &&
>     git -C super submodule.active  ":(exclude)sub1" &&
> 
> which would be expected to not be active, as once the configuration
> is there it takes precedence over any (no)URL setting?

The last test, tests this functionality as the URL settings for both
sub1 and sub2 are in the config.  You'll notice in the tests where I
unset the URL config, that they get added back at the end of the test so
that future tests have a clean state to work with.

-- 
Brandon Williams

[PATCH 06/10] submodule add: respect submodule.active

[Brandon Williams]

When submodule.active is configured, in addition to adding
submodule."<name>".url to the config, add the path of the added
submodule if it isn't already covered by the current config values.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh               | 11 +++++++++++
 t/t7413-submodule-is-active.sh | 22 ++++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/git-submodule.sh b/git-submodule.sh
index e5b9a8920..4633a4336 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -271,6 +271,17 @@ or you are unsure what this means choose another name with the '--name' option."
 	fi &&
 	git add --force .gitmodules ||
 	die "$(eval_gettext "Failed to register submodule '\$sm_path'")"
+
+	if git config --get submodule.active >/dev/null
+	then
+		# If the submodule being adding isn't already covered by the
+		# current configured pathspec, add the submodule's path to
+		# 'submodule.active'
+		if ! git submodule--helper is-active "$sm_path"
+		then
+			git config --add submodule.active "$sm_path"
+		fi
+	fi
 }
 
 #
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
index 683487020..8a519163b 100755
--- a/t/t7413-submodule-is-active.sh
+++ b/t/t7413-submodule-is-active.sh
@@ -60,4 +60,26 @@ test_expect_success 'is-active works with exclusions in submodule.active config'
 	git -C super config --unset-all submodule.active
 '
 
+test_expect_success 'is-active and submodule add' '
+	test_when_finished "rm -rf super2" &&
+	git init super2 &&
+	test_commit -C super2 initial &&
+	git -C super2 config --add submodule.active "sub*" &&
+
+	cat >expect <<-\EOF &&
+	sub*
+	EOF
+	git -C super2 submodule add ../sub sub1 &&
+	git -C super2 config --get-all submodule.active >actual &&
+	test_cmp actual expect &&
+
+	cat >expect <<-\EOF &&
+	sub*
+	mod
+	EOF
+	git -C super2 submodule add ../sub mod &&
+	git -C super2 config --get-all submodule.active >actual &&
+	test_cmp actual expect
+'
+
 test_done
-- 
2.11.0.483.g087da7b7c-goog

[PATCH 07/10] submodule status: use submodule--helper is-active

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 4633a4336..f8adfb179 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1026,14 +1026,13 @@ cmd_status()
 	do
 		die_if_unmatched "$mode" "$sha1"
 		name=$(git submodule--helper name "$sm_path") || exit
-		url=$(git config submodule."$name".url)
 		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
 		if test "$stage" = U
 		then
 			say "U$sha1 $displaypath"
 			continue
 		fi
-		if test -z "$url" ||
+		if ! git submodule--helper is-active "$sm_path" ||
 		{
 			! test -d "$sm_path"/.git &&
 			! test -f "$sm_path"/.git
-- 
2.11.0.483.g087da7b7c-goog

[PATCH 08/10] submodule deinit: use most reliable url

[Brandon Williams]

The user could have configured the submodule to have a different URL
from the one in the superproject's config.  To account for this read
what the submodule has configured for remote.origin.url and use that
instead.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index f8adfb179..02b85dceb 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -466,7 +466,7 @@ Submodule work tree '\$displaypath' contains a .git directory
 		then
 			# Remove the whole section so we have a clean state when
 			# the user later decides to init this submodule again
-			url=$(git config submodule."$name".url)
+			url=$(GIT_DIR="$(git rev-parse --git-path modules/$name)" git config remote.origin.url)
 			git config --remove-section submodule."$name" 2>/dev/null &&
 			say "$(eval_gettext "Submodule '\$name' (\$url) unregistered for path '\$displaypath'")"
 		fi
-- 
2.11.0.483.g087da7b7c-goog

[PATCH 09/10] submodule sync: use submodule--helper is-active

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 02b85dceb..f35345775 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1127,7 +1127,7 @@ cmd_sync()
 			;;
 		esac
 
-		if git config "submodule.$name.url" >/dev/null 2>/dev/null
+		if git submodule--helper is-active "$sm_path"
 		then
 			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
 			say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
-- 
2.11.0.483.g087da7b7c-goog

[PATCH 10/10] submodule--helper clone: check for configured submodules using helper

[Brandon Williams]

Use the 'is_submodule_initialized()' helper to check for configured
submodules instead of manually checking for the submodule's URL in the
config.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index dac02604d..68d21b600 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -760,7 +760,6 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 	struct strbuf displaypath_sb = STRBUF_INIT;
 	struct strbuf sb = STRBUF_INIT;
 	const char *displaypath = NULL;
-	char *url = NULL;
 	int needs_cloning = 0;
 
 	if (ce_stage(ce)) {
@@ -795,14 +794,11 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 	}
 
 	/*
-	 * Looking up the url in .git/config.
+	 * Check if the submodule has been initialized.
 	 * We must not fall back to .gitmodules as we only want
 	 * to process configured submodules.
 	 */
-	strbuf_reset(&sb);
-	strbuf_addf(&sb, "submodule.%s.url", sub->name);
-	git_config_get_string(sb.buf, &url);
-	if (!url) {
+	if (!is_submodule_initialized(ce->name)) {
 		next_submodule_warn_missing(suc, out, displaypath);
 		goto cleanup;
 	}
@@ -836,7 +832,7 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		argv_array_push(&child->args, "--depth=1");
 	argv_array_pushl(&child->args, "--path", sub->path, NULL);
 	argv_array_pushl(&child->args, "--name", sub->name, NULL);
-	argv_array_pushl(&child->args, "--url", url, NULL);
+	argv_array_pushl(&child->args, "--url", sub->url, NULL);
 	if (suc->references.nr) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, &suc->references)
@@ -846,7 +842,6 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		argv_array_push(&child->args, suc->depth);
 
 cleanup:
-	free(url);
 	strbuf_reset(&displaypath_sb);
 	strbuf_reset(&sb);
 
-- 
2.11.0.483.g087da7b7c-goog

Re: [PATCH 10/10] submodule--helper clone: check for configured submodules using helper

[Stefan Beller]

On Thu, Feb 23, 2017 at 3:47 PM, Brandon Williams <bmwill@google.com> wrote:

> @@ -795,14 +794,11 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
>         }
>
>         /*
> -        * Looking up the url in .git/config.
> +        * Check if the submodule has been initialized.
>          * We must not fall back to .gitmodules as we only want
>          * to process configured submodules.

This sentence "we must not ..." is also no longer accurate,
as we do exactly that when using sub->url instead of just url
below.

Re: [PATCH 10/10] submodule--helper clone: check for configured submodules using helper

[Brandon Williams]

On 02/23, Stefan Beller wrote:
> On Thu, Feb 23, 2017 at 3:47 PM, Brandon Williams <bmwill@google.com> wrote:
> 
> > @@ -795,14 +794,11 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
> >         }
> >
> >         /*
> > -        * Looking up the url in .git/config.
> > +        * Check if the submodule has been initialized.
> >          * We must not fall back to .gitmodules as we only want
> >          * to process configured submodules.
> 
> This sentence "we must not ..." is also no longer accurate,
> as we do exactly that when using sub->url instead of just url
> below.

You're right, I'll fix that.

-- 
Brandon Williams

Re: [PATCH 00/10] decoupling a submodule's existence and its url

[Stefan Beller]

On Thu, Feb 23, 2017 at 3:47 PM, Brandon Williams <bmwill@google.com> wrote:
> There are two motivations for decoupling a submodule's existence from the url
> that is stored in .git/config.
>
> 1. Worktrees can't really be used with submodules since the existence is
>    checked based on the shared .git/config.  This means that two different
>    worktress have to have the same initialized submodules.  By decoupling a
>    submodule's existence from the url, two different work trees can be
>    configured to have different submodules checked out.

cc Duy for this one:

Well once we have the per-worktree configuration, e.g. [1] we can *technically*
have different submodules in different worktrees by saying

  workingtree0: submodule.<name>.url = git://example.org/real-submodule
  workingtree1: submodule.<name>.url = bogus
  workingtree2: submodule.<name>.url = more bogus

and once we used the URL in the first workingtree all urls are
degraded to a boolean flag,
so the different (possible bogus) content does not do harm, only user-confusion,
because the model of the URL being the flag indicating existence doesn't quite
fit multiple working trees.

[1] https://public-inbox.org/git/20170110112524.12870-3-pclouds@gmail.com/

> 2. Easily configure gorups of submodules that a user is interested in.  In a
>    repository with hundreds of submodules it would be difficult to easily tell git
>    which modules to worry about without having to individually init all of
>    them.  Instead, a pathspec can be used to more easily select or deselect
>    groups of submodules.
>
> This patch series works to do this decoupling and instead allows a user to
> configure submodule.active with a pathspec to use to check if a submodule is
> initialized.

Thanks for stating both intentions!
Stefan

[PATCH v2 00/11] decoupling a submodule's existence and its url

[Brandon Williams]

Differences from v1:
* Reordered some of the patches
* added submodule.<name>.active boolean config option which can be used to do
  more fine grain selection of submodules while still decoupling the existence
  from the URL.
* submodule.<name>.active has precedence over the pathspec stored in
  submodule.active

Brandon Williams (11):
  submodule--helper: add is_active command
  submodule status: use submodule--helper is-active
  submodule deinit: use most reliable url
  submodule sync: use submodule--helper is-active
  submodule--helper clone: check for configured submodules using helper
  submodule: decouple url and submodule existence
  submodule update: add `--init-active` switch
  clone: add --submodule-spec=<pathspec> switch
  completion: clone can initialize specific submodules
  submodule--helper init: set submodule.<name>.active
  submodule add: respect submodule.active and submodule.<name>.active

 Documentation/config.txt               |  15 ++++-
 Documentation/git-clone.txt            |  23 ++++---
 Documentation/git-submodule.txt        |  11 +++-
 builtin/clone.c                        |  36 ++++++++++-
 builtin/submodule--helper.c            |  33 ++++++----
 contrib/completion/git-completion.bash |   1 +
 git-submodule.sh                       |  40 +++++++++---
 submodule.c                            |  36 +++++++++--
 t/t7400-submodule-basic.sh             | 109 +++++++++++++++++++++++++++++++++
 t/t7413-submodule-is-active.sh         | 107 ++++++++++++++++++++++++++++++++
 10 files changed, 376 insertions(+), 35 deletions(-)
 create mode 100755 t/t7413-submodule-is-active.sh

-- 
2.12.0.246.ga2ecc84866-goog

[PATCH v2 01/11] submodule--helper: add is_active command

[Brandon Williams]

There are a lot of places where an explicit check for
submodule."<name>".url is done to see if a submodule exists.  In order
to centralize this check introduce a helper which can be used to query
if a submodule is active or not.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c    | 11 +++++++++++
 t/t7413-submodule-is-active.sh | 31 +++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100755 t/t7413-submodule-is-active.sh

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index df0d9c166..dac02604d 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -1128,6 +1128,16 @@ static int absorb_git_dirs(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+static int is_active(int argc, const char **argv, const char *prefix)
+{
+	if (argc != 2)
+		die("submodule--helper is-active takes exactly 1 arguments");
+
+	gitmodules_config();
+
+	return !is_submodule_initialized(argv[1]);
+}
+
 #define SUPPORT_SUPER_PREFIX (1<<0)
 
 struct cmd_struct {
@@ -1147,6 +1157,7 @@ static struct cmd_struct commands[] = {
 	{"init", module_init, 0},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
+	{"is-active", is_active, 0},
 };
 
 int cmd_submodule__helper(int argc, const char **argv, const char *prefix)
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
new file mode 100755
index 000000000..f18e0c925
--- /dev/null
+++ b/t/t7413-submodule-is-active.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+
+test_description='Test submodule--helper is-active
+
+This test verifies that `git submodue--helper is-active` correclty identifies
+submodules which are "active" and interesting to the user.
+'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	git init sub &&
+	test_commit -C sub initial &&
+	git init super &&
+	test_commit -C super initial &&
+	git -C super submodule add ../sub sub1 &&
+	git -C super submodule add ../sub sub2 &&
+	git -C super commit -a -m "add 2 submodules at sub{1,2}"
+'
+
+test_expect_success 'is-active works with urls' '
+	git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config --unset submodule.sub1.URL &&
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super config submodule.sub1.URL ../sub &&
+	git -C super submodule--helper is-active sub1
+'
+
+test_done
-- 
2.12.0.246.ga2ecc84866-goog

[PATCH v2 02/11] submodule status: use submodule--helper is-active

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 554bd1c49..19660b9c0 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1000,14 +1000,13 @@ cmd_status()
 	do
 		die_if_unmatched "$mode" "$sha1"
 		name=$(git submodule--helper name "$sm_path") || exit
-		url=$(git config submodule."$name".url)
 		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
 		if test "$stage" = U
 		then
 			say "U$sha1 $displaypath"
 			continue
 		fi
-		if test -z "$url" ||
+		if ! git submodule--helper is-active "$sm_path" ||
 		{
 			! test -d "$sm_path"/.git &&
 			! test -f "$sm_path"/.git
-- 
2.12.0.246.ga2ecc84866-goog

[PATCH v2 03/11] submodule deinit: use most reliable url

[Brandon Williams]

The user could have configured the submodule to have a different URL
from the one in the superproject's config.  To account for this read
what the submodule has configured for remote.origin.url and use that
instead.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 19660b9c0..1c2064cc1 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -455,7 +455,7 @@ Submodule work tree '\$displaypath' contains a .git directory
 		then
 			# Remove the whole section so we have a clean state when
 			# the user later decides to init this submodule again
-			url=$(git config submodule."$name".url)
+			url=$(GIT_DIR="$(git rev-parse --git-path modules/$name)" git config remote.origin.url)
 			git config --remove-section submodule."$name" 2>/dev/null &&
 			say "$(eval_gettext "Submodule '\$name' (\$url) unregistered for path '\$displaypath'")"
 		fi
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v2 03/11] submodule deinit: use most reliable url

[Stefan Beller]

On Wed, Mar 8, 2017 at 5:23 PM, Brandon Williams <bmwill@google.com> wrote:
> The user could have configured the submodule to have a different URL
> from the one in the superproject's config.  To account for this read
> what the submodule has configured for remote.origin.url and use that
> instead.

When reading this commit message, I first thought this is an unrelated
bug fix. However it is not unrelated. In a later patch you introduce a mode
where submodule.<name>.url may not exist in .git/config any more,
(there may be a .existence flag instead?) such that url="", which is
obviously a bad UI:

    Submodule '$name' (<empty string>) unregistered for path '$displaypath'"

Like the first patch of this series, we could have a helper function
"git submodule--helper url <name>" that figures out how to get the URL:
1) Look into that GIT_DIR
2) if non-existent check .git/config for the URL or
3) fall back to .gitmodules?

Instead of having such a helper, we directly look into that first option
hoping it exists, however it doesn't have to:

  git submodule init <ps>
  # no command in between
  git submodule deinit <ps>
  # submodule was not cloned yet, but we still test positive for
    > # Remove the .git/config entries (unless the user already did it)
    > if test -n "$(git config --get-regexp submodule."$name\.")"

I am not sure if there is an easy way out here.

Thinking about the name for such a url helper lookup, I wonder if
we want to have

    git submodule--helper show-url <name>

as potentially we end up having this issue for a lot
of different things (e.g. submodule.<name>.shallow = (true,false),
or in case the submodule is cloned you can give the actual depth
as an integral number), so maybe we'd want to introduce one
layer of indirection

    git submodule--helper ask-property \
       (is-active/URL/depth/size/..) <name>

So with that said, I wonder if we also want to ease up:

    GIT_DIR="$(git rev-parse --git-path modules/$name

to be

    GIT_DIR=$(git submodule--helper show-git-dir $name)

>                 then
>                         # Remove the whole section so we have a clean state when
>                         # the user later decides to init this submodule again
> -                       url=$(git config submodule."$name".url)
> +                       url=$(GIT_DIR="$(git rev-parse --git-path modules/$name)" git config remote.origin.url)

In the submodule helper we have get_default_remote(), which we do not
have in shell
(which we seemed to have in shell?), so maybe it's worth not using origin here,
although I guess it is rare that the original remote is named other than origin.

>                         git config --remove-section submodule."$name" 2>/dev/null &&
>                         say "$(eval_gettext "Submodule '\$name' (\$url) unregistered for path '\$displaypath'")"
>                 fi
> --
> 2.12.0.246.ga2ecc84866-goog
>

Thanks,
Stefan

Re: [PATCH v2 03/11] submodule deinit: use most reliable url

[Brandon Williams]

On 03/08, Stefan Beller wrote:
> On Wed, Mar 8, 2017 at 5:23 PM, Brandon Williams <bmwill@google.com> wrote:
> > The user could have configured the submodule to have a different URL
> > from the one in the superproject's config.  To account for this read
> > what the submodule has configured for remote.origin.url and use that
> > instead.
> 
> When reading this commit message, I first thought this is an unrelated
> bug fix. However it is not unrelated. In a later patch you introduce a mode
> where submodule.<name>.url may not exist in .git/config any more,
> (there may be a .existence flag instead?) such that url="", which is
> obviously a bad UI:

So the later patch doesn't actually omit submodule.<name>.url but we
could end up with this if you init and then deinit without actually
cloning the submodule.  So maybe the best bet is to drop this patch from
the series and we can include one like it in a follow up which uses a
helper function that you suggest.

> 
>     Submodule '$name' (<empty string>) unregistered for path '$displaypath'"
> 
> Like the first patch of this series, we could have a helper function
> "git submodule--helper url <name>" that figures out how to get the URL:
> 1) Look into that GIT_DIR
> 2) if non-existent check .git/config for the URL or
> 3) fall back to .gitmodules?
> 
> Instead of having such a helper, we directly look into that first option
> hoping it exists, however it doesn't have to:
> 
>   git submodule init <ps>
>   # no command in between
>   git submodule deinit <ps>
>   # submodule was not cloned yet, but we still test positive for
>     > # Remove the .git/config entries (unless the user already did it)
>     > if test -n "$(git config --get-regexp submodule."$name\.")"
> 
> I am not sure if there is an easy way out here.
> 
> Thinking about the name for such a url helper lookup, I wonder if
> we want to have
> 
>     git submodule--helper show-url <name>
> 
> as potentially we end up having this issue for a lot
> of different things (e.g. submodule.<name>.shallow = (true,false),
> or in case the submodule is cloned you can give the actual depth
> as an integral number), so maybe we'd want to introduce one
> layer of indirection
> 
>     git submodule--helper ask-property \
>        (is-active/URL/depth/size/..) <name>
> 
> So with that said, I wonder if we also want to ease up:
> 
>     GIT_DIR="$(git rev-parse --git-path modules/$name
> 
> to be
> 
>     GIT_DIR=$(git submodule--helper show-git-dir $name)
> 
> >                 then
> >                         # Remove the whole section so we have a clean state when
> >                         # the user later decides to init this submodule again
> > -                       url=$(git config submodule."$name".url)
> > +                       url=$(GIT_DIR="$(git rev-parse --git-path modules/$name)" git config remote.origin.url)
> 
> In the submodule helper we have get_default_remote(), which we do not
> have in shell
> (which we seemed to have in shell?), so maybe it's worth not using origin here,
> although I guess it is rare that the original remote is named other than origin.
> 
> >                         git config --remove-section submodule."$name" 2>/dev/null &&
> >                         say "$(eval_gettext "Submodule '\$name' (\$url) unregistered for path '\$displaypath'")"
> >                 fi
> > --
> > 2.12.0.246.ga2ecc84866-goog
> >
> 
> Thanks,
> Stefan

-- 
Brandon Williams

[PATCH v2 04/11] submodule sync: use submodule--helper is-active

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 1c2064cc1..7ed1aaba3 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1101,7 +1101,7 @@ cmd_sync()
 			;;
 		esac
 
-		if git config "submodule.$name.url" >/dev/null 2>/dev/null
+		if git submodule--helper is-active "$sm_path"
 		then
 			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
 			say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
-- 
2.12.0.246.ga2ecc84866-goog

[PATCH v2 05/11] submodule--helper clone: check for configured submodules using helper

[Brandon Williams]

Use the 'is_submodule_initialized()' helper to check for configured
submodules instead of manually checking for the submodule's URL in the
config.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index dac02604d..bceb62a20 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -760,7 +760,6 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 	struct strbuf displaypath_sb = STRBUF_INIT;
 	struct strbuf sb = STRBUF_INIT;
 	const char *displaypath = NULL;
-	char *url = NULL;
 	int needs_cloning = 0;
 
 	if (ce_stage(ce)) {
@@ -794,15 +793,8 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		goto cleanup;
 	}
 
-	/*
-	 * Looking up the url in .git/config.
-	 * We must not fall back to .gitmodules as we only want
-	 * to process configured submodules.
-	 */
-	strbuf_reset(&sb);
-	strbuf_addf(&sb, "submodule.%s.url", sub->name);
-	git_config_get_string(sb.buf, &url);
-	if (!url) {
+	/* Check if the submodule has been initialized. */
+	if (!is_submodule_initialized(ce->name)) {
 		next_submodule_warn_missing(suc, out, displaypath);
 		goto cleanup;
 	}
@@ -836,7 +828,7 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		argv_array_push(&child->args, "--depth=1");
 	argv_array_pushl(&child->args, "--path", sub->path, NULL);
 	argv_array_pushl(&child->args, "--name", sub->name, NULL);
-	argv_array_pushl(&child->args, "--url", url, NULL);
+	argv_array_pushl(&child->args, "--url", sub->url, NULL);
 	if (suc->references.nr) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, &suc->references)
@@ -846,7 +838,6 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		argv_array_push(&child->args, suc->depth);
 
 cleanup:
-	free(url);
 	strbuf_reset(&displaypath_sb);
 	strbuf_reset(&sb);
 
-- 
2.12.0.246.ga2ecc84866-goog

[PATCH v2 06/11] submodule: decouple url and submodule existence

[Brandon Williams]

Currently the submodule.<name>.url config option is used to determine
if a given submodule exists and is interesting to the user.  This
however doesn't work very well because the URL is a config option for
the scope of a repository, whereas the existence of a submodule is an
option scoped to the working tree.

In a future with worktree support for submodules, there will be multiple
working trees, each of which may only need a subset of the submodules
checked out.  The URL (which is where the submodule repository can be
obtained) should not differ between different working trees.

It may also be convenient for users to more easily specify groups of
submodules they are interested in as apposed to running "git submodule
init <path>" on each submodule they want checked out in their working
tree.

To this end two config options are introduced, submodule.active and
submodule.<name>.active.  The submodule.active config holds a pathspec
that specifies which submodules should exist in the working tree.  The
submodule.<name>.active config is a boolean flag used to indicate if
that particular submodule should exist in the working tree.

Given these multiple ways to check for a submodule's existence the more
fine-grained submodule.<name>.active option has the highest order of
precedence followed by the pathspec check against submodule.active. To
ensure backwards compatibility, if neither of these options are set git
falls back to checking the submodule.<name>.url option to determine a
submodule's existence.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/config.txt       | 15 ++++++++++--
 submodule.c                    | 36 ++++++++++++++++++++++++---
 t/t7413-submodule-is-active.sh | 55 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 100 insertions(+), 6 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 506431267..21037e8c5 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -2888,8 +2888,9 @@ submodule.<name>.url::
 	The URL for a submodule. This variable is copied from the .gitmodules
 	file to the git config via 'git submodule init'. The user can change
 	the configured URL before obtaining the submodule via 'git submodule
-	update'. After obtaining the submodule, the presence of this variable
-	is used as a sign whether the submodule is of interest to git commands.
+	update'. If neither submodule.<name>.active or submodule.active are
+	set, the presence of this variable is used as a fallback to indicate
+	whether the submodule is of interest to git commands.
 	See linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.
 
 submodule.<name>.update::
@@ -2927,6 +2928,16 @@ submodule.<name>.ignore::
 	"--ignore-submodules" option. The 'git submodule' commands are not
 	affected by this setting.
 
+submodule.<name>.active::
+	Boolean value indicating if the submodule is of interest to git
+	commands.  This config option takes precedence over the
+	submodule.active config option.
+
+submodule.active::
+	A repeated field which contains a pathspec used to match against a
+	submodule's path to determine if the submodule is of interest to git
+	commands.
+
 submodule.fetchJobs::
 	Specifies how many submodules are fetched/cloned at the same time.
 	A positive integer allows up to that number of submodules fetched
diff --git a/submodule.c b/submodule.c
index 4c4f033e8..6125736fc 100644
--- a/submodule.c
+++ b/submodule.c
@@ -217,13 +217,41 @@ void gitmodules_config_sha1(const unsigned char *commit_sha1)
 int is_submodule_initialized(const char *path)
 {
 	int ret = 0;
-	const struct submodule *module = NULL;
+	char *key;
+	const struct string_list *sl;
+	const struct submodule *module = submodule_from_path(null_sha1, path);
 
-	module = submodule_from_path(null_sha1, path);
+	/* early return if there isn't a path->module mapping */
+	if (!module)
+		return 0;
+
+	/* submodule.<name>.active is set */
+	key = xstrfmt("submodule.%s.active", module->name);
+	if (!git_config_get_bool(key, &ret)) {
+		free(key);
+		return ret;
+	}
+	free(key);
+
+	sl = git_config_get_value_multi("submodule.active");
 
-	if (module) {
-		char *key = xstrfmt("submodule.%s.url", module->name);
+	if (sl) {
+		struct pathspec ps;
+		struct argv_array args = ARGV_ARRAY_INIT;
+		const struct string_list_item *item;
+
+		for_each_string_list_item(item, sl) {
+			argv_array_push(&args, item->string);
+		}
+
+		parse_pathspec(&ps, 0, 0, 0, args.argv);
+		ret = match_pathspec(&ps, path, strlen(path), 0, NULL, 1);
+
+		argv_array_clear(&args);
+		clear_pathspec(&ps);
+	} else {
 		char *value = NULL;
+		key = xstrfmt("submodule.%s.url", module->name);
 
 		ret = !git_config_get_string(key, &value);
 
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
index f18e0c925..c41b899ab 100755
--- a/t/t7413-submodule-is-active.sh
+++ b/t/t7413-submodule-is-active.sh
@@ -28,4 +28,59 @@ test_expect_success 'is-active works with urls' '
 	git -C super submodule--helper is-active sub1
 '
 
+test_expect_success 'is-active works with submodule.<name>.active config' '
+	git -C super config --bool submodule.sub1.active "false" &&
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+
+	git -C super config --bool submodule.sub1.active "true" &&
+	git -C super config --unset submodule.sub1.URL &&
+	git -C super submodule--helper is-active sub1 &&
+
+	git -C super config submodule.sub1.URL ../sub &&
+	git -C super config --unset submodule.sub1.active
+'
+
+test_expect_success 'is-active works with basic submodule.active config' '
+	git -C super config --add submodule.active "." &&
+	git -C super config --unset submodule.sub1.URL &&
+
+	git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config submodule.sub1.URL ../sub &&
+	git -C super config --unset-all submodule.active
+'
+
+test_expect_success 'is-active correctly works with paths that are not submodules' '
+	test_must_fail git -C super submodule--helper is-active not-a-submodule &&
+
+	git -C super config --add submodule.active "." &&
+	test_must_fail git -C super submodule--helper is-active not-a-submodule &&
+
+	git -C super config --unset-all submodule.active
+'
+
+test_expect_success 'is-active works with exclusions in submodule.active config' '
+	git -C super config --add submodule.active "." &&
+	git -C super config --add submodule.active ":(exclude)sub1" &&
+
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config --unset-all submodule.active
+'
+
+test_expect_success 'is-active with submodule.active and submodule.<name>.active' '
+	git -C super config --add submodule.active "sub1" &&
+	git -C super config --bool submodule.sub1.active "false" &&
+	git -C super config --bool submodule.sub2.active "true" &&
+
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config --unset-all submodule.active &&
+	git -C super config --unset submodule.sub1.active &&
+	git -C super config --unset submodule.sub2.active
+'
+
 test_done
-- 
2.12.0.246.ga2ecc84866-goog

[PATCH v2 07/11] submodule update: add `--init-active` switch

[Brandon Williams]

The new switch `--init-active` initializes the submodules which are
configured in `submodule.active` instead of those given as
command line arguments before updating. In the first implementation this
is made incompatible with further command line arguments as it is
unclear what the user means by

    git submodule update --init --init-active <paths>

This new switch allows users to record more complex patterns as it saves
retyping them whenever you invoke update.

Based on a patch by Stefan Beller <sbeller@google.com>

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/git-submodule.txt | 11 ++++++++++-
 git-submodule.sh                | 21 ++++++++++++++++++---
 t/t7400-submodule-basic.sh      | 39 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 67 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 918bd1d1b..626b9760a 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -14,7 +14,7 @@ SYNOPSIS
 'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]
 'git submodule' [--quiet] init [--] [<path>...]
 'git submodule' [--quiet] deinit [-f|--force] (--all|[--] <path>...)
-'git submodule' [--quiet] update [--init] [--remote] [-N|--no-fetch]
+'git submodule' [--quiet] update [--init[-active]] [--remote] [-N|--no-fetch]
 	      [--[no-]recommend-shallow] [-f|--force] [--rebase|--merge]
 	      [--reference <repository>] [--depth <depth>] [--recursive]
 	      [--jobs <n>] [--] [<path>...]
@@ -195,6 +195,10 @@ If the submodule is not yet initialized, and you just want to use the
 setting as stored in .gitmodules, you can automatically initialize the
 submodule with the `--init` option.
 
+You can configure a set of submodules using pathspec syntax in
+submodule.active you can use `--init-active` to initialize
+those before updating.
+
 If `--recursive` is specified, this command will recurse into the
 registered submodules, and update any nested submodules within.
 --
@@ -378,6 +382,11 @@ the submodule itself.
 	Initialize all submodules for which "git submodule init" has not been
 	called so far before updating.
 
+--init-active::
+	This option is only valid for the update command.
+	Initialize all submodules configured in "`submodule.active`"
+	that have not been updated before.
+
 --name::
 	This option is only valid for the add command. It sets the submodule's
 	name to the given string instead of defaulting to its path. The name
diff --git a/git-submodule.sh b/git-submodule.sh
index 7ed1aaba3..43a393d7e 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -9,7 +9,7 @@ USAGE="[--quiet] add [-b <branch>] [-f|--force] [--name <name>] [--reference <re
    or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]
    or: $dashless [--quiet] init [--] [<path>...]
    or: $dashless [--quiet] deinit [-f|--force] (--all| [--] <path>...)
-   or: $dashless [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference <repository>] [--recursive] [--] [<path>...]
+   or: $dashless [--quiet] update [--init[-active]] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference <repository>] [--recursive] [--] [<path>...]
    or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
    or: $dashless [--quiet] foreach [--recursive] <command>
    or: $dashless [--quiet] sync [--recursive] [--] [<path>...]"
@@ -499,7 +499,12 @@ cmd_update()
 			progress="--progress"
 			;;
 		-i|--init)
-			init=1
+			test -z $init || test $init = by_args || die "$(gettext "Only one of --init or --init-active may be used.")"
+			init=by_args
+			;;
+		--init-active)
+			test -z $init || test $init = by_config || die "$(gettext "Only one of --init or --init-active may be used.")"
+			init=by_config
 			;;
 		--remote)
 			remote=1
@@ -568,7 +573,17 @@ cmd_update()
 
 	if test -n "$init"
 	then
-		cmd_init "--" "$@" || return
+		if test "$init" = "by_config"
+		then
+			if test $# -gt 0
+			then
+				die "$(gettext "path arguments are incompatible with --init-active")"
+			fi
+			cmd_init "--" $(git config --get-all submodule.active) || return
+		else
+			cmd_init "--" "$@" || return
+		fi
+
 	fi
 
 	{
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index b77cce8e4..f5df0a5b4 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1116,5 +1116,44 @@ test_expect_success 'submodule helper list is not confused by common prefixes' '
 	test_cmp expect actual
 '
 
+test_expect_success 'setup superproject with submodules' '
+	git init sub1 &&
+	test_commit -C sub1 test &&
+	test_commit -C sub1 test2 &&
+	git init multisuper &&
+	git -C multisuper submodule add ../sub1 sub0 &&
+	git -C multisuper submodule add ../sub1 sub1 &&
+	git -C multisuper submodule add ../sub1 sub2 &&
+	git -C multisuper submodule add ../sub1 sub3 &&
+	git -C multisuper commit -m "add some submodules"
+'
+
+cat >expect <<-EOF
+-sub0
+ sub1 (test2)
+ sub2 (test2)
+ sub3 (test2)
+EOF
+
+test_expect_success 'submodule update --init with a specification' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone submodule update --init . ":(exclude)sub0" &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'submodule update --init-active' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone config submodule.active "." &&
+	git -C multisuper_clone config --add submodule.active ":(exclude)sub0" &&
+	git -C multisuper_clone submodule update --init-active &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_must_fail git -C multisuper_clone submodule update --init-active sub0 &&
+	test_cmp expect actual
+'
 
 test_done
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v2 07/11] submodule update: add `--init-active` switch

[Stefan Beller]

On Wed, Mar 8, 2017 at 5:23 PM, Brandon Williams <bmwill@google.com> wrote:
> The new switch `--init-active` initializes the submodules which are
> configured in `submodule.active` instead of those given as
> command line arguments before updating. In the first implementation this
> is made incompatible with further command line arguments as it is
> unclear what the user means by
>
>     git submodule update --init --init-active <paths>
>
> This new switch allows users to record more complex patterns as it saves
> retyping them whenever you invoke update.
>
> Based on a patch by Stefan Beller <sbeller@google.com>
>
> Signed-off-by: Brandon Williams <bmwill@google.com>



> @@ -568,7 +573,17 @@ cmd_update()
>
>         if test -n "$init"
>         then
> -               cmd_init "--" "$@" || return
> +               if test "$init" = "by_config"
> +               then
> +                       if test $# -gt 0
> +                       then
> +                               die "$(gettext "path arguments are incompatible with --init-active")"
> +                       fi
> +                       cmd_init "--" $(git config --get-all submodule.active) || return

What happens with submodule.<name>.actives here?

Re: [PATCH v2 07/11] submodule update: add `--init-active` switch

[Brandon Williams]

On 03/08, Stefan Beller wrote:
> On Wed, Mar 8, 2017 at 5:23 PM, Brandon Williams <bmwill@google.com> wrote:
> > The new switch `--init-active` initializes the submodules which are
> > configured in `submodule.active` instead of those given as
> > command line arguments before updating. In the first implementation this
> > is made incompatible with further command line arguments as it is
> > unclear what the user means by
> >
> >     git submodule update --init --init-active <paths>
> >
> > This new switch allows users to record more complex patterns as it saves
> > retyping them whenever you invoke update.
> >
> > Based on a patch by Stefan Beller <sbeller@google.com>
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> 
> 
> 
> > @@ -568,7 +573,17 @@ cmd_update()
> >
> >         if test -n "$init"
> >         then
> > -               cmd_init "--" "$@" || return
> > +               if test "$init" = "by_config"
> > +               then
> > +                       if test $# -gt 0
> > +                       then
> > +                               die "$(gettext "path arguments are incompatible with --init-active")"
> > +                       fi
> > +                       cmd_init "--" $(git config --get-all submodule.active) || return
> 
> What happens with submodule.<name>.actives here?

Nothing?  I am still not 100% certain that we really need the
submodule.<name>.active config but it seemed like what we decided upon
last we discussed this.

Maybe we don't even need this as an option?  Maybe update should be able
to just 'update' all active submodules without having to make an
explicit call to cmd_init?

-- 
Brandon Williams

[PATCH v2 08/11] clone: add --submodule-spec=<pathspec> switch

[Brandon Williams]

The new switch passes the pathspec to `git submodule update
--init-active` which is called after the actual clone is done.

Additionally this configures the submodule.active option to
be the given pathspec, such that any future invocation of
`git submodule update --init-active` will keep up
with the pathspec.

Based on a patch by Stefan Beller <sbeller@google.com>

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/git-clone.txt | 23 ++++++++++-----
 builtin/clone.c             | 36 +++++++++++++++++++++--
 t/t7400-submodule-basic.sh  | 70 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 120 insertions(+), 9 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 35cc34b2f..9692eab30 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -15,7 +15,8 @@ SYNOPSIS
 	  [--dissociate] [--separate-git-dir <git dir>]
 	  [--depth <depth>] [--[no-]single-branch]
 	  [--recursive | --recurse-submodules] [--[no-]shallow-submodules]
-	  [--jobs <n>] [--] <repository> [<directory>]
+	  [--submodule-spec <pathspec>] [--jobs <n>] [--]
+	  <repository> [<directory>]
 
 DESCRIPTION
 -----------
@@ -217,12 +218,20 @@ objects from the source repository into a pack in the cloned repository.
 
 --recursive::
 --recurse-submodules::
-	After the clone is created, initialize all submodules within,
-	using their default settings. This is equivalent to running
-	`git submodule update --init --recursive` immediately after
-	the clone is finished. This option is ignored if the cloned
-	repository does not have a worktree/checkout (i.e. if any of
-	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
+	After the clone is created, initialize and clone all submodules
+	within, using their default settings. This is equivalent to
+	running `git submodule update --recursive --init` immediately
+	after the clone is finished. This option is ignored if the
+	cloned repository does not have a worktree/checkout (i.e.  if
+	any of `--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
+
+--submodule-spec::
+	After the clone is created, initialize and clone specified
+	submodules within, using their default settings. It is possible
+	to give multiple specifications by giving this argument multiple
+	times. This is equivalent to configuring `submodule.active`
+	and then running `git submodule update --init-active`
+	immediately after the clone is finished.
 
 --[no-]shallow-submodules::
 	All submodules which are cloned will be shallow with a depth of 1.
diff --git a/builtin/clone.c b/builtin/clone.c
index 5ef81927a..15a8d3ef0 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -56,6 +56,16 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;
 static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;
 static int option_dissociate;
 static int max_jobs = -1;
+static struct string_list submodule_spec;
+
+static int submodule_spec_cb(const struct option *opt, const char *arg, int unset)
+{
+	if (unset)
+		return -1;
+
+	string_list_append((struct string_list *)opt->value, arg);
+	return 0;
+}
 
 static struct option builtin_clone_options[] = {
 	OPT__VERBOSITY(&option_verbosity),
@@ -112,6 +122,9 @@ static struct option builtin_clone_options[] = {
 			TRANSPORT_FAMILY_IPV4),
 	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
 			TRANSPORT_FAMILY_IPV6),
+	OPT_CALLBACK(0, "submodule-spec", &submodule_spec, N_("<pathspec>"),
+			N_("clone specific submodules. Pass multiple times for complex pathspecs"),
+			submodule_spec_cb),
 	OPT_END()
 };
 
@@ -733,13 +746,21 @@ static int checkout(int submodule_progress)
 	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
 			   sha1_to_hex(sha1), "1", NULL);
 
-	if (!err && option_recursive) {
+	if (!err && (option_recursive || submodule_spec.nr > 0)) {
 		struct argv_array args = ARGV_ARRAY_INIT;
-		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
+		argv_array_pushl(&args, "submodule", "update", NULL);
+
+		if (submodule_spec.nr > 0)
+			argv_array_pushf(&args, "--init-active");
+		else
+			argv_array_pushf(&args, "--init");
 
 		if (option_shallow_submodules == 1)
 			argv_array_push(&args, "--depth=1");
 
+		if (option_recursive)
+			argv_array_pushf(&args, "--recursive");
+
 		if (max_jobs != -1)
 			argv_array_pushf(&args, "--jobs=%d", max_jobs);
 
@@ -887,6 +908,17 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		option_no_checkout = 1;
 	}
 
+	if (submodule_spec.nr > 0) {
+		struct string_list_item *item;
+		struct strbuf sb = STRBUF_INIT;
+		for_each_string_list_item(item, &submodule_spec) {
+			strbuf_addf(&sb, "submodule.active=%s",
+				    item->string);
+			string_list_append(&option_config,
+					   strbuf_detach(&sb, NULL));
+		}
+	}
+
 	if (!option_origin)
 		option_origin = "origin";
 
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index f5df0a5b4..39364ef68 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1156,4 +1156,74 @@ test_expect_success 'submodule update --init-active' '
 	test_cmp expect actual
 '
 
+test_expect_success 'clone --submodule-spec works' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat >expected <<-\EOF &&
+	 sub0 (test2)
+	-sub1
+	-sub2
+	-sub3
+	EOF
+
+	git clone --recurse-submodules --submodule-spec="sub0" multisuper multisuper_clone &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp actual expected
+'
+
+test_expect_success 'clone with multiple --submodule-spec options' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat >expect <<-\EOF &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	EOF
+
+	git clone --recurse-submodules \
+		  --submodule-spec="." \
+		  --submodule-spec ":(exclude)sub0" \
+		  --submodule-spec ":(exclude)sub2" \
+		  multisuper multisuper_clone &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'clone and subsequent updates correctly auto-initialize submodules' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat <<-\EOF >expect &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	EOF
+
+	cat <<-\EOF >expect2 &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	-sub4
+	 sub5 (test2)
+	EOF
+
+	git clone --recurse-submodules \
+		  --submodule-spec="." \
+		  --submodule-spec ":(exclude)sub0" \
+		  --submodule-spec ":(exclude)sub2" \
+		  --submodule-spec ":(exclude)sub4" \
+		  multisuper multisuper_clone &&
+
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect actual &&
+
+	git -C multisuper submodule add ../sub1 sub4 &&
+	git -C multisuper submodule add ../sub1 sub5 &&
+	git -C multisuper commit -m "add more submodules" &&
+	# obtain the new superproject
+	git -C multisuper_clone pull &&
+	git -C multisuper_clone submodule update --init-active &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect2 actual
+'
+
 test_done
-- 
2.12.0.246.ga2ecc84866-goog

[PATCH v2 09/11] completion: clone can initialize specific submodules

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 contrib/completion/git-completion.bash | 1 +
 1 file changed, 1 insertion(+)

diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index 6721ff80f..4e473aa90 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -1138,6 +1138,7 @@ _git_clone ()
 			--single-branch
 			--branch
 			--recurse-submodules
+			--submodule-spec
 			"
 		return
 		;;
-- 
2.12.0.246.ga2ecc84866-goog

[PATCH v2 10/11] submodule--helper init: set submodule.<name>.active

[Brandon Williams]

When initializing a submodule set the submodule.<name>.active config to
true to indicate that the submodule is active.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index bceb62a20..44f2c02ba 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -329,6 +329,13 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 		die(_("No url found for submodule path '%s' in .gitmodules"),
 			displaypath);
 
+	/* Set active flag for the submodule being initialized */
+	if (!is_submodule_initialized(path)) {
+		strbuf_reset(&sb);
+		strbuf_addf(&sb, "submodule.%s.active", sub->name);
+		git_config_set_gently(sb.buf, "true");
+	}
+
 	/*
 	 * Copy url setting when it is not set yet.
 	 * To look up the url in .git/config, we must not fall back to
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v2 10/11] submodule--helper init: set submodule.<name>.active

[Stefan Beller]

On Wed, Mar 8, 2017 at 5:23 PM, Brandon Williams <bmwill@google.com> wrote:
> When initializing a submodule set the submodule.<name>.active config to
> true to indicate that the submodule is active.

So by this patch an init of a submodule performs both
a "regular init" (copy URL and the update setting)
as well as setting the new submodule.<name>.active.

And both happens in the same config file (i.e. no worktree
support here)

Later on we *could* remove the URL as the .active is the new
flag of existence.

But enough of my speculation, I am left wondering what this
patch accomplishes, as this states no agenda as why it is useful
on its own.

>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  builtin/submodule--helper.c | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index bceb62a20..44f2c02ba 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -329,6 +329,13 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
>                 die(_("No url found for submodule path '%s' in .gitmodules"),
>                         displaypath);
>
> +       /* Set active flag for the submodule being initialized */
> +       if (!is_submodule_initialized(path)) {
> +               strbuf_reset(&sb);
> +               strbuf_addf(&sb, "submodule.%s.active", sub->name);
> +               git_config_set_gently(sb.buf, "true");
> +       }
> +
>         /*
>          * Copy url setting when it is not set yet.
>          * To look up the url in .git/config, we must not fall back to
> --
> 2.12.0.246.ga2ecc84866-goog
>

Re: [PATCH v2 10/11] submodule--helper init: set submodule.<name>.active

[Brandon Williams]

On 03/08, Stefan Beller wrote:
> On Wed, Mar 8, 2017 at 5:23 PM, Brandon Williams <bmwill@google.com> wrote:
> > When initializing a submodule set the submodule.<name>.active config to
> > true to indicate that the submodule is active.
> 
> So by this patch an init of a submodule performs both
> a "regular init" (copy URL and the update setting)
> as well as setting the new submodule.<name>.active.
> 
> And both happens in the same config file (i.e. no worktree
> support here)

Well there isn't any per-worktree config yet (unless that series managed
to land and I didn't notice that).  So once their are per-worktree
configs we would need to push this to there.

> 
> Later on we *could* remove the URL as the .active is the new
> flag of existence.
> 
> But enough of my speculation, I am left wondering what this
> patch accomplishes, as this states no agenda as why it is useful
> on its own.
> 
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> >  builtin/submodule--helper.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> > index bceb62a20..44f2c02ba 100644
> > --- a/builtin/submodule--helper.c
> > +++ b/builtin/submodule--helper.c
> > @@ -329,6 +329,13 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
> >                 die(_("No url found for submodule path '%s' in .gitmodules"),
> >                         displaypath);
> >
> > +       /* Set active flag for the submodule being initialized */
> > +       if (!is_submodule_initialized(path)) {
> > +               strbuf_reset(&sb);
> > +               strbuf_addf(&sb, "submodule.%s.active", sub->name);
> > +               git_config_set_gently(sb.buf, "true");
> > +       }
> > +
> >         /*
> >          * Copy url setting when it is not set yet.
> >          * To look up the url in .git/config, we must not fall back to
> > --
> > 2.12.0.246.ga2ecc84866-goog
> >

-- 
Brandon Williams

[PATCH v2 11/11] submodule add: respect submodule.active and submodule.<name>.active

[Brandon Williams]

In addition to adding submodule.<name>.url to the config, set
submodule.<name>.active to true unless submodule.active is configured
and the submodule's path matches the configured pathspec.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh               | 12 ++++++++++++
 t/t7413-submodule-is-active.sh | 21 +++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/git-submodule.sh b/git-submodule.sh
index 43a393d7e..d90fde655 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -271,6 +271,18 @@ or you are unsure what this means choose another name with the '--name' option."
 	fi &&
 	git add --force .gitmodules ||
 	die "$(eval_gettext "Failed to register submodule '\$sm_path'")"
+
+	if git config --get submodule.active >/dev/null
+	then
+		# If the submodule being adding isn't already covered by the
+		# current configured pathspec, set the submodule's active flag
+		if ! git submodule--helper is-active "$sm_path"
+		then
+			git config --add submodule."$sm_name".active "true"
+		fi
+	else
+		git config --add submodule."$sm_name".active "true"
+	fi
 }
 
 #
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
index c41b899ab..865931978 100755
--- a/t/t7413-submodule-is-active.sh
+++ b/t/t7413-submodule-is-active.sh
@@ -15,6 +15,12 @@ test_expect_success 'setup' '
 	test_commit -C super initial &&
 	git -C super submodule add ../sub sub1 &&
 	git -C super submodule add ../sub sub2 &&
+
+	# Remove submodule.<name>.active entries in order to test in an
+	# environment where only URLs are present in the conifg
+	git -C super config --unset submodule.sub1.active &&
+	git -C super config --unset submodule.sub2.active &&
+
 	git -C super commit -a -m "add 2 submodules at sub{1,2}"
 '
 
@@ -83,4 +89,19 @@ test_expect_success 'is-active with submodule.active and submodule.<name>.active
 	git -C super config --unset submodule.sub2.active
 '
 
+test_expect_success 'is-active, submodule.active and submodule add' '
+	test_when_finished "rm -rf super2" &&
+	git init super2 &&
+	test_commit -C super2 initial &&
+	git -C super2 config --add submodule.active "sub*" &&
+
+	# submodule add should only add submodule.<name>.active
+	# to the config if not matched by the pathspec
+	git -C super2 submodule add ../sub sub1 &&
+	test_must_fail git -C super2 config --get submodule.sub1.active &&
+
+	git -C super2 submodule add ../sub mod &&
+	git -C super2 config --get submodule.mod.active
+'
+
 test_done
-- 
2.12.0.246.ga2ecc84866-goog

[PATCH v3 00/10] decoupling a submodule's existence and its url

[Brandon Williams]

changes in v3:

* Droped a patch which tried to use a more accurate URL for deinit.  It didn't
  really fit inside the scope of this series.  It may be something we want to
  revisit later though.

* The --init-active flag now ensure that all submodules which are configured to
  be 'active' (either via 'submodule.active' or 'submodule.<name>.active') go
  through the initialization phase and have their relevent info copied over to
  the config.


Brandon Williams (10):
  submodule--helper: add is_active command
  submodule status: use submodule--helper is-active
  submodule sync: use submodule--helper is-active
  submodule--helper clone: check for configured submodules using helper
  submodule: decouple url and submodule existence
  submodule update: add `--init-active` switch
  clone: add --submodule-spec=<pathspec> switch
  completion: clone can initialize specific submodules
  submodule--helper init: set submodule.<name>.active
  submodule add: respect submodule.active and submodule.<name>.active

 Documentation/config.txt               |  15 +++-
 Documentation/git-clone.txt            |  23 ++++--
 Documentation/git-submodule.txt        |   9 +++
 builtin/clone.c                        |  36 +++++++++-
 builtin/submodule--helper.c            |  68 ++++++++++++++----
 contrib/completion/git-completion.bash |   1 +
 git-submodule.sh                       |  43 +++++++++--
 submodule.c                            |  36 ++++++++--
 t/t7400-submodule-basic.sh             | 128 +++++++++++++++++++++++++++++++++
 t/t7413-submodule-is-active.sh         | 107 +++++++++++++++++++++++++++
 10 files changed, 431 insertions(+), 35 deletions(-)
 create mode 100755 t/t7413-submodule-is-active.sh

-- 
2.12.0.246.ga2ecc84866-goog

[PATCH v3 01/10] submodule--helper: add is_active command

[Brandon Williams]

There are a lot of places where an explicit check for
submodule."<name>".url is done to see if a submodule exists.  In order
to centralize this check introduce a helper which can be used to query
if a submodule is active or not.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c    | 11 +++++++++++
 t/t7413-submodule-is-active.sh | 31 +++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100755 t/t7413-submodule-is-active.sh

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 15a5430c0..5fe7e23b1 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -1127,6 +1127,16 @@ static int absorb_git_dirs(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+static int is_active(int argc, const char **argv, const char *prefix)
+{
+	if (argc != 2)
+		die("submodule--helper is-active takes exactly 1 arguments");
+
+	gitmodules_config();
+
+	return !is_submodule_initialized(argv[1]);
+}
+
 #define SUPPORT_SUPER_PREFIX (1<<0)
 
 struct cmd_struct {
@@ -1146,6 +1156,7 @@ static struct cmd_struct commands[] = {
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
+	{"is-active", is_active, 0},
 };
 
 int cmd_submodule__helper(int argc, const char **argv, const char *prefix)
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
new file mode 100755
index 000000000..f18e0c925
--- /dev/null
+++ b/t/t7413-submodule-is-active.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+
+test_description='Test submodule--helper is-active
+
+This test verifies that `git submodue--helper is-active` correclty identifies
+submodules which are "active" and interesting to the user.
+'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	git init sub &&
+	test_commit -C sub initial &&
+	git init super &&
+	test_commit -C super initial &&
+	git -C super submodule add ../sub sub1 &&
+	git -C super submodule add ../sub sub2 &&
+	git -C super commit -a -m "add 2 submodules at sub{1,2}"
+'
+
+test_expect_success 'is-active works with urls' '
+	git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config --unset submodule.sub1.URL &&
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super config submodule.sub1.URL ../sub &&
+	git -C super submodule--helper is-active sub1
+'
+
+test_done
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v3 01/10] submodule--helper: add is_active command

[Stefan Beller]

On Mon, Mar 13, 2017 at 2:43 PM, Brandon Williams <bmwill@google.com> wrote:
> There are a lot of places where an explicit check for
> submodule."<name>".url is done to see if a submodule exists.  In order
> to centralize this check introduce a helper which can be used to query
> if a submodule is active or not.

With this patch in mind, I need to take notes for rerolling
http://public-inbox.org/git/20170209020855.23486-1-sbeller@google.com/

>  #define SUPPORT_SUPER_PREFIX (1<<0)
...
> +       {"is-active", is_active, 0},

I think you can even mark it as SUPPORT_SUPER_PREFIX.
The only messages produced are from die()ing in git_config_get_string
in is_submodule_initialized.

This alone doesn't warrant a reroll though; just in case you do reroll, this
may be worth checking.

Re: [PATCH v3 01/10] submodule--helper: add is_active command

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> There are a lot of places where an explicit check for
> submodule."<name>".url is done to see if a submodule exists.  In order
> to centralize this check introduce a helper which can be used to query
> if a submodule is active or not.

    "Right now, is_submodule_initialized(), the underlying
    implementation of "submodule--helper is-active", checks exactly
    the same thing.  When submodule.<name>.url is set to a non-empty
    string, then it is initialized.  We'd want to change this
    definition in later steps of this series and having the checks
    centralized is a necessary preparatory step."

or something like that need to be tacked at the end to make it known
that 02-04/10 are meant as pure code clean-up without improvement in
behaviour.

"Tacking at the end" will make the whole thing awkward to read, and
that is because the flow of logic is inverted.  The first sentence,
"there are a lot of places...", is trying to state that there is a
problem worthy of fixing, but it does so without saying why it is a
problem in the first place.  After step 04/10 of this series, we
would still be able to say "There are a log of places where an
explicit call to "submodule--helper is-active" is done to see if a
submodule exists."  Is that still a problem?  The answer depends on
what motivated this change, and it is better to make the motivation
known in advance.

Here is my attempt to rewrite the whole thing.

    The definition of which submodules are of interest by the user
    is tied to the configuration submodule.<name>.url; when it is
    set to a non-empty string, it is of interest.  We'd want to be
    able to later change this definition, but there are many places
    that explicitly check this condition in the scripted Porcelain.

    Introduce the "is-active" subcommand to "submodule--helper", so
    that the exact definition of what submodule is of interest can
    be centrally defined (and changed in later steps).  In a few
    patches that follow, this helper is used to replace the explicit
    checks of the configuration variable in scripts.

You are adding the "is-active" subcommand, and the implementation
detail to do so is by using a new is_active() function, so the patch
title needs a bit of tweaking, too.

	Subject: submodule--helper: add is-active subcommand

> diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
> new file mode 100755
> index 000000000..f18e0c925
> --- /dev/null
> +++ b/t/t7413-submodule-is-active.sh
> @@ -0,0 +1,31 @@
> +#!/bin/sh
> +
> +test_description='Test submodule--helper is-active
> +
> +This test verifies that `git submodue--helper is-active` correclty identifies
> +submodules which are "active" and interesting to the user.
> +'
> +
> +. ./test-lib.sh
> +
> +test_expect_success 'setup' '
> +	git init sub &&
> +	test_commit -C sub initial &&
> +	git init super &&
> +	test_commit -C super initial &&
> +	git -C super submodule add ../sub sub1 &&
> +	git -C super submodule add ../sub sub2 &&
> +	git -C super commit -a -m "add 2 submodules at sub{1,2}"
> +'
> +
> +test_expect_success 'is-active works with urls' '
> +	git -C super submodule--helper is-active sub1 &&
> +	git -C super submodule--helper is-active sub2 &&
> +
> +	git -C super config --unset submodule.sub1.URL &&
> +	test_must_fail git -C super submodule--helper is-active sub1 &&
> +	git -C super config submodule.sub1.URL ../sub &&
> +	git -C super submodule--helper is-active sub1
> +'

Looks sensible.

Thanks.

Re: [PATCH v3 01/10] submodule--helper: add is_active command

[Brandon Williams]

On 03/14, Junio C Hamano wrote:
> Here is my attempt to rewrite the whole thing.
> 
>     The definition of which submodules are of interest by the user
>     is tied to the configuration submodule.<name>.url; when it is
>     set to a non-empty string, it is of interest.  We'd want to be
>     able to later change this definition, but there are many places
>     that explicitly check this condition in the scripted Porcelain.
> 
>     Introduce the "is-active" subcommand to "submodule--helper", so
>     that the exact definition of what submodule is of interest can
>     be centrally defined (and changed in later steps).  In a few
>     patches that follow, this helper is used to replace the explicit
>     checks of the configuration variable in scripts.
> 
> You are adding the "is-active" subcommand, and the implementation
> detail to do so is by using a new is_active() function, so the patch
> title needs a bit of tweaking, too.
> 
> 	Subject: submodule--helper: add is-active subcommand

Thanks for helping wordsmith this.  Since starting to contribute I've
realized that writing a good commit message is probably the hardest
part of the whole process.

-- 
Brandon Williams

[PATCH v3 02/10] submodule status: use submodule--helper is-active

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 136e26a2c..ab233712d 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1010,14 +1010,13 @@ cmd_status()
 	do
 		die_if_unmatched "$mode" "$sha1"
 		name=$(git submodule--helper name "$sm_path") || exit
-		url=$(git config submodule."$name".url)
 		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
 		if test "$stage" = U
 		then
 			say "U$sha1 $displaypath"
 			continue
 		fi
-		if test -z "$url" ||
+		if ! git submodule--helper is-active "$sm_path" ||
 		{
 			! test -d "$sm_path"/.git &&
 			! test -f "$sm_path"/.git
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v3 02/10] submodule status: use submodule--helper is-active

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  git-submodule.sh | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/git-submodule.sh b/git-submodule.sh
> index 136e26a2c..ab233712d 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -1010,14 +1010,13 @@ cmd_status()
>  	do
>  		die_if_unmatched "$mode" "$sha1"
>  		name=$(git submodule--helper name "$sm_path") || exit
> -		url=$(git config submodule."$name".url)
>  		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
>  		if test "$stage" = U
>  		then
>  			say "U$sha1 $displaypath"
>  			continue
>  		fi
> -		if test -z "$url" ||
> +		if ! git submodule--helper is-active "$sm_path" ||
>  		{
>  			! test -d "$sm_path"/.git &&
>  			! test -f "$sm_path"/.git

The $name is no longer used after this step in cmd_status function, 
as the sole purpose of learning the name from the path was so that
we can ask if the submodule has .URL defined and the query is done
by name, not path.

This actually raises a question.  

Shouldn't "submodule--helper is-active" ask about submodule while
identifying the submodule in question by name?  Or do all (or most
of) the callers start from path and ask is-active on them so that it
is handier to let them ask by path?

Re: [PATCH v3 02/10] submodule status: use submodule--helper is-active

[Stefan Beller]

On Tue, Mar 14, 2017 at 10:46 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Brandon Williams <bmwill@google.com> writes:
>
>> Signed-off-by: Brandon Williams <bmwill@google.com>
>> ---
>>  git-submodule.sh | 3 +--
>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>
>> diff --git a/git-submodule.sh b/git-submodule.sh
>> index 136e26a2c..ab233712d 100755
>> --- a/git-submodule.sh
>> +++ b/git-submodule.sh
>> @@ -1010,14 +1010,13 @@ cmd_status()
>>       do
>>               die_if_unmatched "$mode" "$sha1"
>>               name=$(git submodule--helper name "$sm_path") || exit
>> -             url=$(git config submodule."$name".url)
>>               displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
>>               if test "$stage" = U
>>               then
>>                       say "U$sha1 $displaypath"
>>                       continue
>>               fi
>> -             if test -z "$url" ||
>> +             if ! git submodule--helper is-active "$sm_path" ||
>>               {
>>                       ! test -d "$sm_path"/.git &&
>>                       ! test -f "$sm_path"/.git
>
> The $name is no longer used after this step in cmd_status function,
> as the sole purpose of learning the name from the path was so that
> we can ask if the submodule has .URL defined and the query is done
> by name, not path.
>
> This actually raises a question.
>
> Shouldn't "submodule--helper is-active" ask about submodule while
> identifying the submodule in question by name?  Or do all (or most
> of) the callers start from path and ask is-active on them so that it
> is handier to let them ask by path?

A similar observation can be made, when looking at
submodule_from_name and submodule_from_path
(both defined in submodule-config.h)

submodule_from_name has only one real world use case in code,
which I suspect is even a bug.

That line was last touched with 851e18c3859a (submodule: use new config
API for worktree configurations). but the original mechanism comes
from 7dce19d374 (fetch/pull: Add the --recurse-submodules option).

There we take the path as name and if a real name exists, the name
is overwritten with the real name, i.e.

    name = name_for_path(path) ? name_for_path(path) : path;

which IMHO is overly accepting and we should just die in case of
!name_for_path.

So to answer your original question, I think the codebase currently
thinks by_path is handier, the name is a mere internal field in
"struct submodule", useful for looking up its git dir.

Thanks,
Stefan

Re: [PATCH v3 02/10] submodule status: use submodule--helper is-active

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> There we take the path as name and if a real name exists, the name
> is overwritten with the real name, i.e.
>
>     name = name_for_path(path) ? name_for_path(path) : path;
>
> which IMHO is overly accepting and we should just die in case of
> !name_for_path.

Yes, I agree that falling back (unless path is a submodule yet to be
added to .gitmodules) is bad.

Re: [PATCH v3 02/10] submodule status: use submodule--helper is-active

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> So to answer your original question, I think the codebase currently
> thinks by_path is handier, the name is a mere internal field in
> "struct submodule", useful for looking up its git dir.

OK, that makes sense at the conceptual level, too.  When you have a
specific version of the superproject checked out, you would want to
identify which submodule to work on by the path to it inside that
version of the superproject.  The name is used only to keep track
of attributes of the submodule that will not change across versions
of the superproject, no matter where in its tree it binds the
submodule at.

Thanks.

Re: [PATCH v3 02/10] submodule status: use submodule--helper is-active

[Brandon Williams]

On 03/14, Junio C Hamano wrote:
> Brandon Williams <bmwill@google.com> writes:
> 
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> >  git-submodule.sh | 3 +--
> >  1 file changed, 1 insertion(+), 2 deletions(-)
> >
> > diff --git a/git-submodule.sh b/git-submodule.sh
> > index 136e26a2c..ab233712d 100755
> > --- a/git-submodule.sh
> > +++ b/git-submodule.sh
> > @@ -1010,14 +1010,13 @@ cmd_status()
> >  	do
> >  		die_if_unmatched "$mode" "$sha1"
> >  		name=$(git submodule--helper name "$sm_path") || exit
> > -		url=$(git config submodule."$name".url)
> >  		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
> >  		if test "$stage" = U
> >  		then
> >  			say "U$sha1 $displaypath"
> >  			continue
> >  		fi
> > -		if test -z "$url" ||
> > +		if ! git submodule--helper is-active "$sm_path" ||
> >  		{
> >  			! test -d "$sm_path"/.git &&
> >  			! test -f "$sm_path"/.git
> 
> The $name is no longer used after this step in cmd_status function, 
> as the sole purpose of learning the name from the path was so that
> we can ask if the submodule has .URL defined and the query is done
> by name, not path.

Thanks, I'll remove computing the name.

> 
> This actually raises a question.  
> 
> Shouldn't "submodule--helper is-active" ask about submodule while
> identifying the submodule in question by name?  Or do all (or most
> of) the callers start from path and ask is-active on them so that it
> is handier to let them ask by path?

As you can Stefan already discussed the path is a much easier handle to
work with simply because that is what is more readily available as the
name is an internal detail.

-- 
Brandon Williams

Re: [PATCH v3 02/10] submodule status: use submodule--helper is-active

[Brandon Williams]

On 03/14, Brandon Williams wrote:
> On 03/14, Junio C Hamano wrote:
> > Brandon Williams <bmwill@google.com> writes:
> > 
> > > Signed-off-by: Brandon Williams <bmwill@google.com>
> > > ---
> > >  git-submodule.sh | 3 +--
> > >  1 file changed, 1 insertion(+), 2 deletions(-)
> > >
> > > diff --git a/git-submodule.sh b/git-submodule.sh
> > > index 136e26a2c..ab233712d 100755
> > > --- a/git-submodule.sh
> > > +++ b/git-submodule.sh
> > > @@ -1010,14 +1010,13 @@ cmd_status()
> > >  	do
> > >  		die_if_unmatched "$mode" "$sha1"
> > >  		name=$(git submodule--helper name "$sm_path") || exit
> > > -		url=$(git config submodule."$name".url)
> > >  		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
> > >  		if test "$stage" = U
> > >  		then
> > >  			say "U$sha1 $displaypath"
> > >  			continue
> > >  		fi
> > > -		if test -z "$url" ||
> > > +		if ! git submodule--helper is-active "$sm_path" ||
> > >  		{
> > >  			! test -d "$sm_path"/.git &&
> > >  			! test -f "$sm_path"/.git
> > 
> > The $name is no longer used after this step in cmd_status function, 
> > as the sole purpose of learning the name from the path was so that
> > we can ask if the submodule has .URL defined and the query is done
> > by name, not path.
> 
> Thanks, I'll remove computing the name.

Actually looks like it should stay in for now. It's used to exit if
there doesn't exist a path mapping in the users .gitmodules for a given
submodule.

> 
> > 
> > This actually raises a question.  
> > 
> > Shouldn't "submodule--helper is-active" ask about submodule while
> > identifying the submodule in question by name?  Or do all (or most
> > of) the callers start from path and ask is-active on them so that it
> > is handier to let them ask by path?
> 
> As you can Stefan already discussed the path is a much easier handle to
> work with simply because that is what is more readily available as the
> name is an internal detail.
> 
> -- 
> Brandon Williams

-- 
Brandon Williams

[PATCH v3 03/10] submodule sync: use submodule--helper is-active

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index ab233712d..e2d08595f 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1111,7 +1111,7 @@ cmd_sync()
 			;;
 		esac
 
-		if git config "submodule.$name.url" >/dev/null 2>/dev/null
+		if git submodule--helper is-active "$sm_path"
 		then
 			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
 			say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v3 03/10] submodule sync: use submodule--helper is-active

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  git-submodule.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/git-submodule.sh b/git-submodule.sh
> index ab233712d..e2d08595f 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -1111,7 +1111,7 @@ cmd_sync()
>  			;;
>  		esac
>  
> -		if git config "submodule.$name.url" >/dev/null 2>/dev/null
> +		if git submodule--helper is-active "$sm_path"
>  		then
>  			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
>  			say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"

This is not a problem this patch introduces, but the loop this hunk
is in seems a bit inefficient.  It maps the sm_path to its name and
then asks .gitmodules the URL the upstream suggests to clone it from,
munges it with a large case statement and discards all of that if
the module is not active.

Adding this patch on top would be a way to remove the inefficiency
and one level of nesting while at it, I think, but I may have missed
something, so please double check, and if you agree that this is a
good way to go, please do so as a preparatory clean-up.

Thanks.

 git-submodule.sh | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index e2d08595f0..dcdd36fa64 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1089,6 +1089,12 @@ cmd_sync()
 	while read mode sha1 stage sm_path
 	do
 		die_if_unmatched "$mode" "$sha1"
+
+		if ! git submodule--helper is-active "$sm_path"
+		then
+			continue
+		fi
+
 		name=$(git submodule--helper name "$sm_path")
 		url=$(git config -f .gitmodules --get submodule."$name".url)
 
@@ -1111,14 +1117,12 @@ cmd_sync()
 			;;
 		esac
 
-		if git submodule--helper is-active "$sm_path"
-		then
-			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-			say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
-			git config submodule."$name".url "$super_config_url"
+		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
+		say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
+		git config submodule."$name".url "$super_config_url"
 
-			if test -e "$sm_path"/.git
-			then
+		if test -e "$sm_path"/.git
+		then
 			(
 				sanitize_submodule_env
 				cd "$sm_path"
@@ -1131,7 +1135,6 @@ cmd_sync()
 					eval cmd_sync
 				fi
 			)
-			fi
 		fi
 	done
 }

Re: [PATCH v3 03/10] submodule sync: use submodule--helper is-active

[Brandon Williams]

On 03/14, Junio C Hamano wrote:
> Brandon Williams <bmwill@google.com> writes:
> 
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> >  git-submodule.sh | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/git-submodule.sh b/git-submodule.sh
> > index ab233712d..e2d08595f 100755
> > --- a/git-submodule.sh
> > +++ b/git-submodule.sh
> > @@ -1111,7 +1111,7 @@ cmd_sync()
> >  			;;
> >  		esac
> >  
> > -		if git config "submodule.$name.url" >/dev/null 2>/dev/null
> > +		if git submodule--helper is-active "$sm_path"
> >  		then
> >  			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
> >  			say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
> 
> This is not a problem this patch introduces, but the loop this hunk
> is in seems a bit inefficient.  It maps the sm_path to its name and
> then asks .gitmodules the URL the upstream suggests to clone it from,
> munges it with a large case statement and discards all of that if
> the module is not active.
> 
> Adding this patch on top would be a way to remove the inefficiency
> and one level of nesting while at it, I think, but I may have missed
> something, so please double check, and if you agree that this is a
> good way to go, please do so as a preparatory clean-up.
> 
> Thanks.

Yeah you're right, some of that work can be avoided.  I'll do the
code cleanup in a prep patch and then convert submodule sync to use the
is-active subcommand.

> 
>  git-submodule.sh | 19 +++++++++++--------
>  1 file changed, 11 insertions(+), 8 deletions(-)
> 
> diff --git a/git-submodule.sh b/git-submodule.sh
> index e2d08595f0..dcdd36fa64 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -1089,6 +1089,12 @@ cmd_sync()
>  	while read mode sha1 stage sm_path
>  	do
>  		die_if_unmatched "$mode" "$sha1"
> +
> +		if ! git submodule--helper is-active "$sm_path"
> +		then
> +			continue
> +		fi
> +
>  		name=$(git submodule--helper name "$sm_path")
>  		url=$(git config -f .gitmodules --get submodule."$name".url)
>  
> @@ -1111,14 +1117,12 @@ cmd_sync()
>  			;;
>  		esac
>  
> -		if git submodule--helper is-active "$sm_path"
> -		then
> -			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
> -			say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
> -			git config submodule."$name".url "$super_config_url"
> +		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
> +		say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
> +		git config submodule."$name".url "$super_config_url"
>  
> -			if test -e "$sm_path"/.git
> -			then
> +		if test -e "$sm_path"/.git
> +		then
>  			(
>  				sanitize_submodule_env
>  				cd "$sm_path"
> @@ -1131,7 +1135,6 @@ cmd_sync()
>  					eval cmd_sync
>  				fi
>  			)
> -			fi
>  		fi
>  	done
>  }

-- 
Brandon Williams

[PATCH v3 04/10] submodule--helper clone: check for configured submodules using helper

[Brandon Williams]

Use the 'is_submodule_initialized()' helper to check for configured
submodules instead of manually checking for the submodule's URL in the
config.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 5fe7e23b1..f38e332c5 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -759,7 +759,6 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 	struct strbuf displaypath_sb = STRBUF_INIT;
 	struct strbuf sb = STRBUF_INIT;
 	const char *displaypath = NULL;
-	char *url = NULL;
 	int needs_cloning = 0;
 
 	if (ce_stage(ce)) {
@@ -793,15 +792,8 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		goto cleanup;
 	}
 
-	/*
-	 * Looking up the url in .git/config.
-	 * We must not fall back to .gitmodules as we only want
-	 * to process configured submodules.
-	 */
-	strbuf_reset(&sb);
-	strbuf_addf(&sb, "submodule.%s.url", sub->name);
-	git_config_get_string(sb.buf, &url);
-	if (!url) {
+	/* Check if the submodule has been initialized. */
+	if (!is_submodule_initialized(ce->name)) {
 		next_submodule_warn_missing(suc, out, displaypath);
 		goto cleanup;
 	}
@@ -835,7 +827,7 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		argv_array_push(&child->args, "--depth=1");
 	argv_array_pushl(&child->args, "--path", sub->path, NULL);
 	argv_array_pushl(&child->args, "--name", sub->name, NULL);
-	argv_array_pushl(&child->args, "--url", url, NULL);
+	argv_array_pushl(&child->args, "--url", sub->url, NULL);
 	if (suc->references.nr) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, &suc->references)
@@ -845,7 +837,6 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		argv_array_push(&child->args, suc->depth);
 
 cleanup:
-	free(url);
 	strbuf_reset(&displaypath_sb);
 	strbuf_reset(&sb);
 
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v3 04/10] submodule--helper clone: check for configured submodules using helper

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> -	/*
> -	 * Looking up the url in .git/config.
> -	 * We must not fall back to .gitmodules as we only want
> -	 * to process configured submodules.
> -	 */
> -	strbuf_reset(&sb);
> -	strbuf_addf(&sb, "submodule.%s.url", sub->name);
> -	git_config_get_string(sb.buf, &url);
> -	if (!url) {
> +	/* Check if the submodule has been initialized. */
> +	if (!is_submodule_initialized(ce->name)) {
>  		next_submodule_warn_missing(suc, out, displaypath);
>  		goto cleanup;
>  	}
> @@ -835,7 +827,7 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
>  		argv_array_push(&child->args, "--depth=1");
>  	argv_array_pushl(&child->args, "--path", sub->path, NULL);
>  	argv_array_pushl(&child->args, "--name", sub->name, NULL);
> -	argv_array_pushl(&child->args, "--url", url, NULL);
> +	argv_array_pushl(&child->args, "--url", sub->url, NULL);

Even without this patch, we already had an instance of struct submodule
available in this function, so the query to .git/config this patch removed
was unnecessary?

I am wondering what was meant by the comment "We must not fall back to..."
that is being removed---is that because sub->url can come from .gitmodules
that is in-tree, not from .git/config?  If that is the case, doesn't the
change in this hunk change behaviour from using the URL the user prefers
to using the URL the upstream suggests, overriding user's configuration?

> @@ -845,7 +837,6 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
>  		argv_array_push(&child->args, suc->depth);
>  
>  cleanup:
> -	free(url);
>  	strbuf_reset(&displaypath_sb);
>  	strbuf_reset(&sb);

Re: [PATCH v3 04/10] submodule--helper clone: check for configured submodules using helper

[Stefan Beller]

On Tue, Mar 14, 2017 at 11:06 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Brandon Williams <bmwill@google.com> writes:
>
>> -     /*
>> -      * Looking up the url in .git/config.
>> -      * We must not fall back to .gitmodules as we only want
>> -      * to process configured submodules.
>> -      */
>> -     strbuf_reset(&sb);
>> -     strbuf_addf(&sb, "submodule.%s.url", sub->name);
>> -     git_config_get_string(sb.buf, &url);
>> -     if (!url) {
>> +     /* Check if the submodule has been initialized. */
>> +     if (!is_submodule_initialized(ce->name)) {
>>               next_submodule_warn_missing(suc, out, displaypath);
>>               goto cleanup;
>>       }
>> @@ -835,7 +827,7 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
>>               argv_array_push(&child->args, "--depth=1");
>>       argv_array_pushl(&child->args, "--path", sub->path, NULL);
>>       argv_array_pushl(&child->args, "--name", sub->name, NULL);
>> -     argv_array_pushl(&child->args, "--url", url, NULL);
>> +     argv_array_pushl(&child->args, "--url", sub->url, NULL);
>
> Even without this patch, we already had an instance of struct submodule
> available in this function, so the query to .git/config this patch removed
> was unnecessary?
>
> I am wondering what was meant by the comment "We must not fall back to..."
> that is being removed---is that because sub->url can come from .gitmodules
> that is in-tree, not from .git/config?

Yes. We want to check for the submodule being "initialized", i.e.
having a url in .git/config. (and the struct submodule reads in both .git/config
and .gitmodules and overlays them with a given precedence order)

>  If that is the case, doesn't the
> change in this hunk change behaviour from using the URL the user prefers
> to using the URL the upstream suggests, overriding user's configuration?

The mentioned precedence makes sure to have the right order:

    /* Overlay the parsed .gitmodules file with .git/config */
    gitmodules_config();
    git_config(submodule_config, NULL);

such that the sub->url is correct as a URL, but not correct as a boolean
indicator if the submodule is "initialized".

Re: [PATCH v3 04/10] submodule--helper clone: check for configured submodules using helper

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

>> I am wondering what was meant by the comment "We must not fall back to..."
>> that is being removed---is that because sub->url can come from .gitmodules
>> that is in-tree, not from .git/config?
>
> Yes. We want to check for the submodule being "initialized", i.e.
> having a url in .git/config. (and the struct submodule reads in both .git/config
> and .gitmodules and overlays them with a given precedence order)

OK, thanks.

I agree that the current code happens to be OK but the arrangement
looks rather brittle to me.  It seems the code liberally call
gitmodules_config() without making sure it is early enough to do so
(iow, git_config() for submodule-related stuff has not been done and
will be done before the values read there are the ones that are
used).

Re: [PATCH v3 04/10] submodule--helper clone: check for configured submodules using helper

[Brandon Williams]

On 03/14, Stefan Beller wrote:
> On Tue, Mar 14, 2017 at 11:06 AM, Junio C Hamano <gitster@pobox.com> wrote:
> > Brandon Williams <bmwill@google.com> writes:
> >
> >> -     /*
> >> -      * Looking up the url in .git/config.
> >> -      * We must not fall back to .gitmodules as we only want
> >> -      * to process configured submodules.
> >> -      */
> >> -     strbuf_reset(&sb);
> >> -     strbuf_addf(&sb, "submodule.%s.url", sub->name);
> >> -     git_config_get_string(sb.buf, &url);
> >> -     if (!url) {
> >> +     /* Check if the submodule has been initialized. */
> >> +     if (!is_submodule_initialized(ce->name)) {
> >>               next_submodule_warn_missing(suc, out, displaypath);
> >>               goto cleanup;
> >>       }
> >> @@ -835,7 +827,7 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
> >>               argv_array_push(&child->args, "--depth=1");
> >>       argv_array_pushl(&child->args, "--path", sub->path, NULL);
> >>       argv_array_pushl(&child->args, "--name", sub->name, NULL);
> >> -     argv_array_pushl(&child->args, "--url", url, NULL);
> >> +     argv_array_pushl(&child->args, "--url", sub->url, NULL);
> >
> > Even without this patch, we already had an instance of struct submodule
> > available in this function, so the query to .git/config this patch removed
> > was unnecessary?
> >
> > I am wondering what was meant by the comment "We must not fall back to..."
> > that is being removed---is that because sub->url can come from .gitmodules
> > that is in-tree, not from .git/config?
> 
> Yes. We want to check for the submodule being "initialized", i.e.
> having a url in .git/config. (and the struct submodule reads in both .git/config
> and .gitmodules and overlays them with a given precedence order)
> 
> >  If that is the case, doesn't the
> > change in this hunk change behaviour from using the URL the user prefers
> > to using the URL the upstream suggests, overriding user's configuration?
> 
> The mentioned precedence makes sure to have the right order:
> 
>     /* Overlay the parsed .gitmodules file with .git/config */
>     gitmodules_config();
>     git_config(submodule_config, NULL);
> 
> such that the sub->url is correct as a URL, but not correct as a boolean
> indicator if the submodule is "initialized".

Thanks for clarifying this.

-- 
Brandon Williams

[PATCH v3 05/10] submodule: decouple url and submodule existence

[Brandon Williams]

Currently the submodule.<name>.url config option is used to determine
if a given submodule exists and is interesting to the user.  This
however doesn't work very well because the URL is a config option for
the scope of a repository, whereas the existence of a submodule is an
option scoped to the working tree.

In a future with worktree support for submodules, there will be multiple
working trees, each of which may only need a subset of the submodules
checked out.  The URL (which is where the submodule repository can be
obtained) should not differ between different working trees.

It may also be convenient for users to more easily specify groups of
submodules they are interested in as apposed to running "git submodule
init <path>" on each submodule they want checked out in their working
tree.

To this end two config options are introduced, submodule.active and
submodule.<name>.active.  The submodule.active config holds a pathspec
that specifies which submodules should exist in the working tree.  The
submodule.<name>.active config is a boolean flag used to indicate if
that particular submodule should exist in the working tree.

Given these multiple ways to check for a submodule's existence the more
fine-grained submodule.<name>.active option has the highest order of
precedence followed by the pathspec check against submodule.active. To
ensure backwards compatibility, if neither of these options are set git
falls back to checking the submodule.<name>.url option to determine a
submodule's existence.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/config.txt       | 15 ++++++++++--
 submodule.c                    | 36 ++++++++++++++++++++++++---
 t/t7413-submodule-is-active.sh | 55 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 100 insertions(+), 6 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 5e5c2ae5f..d2d79b9d4 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -2920,8 +2920,9 @@ submodule.<name>.url::
 	The URL for a submodule. This variable is copied from the .gitmodules
 	file to the git config via 'git submodule init'. The user can change
 	the configured URL before obtaining the submodule via 'git submodule
-	update'. After obtaining the submodule, the presence of this variable
-	is used as a sign whether the submodule is of interest to git commands.
+	update'. If neither submodule.<name>.active or submodule.active are
+	set, the presence of this variable is used as a fallback to indicate
+	whether the submodule is of interest to git commands.
 	See linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.
 
 submodule.<name>.update::
@@ -2959,6 +2960,16 @@ submodule.<name>.ignore::
 	"--ignore-submodules" option. The 'git submodule' commands are not
 	affected by this setting.
 
+submodule.<name>.active::
+	Boolean value indicating if the submodule is of interest to git
+	commands.  This config option takes precedence over the
+	submodule.active config option.
+
+submodule.active::
+	A repeated field which contains a pathspec used to match against a
+	submodule's path to determine if the submodule is of interest to git
+	commands.
+
 submodule.fetchJobs::
 	Specifies how many submodules are fetched/cloned at the same time.
 	A positive integer allows up to that number of submodules fetched
diff --git a/submodule.c b/submodule.c
index 0a2831d84..2b33bd70f 100644
--- a/submodule.c
+++ b/submodule.c
@@ -217,13 +217,41 @@ void gitmodules_config_sha1(const unsigned char *commit_sha1)
 int is_submodule_initialized(const char *path)
 {
 	int ret = 0;
-	const struct submodule *module = NULL;
+	char *key;
+	const struct string_list *sl;
+	const struct submodule *module = submodule_from_path(null_sha1, path);
 
-	module = submodule_from_path(null_sha1, path);
+	/* early return if there isn't a path->module mapping */
+	if (!module)
+		return 0;
+
+	/* submodule.<name>.active is set */
+	key = xstrfmt("submodule.%s.active", module->name);
+	if (!git_config_get_bool(key, &ret)) {
+		free(key);
+		return ret;
+	}
+	free(key);
+
+	sl = git_config_get_value_multi("submodule.active");
 
-	if (module) {
-		char *key = xstrfmt("submodule.%s.url", module->name);
+	if (sl) {
+		struct pathspec ps;
+		struct argv_array args = ARGV_ARRAY_INIT;
+		const struct string_list_item *item;
+
+		for_each_string_list_item(item, sl) {
+			argv_array_push(&args, item->string);
+		}
+
+		parse_pathspec(&ps, 0, 0, 0, args.argv);
+		ret = match_pathspec(&ps, path, strlen(path), 0, NULL, 1);
+
+		argv_array_clear(&args);
+		clear_pathspec(&ps);
+	} else {
 		char *value = NULL;
+		key = xstrfmt("submodule.%s.url", module->name);
 
 		ret = !git_config_get_string(key, &value);
 
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
index f18e0c925..c41b899ab 100755
--- a/t/t7413-submodule-is-active.sh
+++ b/t/t7413-submodule-is-active.sh
@@ -28,4 +28,59 @@ test_expect_success 'is-active works with urls' '
 	git -C super submodule--helper is-active sub1
 '
 
+test_expect_success 'is-active works with submodule.<name>.active config' '
+	git -C super config --bool submodule.sub1.active "false" &&
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+
+	git -C super config --bool submodule.sub1.active "true" &&
+	git -C super config --unset submodule.sub1.URL &&
+	git -C super submodule--helper is-active sub1 &&
+
+	git -C super config submodule.sub1.URL ../sub &&
+	git -C super config --unset submodule.sub1.active
+'
+
+test_expect_success 'is-active works with basic submodule.active config' '
+	git -C super config --add submodule.active "." &&
+	git -C super config --unset submodule.sub1.URL &&
+
+	git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config submodule.sub1.URL ../sub &&
+	git -C super config --unset-all submodule.active
+'
+
+test_expect_success 'is-active correctly works with paths that are not submodules' '
+	test_must_fail git -C super submodule--helper is-active not-a-submodule &&
+
+	git -C super config --add submodule.active "." &&
+	test_must_fail git -C super submodule--helper is-active not-a-submodule &&
+
+	git -C super config --unset-all submodule.active
+'
+
+test_expect_success 'is-active works with exclusions in submodule.active config' '
+	git -C super config --add submodule.active "." &&
+	git -C super config --add submodule.active ":(exclude)sub1" &&
+
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config --unset-all submodule.active
+'
+
+test_expect_success 'is-active with submodule.active and submodule.<name>.active' '
+	git -C super config --add submodule.active "sub1" &&
+	git -C super config --bool submodule.sub1.active "false" &&
+	git -C super config --bool submodule.sub2.active "true" &&
+
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config --unset-all submodule.active &&
+	git -C super config --unset submodule.sub1.active &&
+	git -C super config --unset submodule.sub2.active
+'
+
 test_done
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v3 05/10] submodule: decouple url and submodule existence

[Stefan Beller]

+ cc Jens, FYI.

Once upon a time I brought up different addressing/activating mechanism for
submodules and I remember Jens having some uneasy thoughts about
that back in the day. This series addresses the user confusion and documentation
better than what I had back then.

On Mon, Mar 13, 2017 at 2:43 PM, Brandon Williams <bmwill@google.com> wrote:
> Currently the submodule.<name>.url config option is used to determine
> if a given submodule exists and is interesting to the user.  This
> however doesn't work very well because the URL is a config option for
> the scope of a repository, whereas the existence of a submodule is an
> option scoped to the working tree.
>
> In a future with worktree support for submodules, there will be multiple
> working trees, each of which may only need a subset of the submodules
> checked out.  The URL (which is where the submodule repository can be
> obtained) should not differ between different working trees.
>
> It may also be convenient for users to more easily specify groups of
> submodules they are interested in as apposed to running "git submodule
> init <path>" on each submodule they want checked out in their working
> tree.
>
> To this end two config options are introduced, submodule.active and
> submodule.<name>.active.  The submodule.active config holds a pathspec
> that specifies which submodules should exist in the working tree.  The
> submodule.<name>.active config is a boolean flag used to indicate if
> that particular submodule should exist in the working tree.
>
> Given these multiple ways to check for a submodule's existence the more
> fine-grained submodule.<name>.active option has the highest order of
> precedence followed by the pathspec check against submodule.active. To
> ensure backwards compatibility, if neither of these options are set git
> falls back to checking the submodule.<name>.url option to determine a
> submodule's existence.
>




>
> +submodule.<name>.active::
> +       Boolean value indicating if the submodule is of interest to git
> +       commands.  This config option takes precedence over the
> +       submodule.active config option.

... which itself takes precedence over the (deprecated) .URL
We conveniently do not talk about the URL here anymore.
But! We need to change submodule.<name>.URL now?

Re: [PATCH v3 05/10] submodule: decouple url and submodule existence

[Brandon Williams]

On 03/13, Stefan Beller wrote:
> + cc Jens, FYI.
> 
> Once upon a time I brought up different addressing/activating mechanism for
> submodules and I remember Jens having some uneasy thoughts about
> that back in the day. This series addresses the user confusion and documentation
> better than what I had back then.
> 
> On Mon, Mar 13, 2017 at 2:43 PM, Brandon Williams <bmwill@google.com> wrote:
> > Currently the submodule.<name>.url config option is used to determine
> > if a given submodule exists and is interesting to the user.  This
> > however doesn't work very well because the URL is a config option for
> > the scope of a repository, whereas the existence of a submodule is an
> > option scoped to the working tree.
> >
> > In a future with worktree support for submodules, there will be multiple
> > working trees, each of which may only need a subset of the submodules
> > checked out.  The URL (which is where the submodule repository can be
> > obtained) should not differ between different working trees.
> >
> > It may also be convenient for users to more easily specify groups of
> > submodules they are interested in as apposed to running "git submodule
> > init <path>" on each submodule they want checked out in their working
> > tree.
> >
> > To this end two config options are introduced, submodule.active and
> > submodule.<name>.active.  The submodule.active config holds a pathspec
> > that specifies which submodules should exist in the working tree.  The
> > submodule.<name>.active config is a boolean flag used to indicate if
> > that particular submodule should exist in the working tree.
> >
> > Given these multiple ways to check for a submodule's existence the more
> > fine-grained submodule.<name>.active option has the highest order of
> > precedence followed by the pathspec check against submodule.active. To
> > ensure backwards compatibility, if neither of these options are set git
> > falls back to checking the submodule.<name>.url option to determine a
> > submodule's existence.
> >
> 
> 
> 
> 
> >
> > +submodule.<name>.active::
> > +       Boolean value indicating if the submodule is of interest to git
> > +       commands.  This config option takes precedence over the
> > +       submodule.active config option.
> 
> ... which itself takes precedence over the (deprecated) .URL
> We conveniently do not talk about the URL here anymore.
> But! We need to change submodule.<name>.URL now?

yeah this patch introduces a change to the documentation for URL to
indicate the change.

-- 
Brandon Williams

Re: [PATCH v3 05/10] submodule: decouple url and submodule existence

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> Currently the submodule.<name>.url config option is used to determine
> if a given submodule exists and is interesting to the user.  This
> however doesn't work very well because the URL is a config option for
> the scope of a repository, whereas the existence of a submodule is an
> option scoped to the working tree.

A submodule exists if it is exists, whether the user is interested
in it or not.  If it should be checked out in the working tree is a
different matter, but that should be a logical AND between "is it of
interest?" and "is the superproject tree has a gitlink for it in its
working tree?".  

So I do not agree with "This however doesn't work" at all.  I'd
understand it if you said "This is cumbersome if we want to do this
and that, which are different from what we have done traditionally"
and explain what this and that are and how they are different.

> In a future with worktree support for submodules, there will be multiple
> working trees, each of which may only need a subset of the submodules
> checked out.  The URL (which is where the submodule repository can be
> obtained) should not differ between different working trees.

And this makes the motivation a bit clearer.  When the user wants to
have multiple worktrees for the same superproject.  In such a
setting, the same submodule in two worktrees typically want to have
the same URL.  It may be different from what the upstream suggests
in the .gitmodules file, but the difference, i.e. the site specific
customization of the URL, should be the same between the two
worktrees.  But one worktree may be and the other worktree may not be
interested in that submodule, and with shared .git/config file, you
cannot have submodule.<name>.url set to one value and unset at the
same time.

This series does not solve the "two worktrees cannot have private
parts in the configuration namespace" issue, but assuming it will be
solved by some other series, it anticipates that submodule.<name>.URL 
would want to be shared between two worktrees most of time (note that
there will be users who want two separate .URL for the same submodule
while sharing the object database via worktrees mechanism, and you'll
need to prepare for them, too), and another "bit" that tells if the
submodule is of interest would want to be private to each worktree.

That is the motivation, the reason why you want .URL to stop serving
the dual purpose of overriding upstream-suggested URL and indicating
the submodule is interesting to the user.

> It may also be convenient for users to more easily specify groups of
> submodules they are interested in as apposed to running "git submodule
> init <path>" on each submodule they want checked out in their working
> tree.
>
> To this end two config options are introduced, submodule.active and
> submodule.<name>.active.  The submodule.active config holds a pathspec
> that specifies which submodules should exist in the working tree.  The
> submodule.<name>.active config is a boolean flag used to indicate if
> that particular submodule should exist in the working tree.

And because two worktrees always share their .git/config, these new
configuration variables are useless to help workflow with multiple
worktrees with the current system, until "per-worktree configuration"
is invented.  But we prepare for that future in this step.

Also submodule.active that takes pathspec and not name is an oddball
(use of "name" not "path" is to prepare for a submodule whose
location in the superproject changes depending on the commit in the
superproject), and we need to justify with an explanation.  I think
you envision two cases.  1. we encourage projects to adopt a
convention that submodules are grouped with leading directory, so
that pathspec e.g. lib/, would cover _all_ library-ish modules to
allow those who are interested in library-ish modules to set
".active = lib/" just once to say any and all modules in lib/ are
interesting.  2. another convention the projects can adopt, when
pathspec-attribute feature is invented, is to label submodules with
attribute to group them, so that a broad pathspec with attribute
requirement, e.g. .:(attr:lib), can be used to say any and all
modules with 'lib' attribute are interesting.

The above two points (justifications, intended uses and future
plans) need to be clarified around here (and possibly in the
documentation), I would think.

> diff --git a/submodule.c b/submodule.c
> index 0a2831d84..2b33bd70f 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -217,13 +217,41 @@ void gitmodules_config_sha1(const unsigned char *commit_sha1)
>  int is_submodule_initialized(const char *path)
>  {
>  	int ret = 0;
> -	const struct submodule *module = NULL;
> +	char *key;
> +	const struct string_list *sl;
> +	const struct submodule *module = submodule_from_path(null_sha1, path);
>  
> -	module = submodule_from_path(null_sha1, path);
> +	/* early return if there isn't a path->module mapping */
> +	if (!module)
> +		return 0;
> +
> +	/* submodule.<name>.active is set */
> +	key = xstrfmt("submodule.%s.active", module->name);
> +	if (!git_config_get_bool(key, &ret)) {
> +		free(key);
> +		return ret;
> +	}
> +	free(key);
> +
> +	sl = git_config_get_value_multi("submodule.active");
>  
> -	if (module) {
> -		char *key = xstrfmt("submodule.%s.url", module->name);
> +	if (sl) {
> +		struct pathspec ps;
> +		struct argv_array args = ARGV_ARRAY_INIT;
> +		const struct string_list_item *item;
> +
> +		for_each_string_list_item(item, sl) {
> +			argv_array_push(&args, item->string);
> +		}
> +
> +		parse_pathspec(&ps, 0, 0, 0, args.argv);
> +		ret = match_pathspec(&ps, path, strlen(path), 0, NULL, 1);
> +
> +		argv_array_clear(&args);
> +		clear_pathspec(&ps);
> +	} else {
>  		char *value = NULL;
> +		key = xstrfmt("submodule.%s.url", module->name);
>  
>  		ret = !git_config_get_string(key, &value);

It probably is easier to read if you had a final "return ret" in the
"if (sl) {...}" part, just like you have one for the codepath that
deals with "submodule.<name>.active", and flatten the else clause.
That would make it clear that we have three ways with decreasing
precedence.

At this point, the answer from function is even less about "is it
initialized?"  but about "is it of interest?" (or "is it to be
initialized?").  We'd probably want a /* NEEDSWORK */ comment before
the function to remind us to come up with a better name after the
dust settles.

> diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
> index f18e0c925..c41b899ab 100755
> --- a/t/t7413-submodule-is-active.sh
> +++ b/t/t7413-submodule-is-active.sh
> @@ -28,4 +28,59 @@ test_expect_success 'is-active works with urls' '
>  	git -C super submodule--helper is-active sub1
>  '
>  
> +test_expect_success 'is-active works with submodule.<name>.active config' '
> +	git -C super config --bool submodule.sub1.active "false" &&
> +	test_must_fail git -C super submodule--helper is-active sub1 &&
> +
> +	git -C super config --bool submodule.sub1.active "true" &&
> +	git -C super config --unset submodule.sub1.URL &&
> +	git -C super submodule--helper is-active sub1 &&
> +
> +	git -C super config submodule.sub1.URL ../sub &&
> +	git -C super config --unset submodule.sub1.active
> +'

The last "unset" is done to clean the customization this test did,
in order to give a predictable beginning state to the next test?  If
so, use test_when_finished instead of &&-cascading it at the end.

> + ...
> +test_expect_success 'is-active with submodule.active and submodule.<name>.active' '
> +	git -C super config --add submodule.active "sub1" &&
> +	git -C super config --bool submodule.sub1.active "false" &&
> +	git -C super config --bool submodule.sub2.active "true" &&
> +
> +	test_must_fail git -C super submodule--helper is-active sub1 &&
> +	git -C super submodule--helper is-active sub2 &&
> +
> +	git -C super config --unset-all submodule.active &&
> +	git -C super config --unset submodule.sub1.active &&
> +	git -C super config --unset submodule.sub2.active
> +'

Likewise for all the new tests in this patch.

Thanks.

Re: [PATCH v3 05/10] submodule: decouple url and submodule existence

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Also submodule.active that takes pathspec and not name is an oddball
> (use of "name" not "path" is to prepare for a submodule whose
> location in the superproject changes depending on the commit in the
> superproject), and we need to justify with an explanation.  I think
> you envision two cases.  1. we encourage projects to adopt a
> convention that submodules are grouped with leading directory, so
> that pathspec e.g. lib/, would cover _all_ library-ish modules to
> allow those who are interested in library-ish modules to set
> ".active = lib/" just once to say any and all modules in lib/ are
> interesting.  2. another convention the projects can adopt, when
> pathspec-attribute feature is invented, is to label submodules with
> attribute to group them, so that a broad pathspec with attribute
> requirement, e.g. .:(attr:lib), can be used to say any and all
> modules with 'lib' attribute are interesting.

One addition to the point 2. above is that because .gitattributes,
just like .gitmodules, is tracked by the superproject, and when a
submodule that moves in the superproject tree, the project can
adjust which path gets the attribute in .gitattributes, just like it
can adjust which path has the submodule in .gitmodules.  That makes
a broad pathspec with attribute requirement just as a viable
solution to identify a submodule as having an unchanging "name" for
a submodule in the .gitmodules file.

> The above two points (justifications, intended uses and future
> plans) need to be clarified around here (and possibly in the
> documentation), I would think.

Re: [PATCH v3 05/10] submodule: decouple url and submodule existence

[Brandon Williams]

On 03/14, Junio C Hamano wrote:
> Brandon Williams <bmwill@google.com> writes:
> 
> > Currently the submodule.<name>.url config option is used to determine
> > if a given submodule exists and is interesting to the user.  This
> > however doesn't work very well because the URL is a config option for
> > the scope of a repository, whereas the existence of a submodule is an
> > option scoped to the working tree.
> 
> A submodule exists if it is exists, whether the user is interested
> in it or not.  If it should be checked out in the working tree is a
> different matter, but that should be a logical AND between "is it of
> interest?" and "is the superproject tree has a gitlink for it in its
> working tree?".  
> 
> So I do not agree with "This however doesn't work" at all.  I'd
> understand it if you said "This is cumbersome if we want to do this
> and that, which are different from what we have done traditionally"
> and explain what this and that are and how they are different.
> 
> > In a future with worktree support for submodules, there will be multiple
> > working trees, each of which may only need a subset of the submodules
> > checked out.  The URL (which is where the submodule repository can be
> > obtained) should not differ between different working trees.
> 
> And this makes the motivation a bit clearer.  When the user wants to
> have multiple worktrees for the same superproject.  In such a
> setting, the same submodule in two worktrees typically want to have
> the same URL.  It may be different from what the upstream suggests
> in the .gitmodules file, but the difference, i.e. the site specific
> customization of the URL, should be the same between the two
> worktrees.  But one worktree may be and the other worktree may not be
> interested in that submodule, and with shared .git/config file, you
> cannot have submodule.<name>.url set to one value and unset at the
> same time.
> 
> This series does not solve the "two worktrees cannot have private
> parts in the configuration namespace" issue, but assuming it will be
> solved by some other series, it anticipates that submodule.<name>.URL 
> would want to be shared between two worktrees most of time (note that
> there will be users who want two separate .URL for the same submodule
> while sharing the object database via worktrees mechanism, and you'll
> need to prepare for them, too), and another "bit" that tells if the
> submodule is of interest would want to be private to each worktree.
> 
> That is the motivation, the reason why you want .URL to stop serving
> the dual purpose of overriding upstream-suggested URL and indicating
> the submodule is interesting to the user.
> 
> > It may also be convenient for users to more easily specify groups of
> > submodules they are interested in as apposed to running "git submodule
> > init <path>" on each submodule they want checked out in their working
> > tree.
> >
> > To this end two config options are introduced, submodule.active and
> > submodule.<name>.active.  The submodule.active config holds a pathspec
> > that specifies which submodules should exist in the working tree.  The
> > submodule.<name>.active config is a boolean flag used to indicate if
> > that particular submodule should exist in the working tree.
> 
> And because two worktrees always share their .git/config, these new
> configuration variables are useless to help workflow with multiple
> worktrees with the current system, until "per-worktree configuration"
> is invented.  But we prepare for that future in this step.
> 
> Also submodule.active that takes pathspec and not name is an oddball
> (use of "name" not "path" is to prepare for a submodule whose
> location in the superproject changes depending on the commit in the
> superproject), and we need to justify with an explanation.  I think
> you envision two cases.  1. we encourage projects to adopt a
> convention that submodules are grouped with leading directory, so
> that pathspec e.g. lib/, would cover _all_ library-ish modules to
> allow those who are interested in library-ish modules to set
> ".active = lib/" just once to say any and all modules in lib/ are
> interesting.  2. another convention the projects can adopt, when
> pathspec-attribute feature is invented, is to label submodules with
> attribute to group them, so that a broad pathspec with attribute
> requirement, e.g. .:(attr:lib), can be used to say any and all
> modules with 'lib' attribute are interesting.
> 
> The above two points (justifications, intended uses and future
> plans) need to be clarified around here (and possibly in the
> documentation), I would think.

I'll overhaul this to better explain the intentions behind the change.
Thanks for pointing out what needs a more detailed explanation.

> 
> > diff --git a/submodule.c b/submodule.c
> > index 0a2831d84..2b33bd70f 100644
> > --- a/submodule.c
> > +++ b/submodule.c
> > @@ -217,13 +217,41 @@ void gitmodules_config_sha1(const unsigned char *commit_sha1)
> >  int is_submodule_initialized(const char *path)
> >  {
> >  	int ret = 0;
> > -	const struct submodule *module = NULL;
> > +	char *key;
> > +	const struct string_list *sl;
> > +	const struct submodule *module = submodule_from_path(null_sha1, path);
> >  
> > -	module = submodule_from_path(null_sha1, path);
> > +	/* early return if there isn't a path->module mapping */
> > +	if (!module)
> > +		return 0;
> > +
> > +	/* submodule.<name>.active is set */
> > +	key = xstrfmt("submodule.%s.active", module->name);
> > +	if (!git_config_get_bool(key, &ret)) {
> > +		free(key);
> > +		return ret;
> > +	}
> > +	free(key);
> > +
> > +	sl = git_config_get_value_multi("submodule.active");
> >  
> > -	if (module) {
> > -		char *key = xstrfmt("submodule.%s.url", module->name);
> > +	if (sl) {
> > +		struct pathspec ps;
> > +		struct argv_array args = ARGV_ARRAY_INIT;
> > +		const struct string_list_item *item;
> > +
> > +		for_each_string_list_item(item, sl) {
> > +			argv_array_push(&args, item->string);
> > +		}
> > +
> > +		parse_pathspec(&ps, 0, 0, 0, args.argv);
> > +		ret = match_pathspec(&ps, path, strlen(path), 0, NULL, 1);
> > +
> > +		argv_array_clear(&args);
> > +		clear_pathspec(&ps);
> > +	} else {
> >  		char *value = NULL;
> > +		key = xstrfmt("submodule.%s.url", module->name);
> >  
> >  		ret = !git_config_get_string(key, &value);
> 
> It probably is easier to read if you had a final "return ret" in the
> "if (sl) {...}" part, just like you have one for the codepath that
> deals with "submodule.<name>.active", and flatten the else clause.
> That would make it clear that we have three ways with decreasing
> precedence.

Sounds like a good change.  I'm all for making the code more readable.

> 
> At this point, the answer from function is even less about "is it
> initialized?"  but about "is it of interest?" (or "is it to be
> initialized?").  We'd probably want a /* NEEDSWORK */ comment before
> the function to remind us to come up with a better name after the
> dust settles.

Yeah I expect we should rename it to 'is_submodule_active' at some
point, or something along those lines.

> 
> > diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
> > index f18e0c925..c41b899ab 100755
> > --- a/t/t7413-submodule-is-active.sh
> > +++ b/t/t7413-submodule-is-active.sh
> > @@ -28,4 +28,59 @@ test_expect_success 'is-active works with urls' '
> >  	git -C super submodule--helper is-active sub1
> >  '
> >  
> > +test_expect_success 'is-active works with submodule.<name>.active config' '
> > +	git -C super config --bool submodule.sub1.active "false" &&
> > +	test_must_fail git -C super submodule--helper is-active sub1 &&
> > +
> > +	git -C super config --bool submodule.sub1.active "true" &&
> > +	git -C super config --unset submodule.sub1.URL &&
> > +	git -C super submodule--helper is-active sub1 &&
> > +
> > +	git -C super config submodule.sub1.URL ../sub &&
> > +	git -C super config --unset submodule.sub1.active
> > +'
> 
> The last "unset" is done to clean the customization this test did,
> in order to give a predictable beginning state to the next test?  If
> so, use test_when_finished instead of &&-cascading it at the end.

Will do.

> 
> > + ...
> > +test_expect_success 'is-active with submodule.active and submodule.<name>.active' '
> > +	git -C super config --add submodule.active "sub1" &&
> > +	git -C super config --bool submodule.sub1.active "false" &&
> > +	git -C super config --bool submodule.sub2.active "true" &&
> > +
> > +	test_must_fail git -C super submodule--helper is-active sub1 &&
> > +	git -C super submodule--helper is-active sub2 &&
> > +
> > +	git -C super config --unset-all submodule.active &&
> > +	git -C super config --unset submodule.sub1.active &&
> > +	git -C super config --unset submodule.sub2.active
> > +'
> 
> Likewise for all the new tests in this patch.
> 
> Thanks.

-- 
Brandon Williams

[PATCH v3 06/10] submodule update: add `--init-active` switch

[Brandon Williams]

The new switch `--init-active` initializes the submodules which are
configured in `submodule.active` and `submodule.<name>.active` instead
of those given as command line arguments before updating. In the first
implementation this is made incompatible with further command line
arguments as it is unclear what the user means by

    git submodule update --init --init-active <paths>

This new switch allows users to record more complex patterns as it saves
retyping them whenever you invoke update.

Based on a patch by Stefan Beller <sbeller@google.com>
Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/git-submodule.txt |  9 +++++++
 builtin/submodule--helper.c     | 35 ++++++++++++++++++++++++-
 git-submodule.sh                | 26 +++++++++++++++---
 t/t7400-submodule-basic.sh      | 58 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 123 insertions(+), 5 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index e05d0cdde..6b17cd707 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -198,6 +198,10 @@ If the submodule is not yet initialized, and you just want to use the
 setting as stored in .gitmodules, you can automatically initialize the
 submodule with the `--init` option.
 
+You can configure a set of submodules using pathspec syntax in
+submodule.active you can use `--init-active` to initialize
+those before updating.
+
 If `--recursive` is specified, this command will recurse into the
 registered submodules, and update any nested submodules within.
 --
@@ -384,6 +388,11 @@ the submodule itself.
 	Initialize all submodules for which "git submodule init" has not been
 	called so far before updating.
 
+--init-active::
+	This option is only valid for the update command.
+	Initialize all submodules configured in "`submodule.active`"
+	that have not been updated before.
+
 --name::
 	This option is only valid for the add command. It sets the submodule's
 	name to the given string instead of defaulting to its path. The name
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index f38e332c5..a3acc9e4c 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -270,6 +270,34 @@ static int module_list_compute(int argc, const char **argv,
 	return result;
 }
 
+static void module_list_active(struct module_list *list)
+{
+	int i;
+
+	if (read_cache() < 0)
+		die(_("index file corrupt"));
+
+	gitmodules_config();
+
+	for (i = 0; i < active_nr; i++) {
+		const struct cache_entry *ce = active_cache[i];
+
+		if (!S_ISGITLINK(ce->ce_mode) ||
+		    !is_submodule_initialized(ce->name))
+			continue;
+
+		ALLOC_GROW(list->entries, list->nr + 1, list->alloc);
+		list->entries[list->nr++] = ce;
+		while (i + 1 < active_nr &&
+		       !strcmp(ce->name, active_cache[i + 1]->name))
+			/*
+			 * Skip entries with the same name in different stages
+			 * to make sure an entry is returned only once.
+			 */
+			i++;
+	}
+}
+
 static int module_list(int argc, const char **argv, const char *prefix)
 {
 	int i;
@@ -402,9 +430,12 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	struct pathspec pathspec;
 	struct module_list list = MODULE_LIST_INIT;
 	int quiet = 0;
+	int active = 0;
 	int i;
 
 	struct option module_init_options[] = {
+		OPT_BOOL(0, "active", &active,
+			 N_("ensure all active submodules are properly initialized")),
 		OPT__QUIET(&quiet, N_("Suppress output for initializing a submodule")),
 		OPT_END()
 	};
@@ -417,7 +448,9 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix, module_init_options,
 			     git_submodule_helper_usage, 0);
 
-	if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
+	if (active)
+		module_list_active(&list);
+	else if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
 		return 1;
 
 	for (i = 0; i < list.nr; i++)
diff --git a/git-submodule.sh b/git-submodule.sh
index e2d08595f..3c7da08aa 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -9,7 +9,7 @@ USAGE="[--quiet] add [-b <branch>] [-f|--force] [--name <name>] [--reference <re
    or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]
    or: $dashless [--quiet] init [--] [<path>...]
    or: $dashless [--quiet] deinit [-f|--force] (--all| [--] <path>...)
-   or: $dashless [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference <repository>] [--recursive] [--] [<path>...]
+   or: $dashless [--quiet] update [--init[-active]] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference <repository>] [--recursive] [--] [<path>...]
    or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
    or: $dashless [--quiet] foreach [--recursive] <command>
    or: $dashless [--quiet] sync [--recursive] [--] [<path>...]
@@ -363,6 +363,9 @@ cmd_init()
 		-q|--quiet)
 			GIT_QUIET=1
 			;;
+		--active)
+			GIT_ACTIVE=1
+			;;
 		--)
 			shift
 			break
@@ -377,7 +380,7 @@ cmd_init()
 		shift
 	done
 
-	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper init ${GIT_QUIET:+--quiet}  "$@"
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper init ${GIT_QUIET:+--quiet} ${GIT_ACTIVE:+--active} "$@"
 }
 
 #
@@ -506,7 +509,12 @@ cmd_update()
 			progress="--progress"
 			;;
 		-i|--init)
-			init=1
+			test -z $init || test $init = by_args || die "$(gettext "Only one of --init or --init-active may be used.")"
+			init=by_args
+			;;
+		--init-active)
+			test -z $init || test $init = by_config || die "$(gettext "Only one of --init or --init-active may be used.")"
+			init=by_config
 			;;
 		--remote)
 			remote=1
@@ -575,7 +583,17 @@ cmd_update()
 
 	if test -n "$init"
 	then
-		cmd_init "--" "$@" || return
+		if test "$init" = "by_config"
+		then
+			if test $# -gt 0
+			then
+				die "$(gettext "path arguments are incompatible with --init-active")"
+			fi
+			cmd_init "--active" || return
+		else
+			cmd_init "--" "$@" || return
+		fi
+
 	fi
 
 	{
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index c09ce0d4c..cc348b807 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1130,5 +1130,63 @@ test_expect_success 'submodule helper list is not confused by common prefixes' '
 	test_cmp expect actual
 '
 
+test_expect_success 'setup superproject with submodules' '
+	git init sub1 &&
+	test_commit -C sub1 test &&
+	test_commit -C sub1 test2 &&
+	git init multisuper &&
+	git -C multisuper submodule add ../sub1 sub0 &&
+	git -C multisuper submodule add ../sub1 sub1 &&
+	git -C multisuper submodule add ../sub1 sub2 &&
+	git -C multisuper submodule add ../sub1 sub3 &&
+	git -C multisuper commit -m "add some submodules"
+'
+
+cat >expect <<-EOF
+-sub0
+ sub1 (test2)
+ sub2 (test2)
+ sub3 (test2)
+EOF
+
+test_expect_success 'submodule update --init with a specification' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone submodule update --init . ":(exclude)sub0" &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'submodule update --init-active' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone config submodule.active "." &&
+	git -C multisuper_clone config --add submodule.active ":(exclude)sub0" &&
+	git -C multisuper_clone submodule update --init-active &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_must_fail git -C multisuper_clone submodule update --init-active sub0 &&
+	test_cmp expect actual
+'
+
+test_expect_success 'submodule init --active and update' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone config --bool submodule.sub0.active "true" &&
+	git -C multisuper_clone config --bool submodule.sub1.active "false" &&
+	git -C multisuper_clone config --bool submodule.sub2.active "true" &&
+
+	cat >expect <<-\EOF &&
+	 sub0 (test2)
+	-sub1
+	 sub2 (test2)
+	-sub3
+	EOF
+	git -C multisuper_clone submodule update &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_cmp expect actual
+'
 
 test_done
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v3 06/10] submodule update: add `--init-active` switch

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> +--init-active::
> +	This option is only valid for the update command.
> +	Initialize all submodules configured in "`submodule.active`"
> +	that have not been updated before.

You mean s/not been updated/not been initialized/ here?

> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index f38e332c5..a3acc9e4c 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -270,6 +270,34 @@ static int module_list_compute(int argc, const char **argv,
>  	return result;
>  }
>  
> +static void module_list_active(struct module_list *list)
> +{
> +	int i;
> +
> +	if (read_cache() < 0)
> +		die(_("index file corrupt"));
> +
> +	gitmodules_config();

To this codepath, it probably would not make any difference, but in
general, calling gitmodules_config() after you did git_config()
would be a bug, right?  Otherwise, gitmodules_config() would
overwrite submodule.* configuration you read with git_config().

I have a feeling that we need to make this function hard to misuse;
it is not a fault of _this_ patch, but still any and all additions
to any of the submodule*.c files needs careful vetting.

Re: [PATCH v3 06/10] submodule update: add `--init-active` switch

[Brandon Williams]

On 03/14, Junio C Hamano wrote:
> Brandon Williams <bmwill@google.com> writes:
> 
> > +--init-active::
> > +	This option is only valid for the update command.
> > +	Initialize all submodules configured in "`submodule.active`"
> > +	that have not been updated before.
> 
> You mean s/not been updated/not been initialized/ here?

Yep you're right, the documentation is a little stale too so I'll update
it.

> 
> > diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> > index f38e332c5..a3acc9e4c 100644
> > --- a/builtin/submodule--helper.c
> > +++ b/builtin/submodule--helper.c
> > @@ -270,6 +270,34 @@ static int module_list_compute(int argc, const char **argv,
> >  	return result;
> >  }
> >  
> > +static void module_list_active(struct module_list *list)
> > +{
> > +	int i;
> > +
> > +	if (read_cache() < 0)
> > +		die(_("index file corrupt"));
> > +
> > +	gitmodules_config();
> 
> To this codepath, it probably would not make any difference, but in
> general, calling gitmodules_config() after you did git_config()
> would be a bug, right?  Otherwise, gitmodules_config() would
> overwrite submodule.* configuration you read with git_config().

gitmodules_config() doesn't interact with the git config, unless you do
git_config(submodule_confi, NULL) to overlay the two.  But yes it
doesn't effect this codepath.

> I have a feeling that we need to make this function hard to misuse;
> it is not a fault of _this_ patch, but still any and all additions
> to any of the submodule*.c files needs careful vetting.

-- 
Brandon Williams

Re: [PATCH v3 06/10] submodule update: add `--init-active` switch

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> +static void module_list_active(struct module_list *list)
> +{
> +	int i;
> +
> +	if (read_cache() < 0)
> +		die(_("index file corrupt"));
> +
> +	gitmodules_config();
> +
> +	for (i = 0; i < active_nr; i++) {
> +		const struct cache_entry *ce = active_cache[i];
> +
> +		if (!S_ISGITLINK(ce->ce_mode) ||
> +		    !is_submodule_initialized(ce->name))
> +			continue;

This, because "is_submodule_initialized()" is not "is it
initialized?" but "is it interesting?", will catch a submodule
that used to be uninteresting but has become interesting (e.g. a new
submodule added to somewhere in "lib/" when submodule.active lists
"lib/" as one of the pathspecs to match interesting submodules) and
automatically updates it.

I anticipate that a corresponding "please deinitialize ones that are
no longer interesting" coming in the series ;-)  Without it, the
interested set will only grow without shrinking.

> +		ALLOC_GROW(list->entries, list->nr + 1, list->alloc);
> +		list->entries[list->nr++] = ce;
> +		while (i + 1 < active_nr &&
> +		       !strcmp(ce->name, active_cache[i + 1]->name))
> +			/*
> +			 * Skip entries with the same name in different stages
> +			 * to make sure an entry is returned only once.
> +			 */
> +			i++;
> +	}
> +}

Re: [PATCH v3 06/10] submodule update: add `--init-active` switch

[Brandon Williams]

On 03/14, Junio C Hamano wrote:
> Brandon Williams <bmwill@google.com> writes:
> 
> > +static void module_list_active(struct module_list *list)
> > +{
> > +	int i;
> > +
> > +	if (read_cache() < 0)
> > +		die(_("index file corrupt"));
> > +
> > +	gitmodules_config();
> > +
> > +	for (i = 0; i < active_nr; i++) {
> > +		const struct cache_entry *ce = active_cache[i];
> > +
> > +		if (!S_ISGITLINK(ce->ce_mode) ||
> > +		    !is_submodule_initialized(ce->name))
> > +			continue;
> 
> This, because "is_submodule_initialized()" is not "is it
> initialized?" but "is it interesting?", will catch a submodule
> that used to be uninteresting but has become interesting (e.g. a new
> submodule added to somewhere in "lib/" when submodule.active lists
> "lib/" as one of the pathspecs to match interesting submodules) and
> automatically updates it.

Yes, is_submodule_initialized should be renamed to something more
descriptive (is_submodule_active, or something) in a follow on series
because it is doing a check to see which submodule's the user finds
interesting.

-- 
Brandon Williams

[PATCH v3 07/10] clone: add --submodule-spec=<pathspec> switch

[Brandon Williams]

The new switch passes the pathspec to `git submodule update
--init-active` which is called after the actual clone is done.

Additionally this configures the submodule.active option to
be the given pathspec, such that any future invocation of
`git submodule update --init-active` will keep up
with the pathspec.

Based on a patch by Stefan Beller <sbeller@google.com>

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/git-clone.txt | 23 ++++++++++-----
 builtin/clone.c             | 36 +++++++++++++++++++++--
 t/t7400-submodule-basic.sh  | 70 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 120 insertions(+), 9 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 35cc34b2f..9692eab30 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -15,7 +15,8 @@ SYNOPSIS
 	  [--dissociate] [--separate-git-dir <git dir>]
 	  [--depth <depth>] [--[no-]single-branch]
 	  [--recursive | --recurse-submodules] [--[no-]shallow-submodules]
-	  [--jobs <n>] [--] <repository> [<directory>]
+	  [--submodule-spec <pathspec>] [--jobs <n>] [--]
+	  <repository> [<directory>]
 
 DESCRIPTION
 -----------
@@ -217,12 +218,20 @@ objects from the source repository into a pack in the cloned repository.
 
 --recursive::
 --recurse-submodules::
-	After the clone is created, initialize all submodules within,
-	using their default settings. This is equivalent to running
-	`git submodule update --init --recursive` immediately after
-	the clone is finished. This option is ignored if the cloned
-	repository does not have a worktree/checkout (i.e. if any of
-	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
+	After the clone is created, initialize and clone all submodules
+	within, using their default settings. This is equivalent to
+	running `git submodule update --recursive --init` immediately
+	after the clone is finished. This option is ignored if the
+	cloned repository does not have a worktree/checkout (i.e.  if
+	any of `--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
+
+--submodule-spec::
+	After the clone is created, initialize and clone specified
+	submodules within, using their default settings. It is possible
+	to give multiple specifications by giving this argument multiple
+	times. This is equivalent to configuring `submodule.active`
+	and then running `git submodule update --init-active`
+	immediately after the clone is finished.
 
 --[no-]shallow-submodules::
 	All submodules which are cloned will be shallow with a depth of 1.
diff --git a/builtin/clone.c b/builtin/clone.c
index 3f63edbbf..c6731379b 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -56,6 +56,16 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;
 static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;
 static int option_dissociate;
 static int max_jobs = -1;
+static struct string_list submodule_spec;
+
+static int submodule_spec_cb(const struct option *opt, const char *arg, int unset)
+{
+	if (unset)
+		return -1;
+
+	string_list_append((struct string_list *)opt->value, arg);
+	return 0;
+}
 
 static struct option builtin_clone_options[] = {
 	OPT__VERBOSITY(&option_verbosity),
@@ -112,6 +122,9 @@ static struct option builtin_clone_options[] = {
 			TRANSPORT_FAMILY_IPV4),
 	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
 			TRANSPORT_FAMILY_IPV6),
+	OPT_CALLBACK(0, "submodule-spec", &submodule_spec, N_("<pathspec>"),
+			N_("clone specific submodules. Pass multiple times for complex pathspecs"),
+			submodule_spec_cb),
 	OPT_END()
 };
 
@@ -733,13 +746,21 @@ static int checkout(int submodule_progress)
 	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
 			   sha1_to_hex(sha1), "1", NULL);
 
-	if (!err && option_recursive) {
+	if (!err && (option_recursive || submodule_spec.nr > 0)) {
 		struct argv_array args = ARGV_ARRAY_INIT;
-		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
+		argv_array_pushl(&args, "submodule", "update", NULL);
+
+		if (submodule_spec.nr > 0)
+			argv_array_pushf(&args, "--init-active");
+		else
+			argv_array_pushf(&args, "--init");
 
 		if (option_shallow_submodules == 1)
 			argv_array_push(&args, "--depth=1");
 
+		if (option_recursive)
+			argv_array_pushf(&args, "--recursive");
+
 		if (max_jobs != -1)
 			argv_array_pushf(&args, "--jobs=%d", max_jobs);
 
@@ -887,6 +908,17 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		option_no_checkout = 1;
 	}
 
+	if (submodule_spec.nr > 0) {
+		struct string_list_item *item;
+		struct strbuf sb = STRBUF_INIT;
+		for_each_string_list_item(item, &submodule_spec) {
+			strbuf_addf(&sb, "submodule.active=%s",
+				    item->string);
+			string_list_append(&option_config,
+					   strbuf_detach(&sb, NULL));
+		}
+	}
+
 	if (!option_origin)
 		option_origin = "origin";
 
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index cc348b807..c2e198a92 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1189,4 +1189,74 @@ test_expect_success 'submodule init --active and update' '
 	test_cmp expect actual
 '
 
+test_expect_success 'clone --submodule-spec works' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat >expected <<-\EOF &&
+	 sub0 (test2)
+	-sub1
+	-sub2
+	-sub3
+	EOF
+
+	git clone --recurse-submodules --submodule-spec="sub0" multisuper multisuper_clone &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp actual expected
+'
+
+test_expect_success 'clone with multiple --submodule-spec options' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat >expect <<-\EOF &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	EOF
+
+	git clone --recurse-submodules \
+		  --submodule-spec="." \
+		  --submodule-spec ":(exclude)sub0" \
+		  --submodule-spec ":(exclude)sub2" \
+		  multisuper multisuper_clone &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'clone and subsequent updates correctly auto-initialize submodules' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat <<-\EOF >expect &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	EOF
+
+	cat <<-\EOF >expect2 &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	-sub4
+	 sub5 (test2)
+	EOF
+
+	git clone --recurse-submodules \
+		  --submodule-spec="." \
+		  --submodule-spec ":(exclude)sub0" \
+		  --submodule-spec ":(exclude)sub2" \
+		  --submodule-spec ":(exclude)sub4" \
+		  multisuper multisuper_clone &&
+
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect actual &&
+
+	git -C multisuper submodule add ../sub1 sub4 &&
+	git -C multisuper submodule add ../sub1 sub5 &&
+	git -C multisuper commit -m "add more submodules" &&
+	# obtain the new superproject
+	git -C multisuper_clone pull &&
+	git -C multisuper_clone submodule update --init-active &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect2 actual
+'
+
 test_done
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v3 07/10] clone: add --submodule-spec=<pathspec> switch

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> The new switch passes the pathspec to `git submodule update
> --init-active` which is called after the actual clone is done.
>
> Additionally this configures the submodule.active option to
> be the given pathspec, such that any future invocation of
> `git submodule update --init-active` will keep up
> with the pathspec.
>
> Based on a patch by Stefan Beller <sbeller@google.com>
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  Documentation/git-clone.txt | 23 ++++++++++-----
>  builtin/clone.c             | 36 +++++++++++++++++++++--
>  t/t7400-submodule-basic.sh  | 70 +++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 120 insertions(+), 9 deletions(-)
>
> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
> index 35cc34b2f..9692eab30 100644
> --- a/Documentation/git-clone.txt
> +++ b/Documentation/git-clone.txt
> @@ -15,7 +15,8 @@ SYNOPSIS
>  	  [--dissociate] [--separate-git-dir <git dir>]
>  	  [--depth <depth>] [--[no-]single-branch]
>  	  [--recursive | --recurse-submodules] [--[no-]shallow-submodules]
> -	  [--jobs <n>] [--] <repository> [<directory>]
> +	  [--submodule-spec <pathspec>] [--jobs <n>] [--]
> +	  <repository> [<directory>]

Hmph.  Can we then make "--recurse-submodules" an obsolete way to
spell "--submodule-spec ."?  I am not actively suggesting to
deprecate it; I am trying to see if there are semantic differences
between the two.

I am also wondering "--recurse-submodules=<pathspec>" would be a
better UI, instead of introducing yet another option.

> @@ -217,12 +218,20 @@ objects from the source repository into a pack in the cloned repository.
>  
>  --recursive::
>  --recurse-submodules::
> -	After the clone is created, initialize all submodules within,
> -	using their default settings. This is equivalent to running
> -	`git submodule update --init --recursive` immediately after
> -	the clone is finished. This option is ignored if the cloned
> -	repository does not have a worktree/checkout (i.e. if any of
> -	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
> +	After the clone is created, initialize and clone all submodules
> +	within, using their default settings. This is equivalent to
> +	running `git submodule update --recursive --init` immediately
> +	after the clone is finished. This option is ignored if the
> +	cloned repository does not have a worktree/checkout (i.e.  if
> +	any of `--no-checkout`/`-n`, `--bare`, or `--mirror` is given)

With reflowing it is unnecessarily harder to spot what got changed.
"and clone" is inserted, "--init" and "--recursive" were swapped.
Any other changes?

> diff --git a/builtin/clone.c b/builtin/clone.c
> index 3f63edbbf..c6731379b 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -56,6 +56,16 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;
>  static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;
>  static int option_dissociate;
>  static int max_jobs = -1;
> +static struct string_list submodule_spec;
> +
> +static int submodule_spec_cb(const struct option *opt, const char *arg, int unset)
> +{
> +	if (unset)
> +		return -1;
> +
> +	string_list_append((struct string_list *)opt->value, arg);
> +	return 0;
> +}

Hmph,  doesn't OPT_STRING_LIST work for this thing?

> +	if (submodule_spec.nr > 0) {
> +		struct string_list_item *item;
> +		struct strbuf sb = STRBUF_INIT;
> +		for_each_string_list_item(item, &submodule_spec) {
> +			strbuf_addf(&sb, "submodule.active=%s",
> +				    item->string);
> +			string_list_append(&option_config,
> +					   strbuf_detach(&sb, NULL));
> +		}
> +	}

OK.  Each pathspec becomes submodule.active in the newly created
repository.

Re: [PATCH v3 07/10] clone: add --submodule-spec=<pathspec> switch

[Brandon Williams]

On 03/14, Junio C Hamano wrote:
> Brandon Williams <bmwill@google.com> writes:
> 
> > The new switch passes the pathspec to `git submodule update
> > --init-active` which is called after the actual clone is done.
> >
> > Additionally this configures the submodule.active option to
> > be the given pathspec, such that any future invocation of
> > `git submodule update --init-active` will keep up
> > with the pathspec.
> >
> > Based on a patch by Stefan Beller <sbeller@google.com>
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> >  Documentation/git-clone.txt | 23 ++++++++++-----
> >  builtin/clone.c             | 36 +++++++++++++++++++++--
> >  t/t7400-submodule-basic.sh  | 70 +++++++++++++++++++++++++++++++++++++++++++++
> >  3 files changed, 120 insertions(+), 9 deletions(-)
> >
> > diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
> > index 35cc34b2f..9692eab30 100644
> > --- a/Documentation/git-clone.txt
> > +++ b/Documentation/git-clone.txt
> > @@ -15,7 +15,8 @@ SYNOPSIS
> >  	  [--dissociate] [--separate-git-dir <git dir>]
> >  	  [--depth <depth>] [--[no-]single-branch]
> >  	  [--recursive | --recurse-submodules] [--[no-]shallow-submodules]
> > -	  [--jobs <n>] [--] <repository> [<directory>]
> > +	  [--submodule-spec <pathspec>] [--jobs <n>] [--]
> > +	  <repository> [<directory>]
> 
> Hmph.  Can we then make "--recurse-submodules" an obsolete way to
> spell "--submodule-spec ."?  I am not actively suggesting to
> deprecate it; I am trying to see if there are semantic differences
> between the two.

We can if you think that would be better.  That way if at clone time you
say "I want the submodules too" that your default config tracks all
submodules even new ones yet to be added.

> 
> I am also wondering "--recurse-submodules=<pathspec>" would be a
> better UI, instead of introducing yet another option.

Yeah we could do that, have --recurse-submodules be a repeated option
and if you don't specify a value it defaults to "."

Any thoughts on this Stefan?

> 
> > @@ -217,12 +218,20 @@ objects from the source repository into a pack in the cloned repository.
> >  
> >  --recursive::
> >  --recurse-submodules::
> > -	After the clone is created, initialize all submodules within,
> > -	using their default settings. This is equivalent to running
> > -	`git submodule update --init --recursive` immediately after
> > -	the clone is finished. This option is ignored if the cloned
> > -	repository does not have a worktree/checkout (i.e. if any of
> > -	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
> > +	After the clone is created, initialize and clone all submodules
> > +	within, using their default settings. This is equivalent to
> > +	running `git submodule update --recursive --init` immediately
> > +	after the clone is finished. This option is ignored if the
> > +	cloned repository does not have a worktree/checkout (i.e.  if
> > +	any of `--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
> 
> With reflowing it is unnecessarily harder to spot what got changed.
> "and clone" is inserted, "--init" and "--recursive" were swapped.
> Any other changes?

No other changes, it just reads a little bit clearer now IMO.

> 
> > diff --git a/builtin/clone.c b/builtin/clone.c
> > index 3f63edbbf..c6731379b 100644
> > --- a/builtin/clone.c
> > +++ b/builtin/clone.c
> > @@ -56,6 +56,16 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;
> >  static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;
> >  static int option_dissociate;
> >  static int max_jobs = -1;
> > +static struct string_list submodule_spec;
> > +
> > +static int submodule_spec_cb(const struct option *opt, const char *arg, int unset)
> > +{
> > +	if (unset)
> > +		return -1;
> > +
> > +	string_list_append((struct string_list *)opt->value, arg);
> > +	return 0;
> > +}
> 
> Hmph,  doesn't OPT_STRING_LIST work for this thing?

You're right, I'll change to that.

-- 
Brandon Williams

Re: [PATCH v3 07/10] clone: add --submodule-spec=<pathspec> switch

[Stefan Beller]

On Wed, Mar 15, 2017 at 4:08 PM, Brandon Williams <bmwill@google.com> wrote:
> On 03/14, Junio C Hamano wrote:
>> Brandon Williams <bmwill@google.com> writes:
>>
>> > The new switch passes the pathspec to `git submodule update
>> > --init-active` which is called after the actual clone is done.
>> >
>> > Additionally this configures the submodule.active option to
>> > be the given pathspec, such that any future invocation of
>> > `git submodule update --init-active` will keep up
>> > with the pathspec.
>> >
>> > Based on a patch by Stefan Beller <sbeller@google.com>
>> >
>> > Signed-off-by: Brandon Williams <bmwill@google.com>
>> > ---
>> >  Documentation/git-clone.txt | 23 ++++++++++-----
>> >  builtin/clone.c             | 36 +++++++++++++++++++++--
>> >  t/t7400-submodule-basic.sh  | 70 +++++++++++++++++++++++++++++++++++++++++++++
>> >  3 files changed, 120 insertions(+), 9 deletions(-)
>> >
>> > diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
>> > index 35cc34b2f..9692eab30 100644
>> > --- a/Documentation/git-clone.txt
>> > +++ b/Documentation/git-clone.txt
>> > @@ -15,7 +15,8 @@ SYNOPSIS
>> >       [--dissociate] [--separate-git-dir <git dir>]
>> >       [--depth <depth>] [--[no-]single-branch]
>> >       [--recursive | --recurse-submodules] [--[no-]shallow-submodules]
>> > -     [--jobs <n>] [--] <repository> [<directory>]
>> > +     [--submodule-spec <pathspec>] [--jobs <n>] [--]
>> > +     <repository> [<directory>]
>>
>> Hmph.  Can we then make "--recurse-submodules" an obsolete way to
>> spell "--submodule-spec ."?  I am not actively suggesting to
>> deprecate it; I am trying to see if there are semantic differences
>> between the two.
>
> We can if you think that would be better.  That way if at clone time you
> say "I want the submodules too" that your default config tracks all
> submodules even new ones yet to be added.
>
>>
>> I am also wondering "--recurse-submodules=<pathspec>" would be a
>> better UI, instead of introducing yet another option.
>
> Yeah we could do that, have --recurse-submodules be a repeated option
> and if you don't specify a value it defaults to "."
>
> Any thoughts on this Stefan?

I think when I first tried to tackle this problem, I realized that the recursing
beyond the first level is orthogonal to the selection of submodules in the
first level.

Consider the following submodules

    /sub1
    /sub1/nestedA
    /sub2
    /sub2/nestedB

To select sub1
  git clone --submodule-spec=sub1

To select sub1 +nestedA
  git clone --submodule-spec=sub1 --recurse-submodules

To select sub1+sub2
  git clone --submodule-spec=. --no-recurse-submodules

How to select sub1+nestedA+sub2, but not nestedB
  not possible in one command

I wonder if we want to be able to differentiate these cases at all,
I guess if you want sub1, you also care about nestedB ?

(Or is there any way to recurse pathspecs,
which could be gitattributes)

That said, I optimized for the complex case, which we might not have
and a combination into one switch makes sense, such that
  git clone --recurse-submodules=<pathspec>
is sufficient and turns on recursion beyond the first level.

Thanks,
Stefan

[PATCH v3 08/10] completion: clone can initialize specific submodules

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 contrib/completion/git-completion.bash | 1 +
 1 file changed, 1 insertion(+)

diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index fc32286a4..eb13433d5 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -1212,6 +1212,7 @@ _git_clone ()
 			--recurse-submodules
 			--no-single-branch
 			--shallow-submodules
+			--submodule-spec
 			"
 		return
 		;;
-- 
2.12.0.246.ga2ecc84866-goog

[PATCH v3 09/10] submodule--helper init: set submodule.<name>.active

[Brandon Williams]

When initializing a submodule set the submodule.<name>.active config to
true to indicate that the submodule is active.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index a3acc9e4c..b669ed031 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -361,6 +361,13 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 		die(_("No url found for submodule path '%s' in .gitmodules"),
 			displaypath);
 
+	/* Set active flag for the submodule being initialized */
+	if (!is_submodule_initialized(path)) {
+		strbuf_reset(&sb);
+		strbuf_addf(&sb, "submodule.%s.active", sub->name);
+		git_config_set_gently(sb.buf, "true");
+	}
+
 	/*
 	 * Copy url setting when it is not set yet.
 	 * To look up the url in .git/config, we must not fall back to
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v3 09/10] submodule--helper init: set submodule.<name>.active

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> When initializing a submodule set the submodule.<name>.active config to
> true to indicate that the submodule is active.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---

Hmph.  When you do

	git clone --submodule-spec=lib/

and resulting repository says "submodule.active = lib/", you are
saying that you are interested in anything in "lib/", now or in the
future--that is the point of 08/10.

It is unclear what your desire would be for a submodule that was
bound to somewhere in "lib/" in the superproject when you cloned
when it later gets moved outside "lib/".  With changes up to 08/10,
that submodule will no longer be interesting, but with this 09/10,
when recursing clone will activate it upon the initial clone, the
submodule is explicitly marked as active by its name, so no matter
where it gets moved later, it will remain to be interesting.

I am not sure if that is a desired behaviour.

>  builtin/submodule--helper.c | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index a3acc9e4c..b669ed031 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -361,6 +361,13 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
>  		die(_("No url found for submodule path '%s' in .gitmodules"),
>  			displaypath);
>  
> +	/* Set active flag for the submodule being initialized */
> +	if (!is_submodule_initialized(path)) {
> +		strbuf_reset(&sb);
> +		strbuf_addf(&sb, "submodule.%s.active", sub->name);
> +		git_config_set_gently(sb.buf, "true");
> +	}
> +
>  	/*
>  	 * Copy url setting when it is not set yet.
>  	 * To look up the url in .git/config, we must not fall back to

Re: [PATCH v3 09/10] submodule--helper init: set submodule.<name>.active

[Brandon Williams]

On 03/14, Junio C Hamano wrote:
> Brandon Williams <bmwill@google.com> writes:
> 
> > When initializing a submodule set the submodule.<name>.active config to
> > true to indicate that the submodule is active.
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> 
> Hmph.  When you do
> 
> 	git clone --submodule-spec=lib/
> 
> and resulting repository says "submodule.active = lib/", you are
> saying that you are interested in anything in "lib/", now or in the
> future--that is the point of 08/10.
> 
> It is unclear what your desire would be for a submodule that was
> bound to somewhere in "lib/" in the superproject when you cloned
> when it later gets moved outside "lib/".  With changes up to 08/10,
> that submodule will no longer be interesting, but with this 09/10,
> when recursing clone will activate it upon the initial clone, the
> submodule is explicitly marked as active by its name, so no matter
> where it gets moved later, it will remain to be interesting.
> 
> I am not sure if that is a desired behaviour.

I'm going to blame my poor commit msg for this.

Essentially it would be nice if when a user explicitly says "I want to
init submodule b" that we obey and do just that.  What this patch does
is allow for a user to do that and the "submodule.<name>.active" config
option will only be set to 'true' in the event that that the submodule
isn't already marked as being active/interesting by some other means.
So if we did

  git clone --submodule-spec=lib/

then each module under 'lib/' won't have their submodule.<name>.active
config set since submodule.active=lib/ already encompasses them.


But if no submodule-spec was given then it would default to all
submodules and set every submodule's submodule.<name>.active config to
true.

I'll add a few tests to illustrate this.

> 
> >  builtin/submodule--helper.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> > index a3acc9e4c..b669ed031 100644
> > --- a/builtin/submodule--helper.c
> > +++ b/builtin/submodule--helper.c
> > @@ -361,6 +361,13 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
> >  		die(_("No url found for submodule path '%s' in .gitmodules"),
> >  			displaypath);
> >  
> > +	/* Set active flag for the submodule being initialized */
> > +	if (!is_submodule_initialized(path)) {
> > +		strbuf_reset(&sb);
> > +		strbuf_addf(&sb, "submodule.%s.active", sub->name);
> > +		git_config_set_gently(sb.buf, "true");
> > +	}
> > +
> >  	/*
> >  	 * Copy url setting when it is not set yet.
> >  	 * To look up the url in .git/config, we must not fall back to

-- 
Brandon Williams

Re: [PATCH v3 09/10] submodule--helper init: set submodule.<name>.active

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> So if we did
>
>   git clone --submodule-spec=lib/
>
> then each module under 'lib/' won't have their submodule.<name>.active
> config set since submodule.active=lib/ already encompasses them.

Ah, OK.  That removes my worries ;-)  Thanks for an explanation.

[PATCH v3 10/10] submodule add: respect submodule.active and submodule.<name>.active

[Brandon Williams]

In addition to adding submodule.<name>.url to the config, set
submodule.<name>.active to true unless submodule.active is configured
and the submodule's path matches the configured pathspec.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh               | 12 ++++++++++++
 t/t7413-submodule-is-active.sh | 21 +++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/git-submodule.sh b/git-submodule.sh
index 3c7da08aa..2c510038d 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -278,6 +278,18 @@ or you are unsure what this means choose another name with the '--name' option."
 	fi &&
 	git add --force .gitmodules ||
 	die "$(eval_gettext "Failed to register submodule '\$sm_path'")"
+
+	if git config --get submodule.active >/dev/null
+	then
+		# If the submodule being adding isn't already covered by the
+		# current configured pathspec, set the submodule's active flag
+		if ! git submodule--helper is-active "$sm_path"
+		then
+			git config --add submodule."$sm_name".active "true"
+		fi
+	else
+		git config --add submodule."$sm_name".active "true"
+	fi
 }
 
 #
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
index c41b899ab..865931978 100755
--- a/t/t7413-submodule-is-active.sh
+++ b/t/t7413-submodule-is-active.sh
@@ -15,6 +15,12 @@ test_expect_success 'setup' '
 	test_commit -C super initial &&
 	git -C super submodule add ../sub sub1 &&
 	git -C super submodule add ../sub sub2 &&
+
+	# Remove submodule.<name>.active entries in order to test in an
+	# environment where only URLs are present in the conifg
+	git -C super config --unset submodule.sub1.active &&
+	git -C super config --unset submodule.sub2.active &&
+
 	git -C super commit -a -m "add 2 submodules at sub{1,2}"
 '
 
@@ -83,4 +89,19 @@ test_expect_success 'is-active with submodule.active and submodule.<name>.active
 	git -C super config --unset submodule.sub2.active
 '
 
+test_expect_success 'is-active, submodule.active and submodule add' '
+	test_when_finished "rm -rf super2" &&
+	git init super2 &&
+	test_commit -C super2 initial &&
+	git -C super2 config --add submodule.active "sub*" &&
+
+	# submodule add should only add submodule.<name>.active
+	# to the config if not matched by the pathspec
+	git -C super2 submodule add ../sub sub1 &&
+	test_must_fail git -C super2 config --get submodule.sub1.active &&
+
+	git -C super2 submodule add ../sub mod &&
+	git -C super2 config --get submodule.mod.active
+'
+
 test_done
-- 
2.12.0.246.ga2ecc84866-goog

Re: [PATCH v3 10/10] submodule add: respect submodule.active and submodule.<name>.active

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> +	if git config --get submodule.active >/dev/null
> +	then
> +		# If the submodule being adding isn't already covered by the
> +		# current configured pathspec, set the submodule's active flag
> +		if ! git submodule--helper is-active "$sm_path"
> +		then
> +			git config --add submodule."$sm_name".active "true"
> +		fi
> +	else
> +		git config --add submodule."$sm_name".active "true"
> +	fi

Why "--add"?  That's a way to add new entry for multi-valued
configuration, but you do not care if the old value (if existed)
were false or true---you want to replace it to true here, no?

Re: [PATCH v3 10/10] submodule add: respect submodule.active and submodule.<name>.active

[Brandon Williams]

On 03/14, Junio C Hamano wrote:
> Brandon Williams <bmwill@google.com> writes:
> 
> > +	if git config --get submodule.active >/dev/null
> > +	then
> > +		# If the submodule being adding isn't already covered by the
> > +		# current configured pathspec, set the submodule's active flag
> > +		if ! git submodule--helper is-active "$sm_path"
> > +		then
> > +			git config --add submodule."$sm_name".active "true"
> > +		fi
> > +	else
> > +		git config --add submodule."$sm_name".active "true"
> > +	fi
> 
> Why "--add"?  That's a way to add new entry for multi-valued
> configuration, but you do not care if the old value (if existed)
> were false or true---you want to replace it to true here, no?

Correct, that was a leftover bit from a previous version of the patch.
Will fix.

-- 
Brandon Williams

Re: [PATCH v3 00/10] decoupling a submodule's existence and its url

[Stefan Beller]

On Mon, Mar 13, 2017 at 2:43 PM, Brandon Williams <bmwill@google.com> wrote:
> changes in v3:
>
> * Droped a patch which tried to use a more accurate URL for deinit.  It didn't
>   really fit inside the scope of this series.  It may be something we want to
>   revisit later though.
>
> * The --init-active flag now ensure that all submodules which are configured to
>   be 'active' (either via 'submodule.active' or 'submodule.<name>.active') go
>   through the initialization phase and have their relevent info copied over to
>   the config.
>

This round looks sensible to me.

Thanks,
Stefan

Re: [PATCH v3 00/10] decoupling a submodule's existence and its url

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> changes in v3:
>
> * Droped a patch which tried to use a more accurate URL for deinit.  It didn't
>   really fit inside the scope of this series.  It may be something we want to
>   revisit later though.
>
> * The --init-active flag now ensure that all submodules which are configured to
>   be 'active' (either via 'submodule.active' or 'submodule.<name>.active') go
>   through the initialization phase and have their relevent info copied over to
>   the config.

Aside from comments I sent already to the individual patches, I
think overall this coherently adds a nicer way to keep track of
which submodules are of interest (and which are not) than what we
traditionally had.  I'll queue it on 'pu', awaiting further
polishing (if necessary).

Thanks.

Re: [PATCH v3 00/10] decoupling a submodule's existence and its url

[Brandon Williams]

On 03/14, Junio C Hamano wrote:
> Brandon Williams <bmwill@google.com> writes:
> 
> > changes in v3:
> >
> > * Droped a patch which tried to use a more accurate URL for deinit.  It didn't
> >   really fit inside the scope of this series.  It may be something we want to
> >   revisit later though.
> >
> > * The --init-active flag now ensure that all submodules which are configured to
> >   be 'active' (either via 'submodule.active' or 'submodule.<name>.active') go
> >   through the initialization phase and have their relevent info copied over to
> >   the config.
> 
> Aside from comments I sent already to the individual patches, I
> think overall this coherently adds a nicer way to keep track of
> which submodules are of interest (and which are not) than what we
> traditionally had.  I'll queue it on 'pu', awaiting further
> polishing (if necessary).

I've just started looking at your comments and I expect I'll do at least
one more round of polish.

Thanks for taking a look!

-- 
Brandon Williams

[PATCH v4 00/10] decoupling url and submodule interest

[Brandon Williams]

Lots of changes in v4 to address comments by Junio, Ramsay, and Stefan.
  * Overhaul a lot of the commit messages (thanks Junio for helping wordsmith)
  * Change how submodule.active functions with `submodule init` and clone
    --recurse-submodules.  This allowed me to not have to add additional
    command line options resulting in what I think is less cluttered UI from
    the last version.  See patch 7 and 8 for more detail.
  * Fixed an incorrect parameter type in patch 6 (thanks Ramsay).

Brandon Williams (10):
  submodule--helper: add is-active subcommand
  submodule status: use submodule--helper is-active
  submodule sync: skip work for inactive submodules
  submodule sync: use submodule--helper is-active
  submodule--helper clone: check for configured submodules using helper
  submodule: decouple url and submodule interest
  submodule init: initialize active submodules
  clone: teach --recurse-submodules to optionally take a pathspec
  submodule--helper init: set submodule.<name>.active
  submodule add: respect submodule.active and submodule.<name>.active

 Documentation/config.txt        |  15 ++++-
 Documentation/git-clone.txt     |  14 +++--
 Documentation/git-submodule.txt |   4 +-
 builtin/clone.c                 |  47 +++++++++++---
 builtin/submodule--helper.c     |  69 ++++++++++++++++----
 git-submodule.sh                |  53 ++++++++++------
 submodule.c                     |  50 ++++++++++++---
 t/t7400-submodule-basic.sh      | 136 ++++++++++++++++++++++++++++++++++++++++
 t/t7413-submodule-is-active.sh  | 107 +++++++++++++++++++++++++++++++
 9 files changed, 440 insertions(+), 55 deletions(-)
 create mode 100755 t/t7413-submodule-is-active.sh

---interdiff with 'origin/bw/submodule-is-active'

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 9692eab30..30052cce4 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -14,9 +14,8 @@ SYNOPSIS
 	  [-o <name>] [-b <name>] [-u <upload-pack>] [--reference <repository>]
 	  [--dissociate] [--separate-git-dir <git dir>]
 	  [--depth <depth>] [--[no-]single-branch]
-	  [--recursive | --recurse-submodules] [--[no-]shallow-submodules]
-	  [--submodule-spec <pathspec>] [--jobs <n>] [--]
-	  <repository> [<directory>]
+	  [--recurse-submodules] [--[no-]shallow-submodules]
+	  [--jobs <n>] [--] <repository> [<directory>]
 
 DESCRIPTION
 -----------
@@ -216,22 +215,18 @@ objects from the source repository into a pack in the cloned repository.
 	branch when `--single-branch` clone was made, no remote-tracking
 	branch is created.
 
---recursive::
---recurse-submodules::
-	After the clone is created, initialize and clone all submodules
-	within, using their default settings. This is equivalent to
-	running `git submodule update --recursive --init` immediately
-	after the clone is finished. This option is ignored if the
-	cloned repository does not have a worktree/checkout (i.e.  if
-	any of `--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
-
---submodule-spec::
-	After the clone is created, initialize and clone specified
-	submodules within, using their default settings. It is possible
-	to give multiple specifications by giving this argument multiple
-	times. This is equivalent to configuring `submodule.active`
-	and then running `git submodule update --init-active`
-	immediately after the clone is finished.
+--recurse-submodules[=<pathspec]::
+	After the clone is created, initialize and clone submodules
+	within based on the provided pathspec.  If no pathspec is
+	provided, all submodules are initialized and cloned.
+	Submodules are initialized and cloned using their default
+	settings.  The resulting clone has `submodule.active` set to
+	the provided pathspec, or "." (meaning all submodules) if no
+	pathspec is provided.  This is equivalent to running
+	`git submodule update --init --recursive` immediately after
+	the clone is finished. This option is ignored if the cloned
+	repository does not have a worktree/checkout (i.e. if any of
+	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
 
 --[no-]shallow-submodules::
 	All submodules which are cloned will be shallow with a depth of 1.
diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 6b17cd707..74bc6200d 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -129,7 +129,9 @@ init [--] [<path>...]::
 	repository will be assumed to be upstream.
 +
 Optional <path> arguments limit which submodules will be initialized.
-If no path is specified, all submodules are initialized.
+If no path is specified and submodule.active has been configured, submodules
+configured to be active will be initialized, otherwise all submodules are
+initialized.
 +
 When present, it will also copy the value of `submodule.$name.update`.
 This command does not alter existing information in .git/config.
@@ -198,10 +200,6 @@ If the submodule is not yet initialized, and you just want to use the
 setting as stored in .gitmodules, you can automatically initialize the
 submodule with the `--init` option.
 
-You can configure a set of submodules using pathspec syntax in
-submodule.active you can use `--init-active` to initialize
-those before updating.
-
 If `--recursive` is specified, this command will recurse into the
 registered submodules, and update any nested submodules within.
 --
@@ -388,11 +386,6 @@ the submodule itself.
 	Initialize all submodules for which "git submodule init" has not been
 	called so far before updating.
 
---init-active::
-	This option is only valid for the update command.
-	Initialize all submodules configured in "`submodule.active`"
-	that have not been updated before.
-
 --name::
 	This option is only valid for the add command. It sets the submodule's
 	name to the given string instead of defaulting to its path. The name
diff --git a/builtin/clone.c b/builtin/clone.c
index c6731379b..3dc8faac5 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -39,7 +39,7 @@ static const char * const builtin_clone_usage[] = {
 };
 
 static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
-static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
+static int option_local = -1, option_no_hardlinks, option_shared;
 static int option_shallow_submodules;
 static int deepen;
 static char *option_template, *option_depth, *option_since;
@@ -56,14 +56,20 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;
 static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;
 static int option_dissociate;
 static int max_jobs = -1;
-static struct string_list submodule_spec;
+static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;
 
-static int submodule_spec_cb(const struct option *opt, const char *arg, int unset)
+static int recurse_submodules_cb(const struct option *opt,
+				 const char *arg, int unset)
 {
 	if (unset)
 		return -1;
 
-	string_list_append((struct string_list *)opt->value, arg);
+	if (arg)
+		string_list_append((struct string_list *)opt->value, arg);
+	else
+		string_list_append((struct string_list *)opt->value,
+				   (const char *)opt->defval);
+
 	return 0;
 }
 
@@ -84,10 +90,13 @@ static struct option builtin_clone_options[] = {
 		    N_("don't use local hardlinks, always copy")),
 	OPT_BOOL('s', "shared", &option_shared,
 		    N_("setup as shared repository")),
-	OPT_BOOL(0, "recursive", &option_recursive,
-		    N_("initialize submodules in the clone")),
-	OPT_BOOL(0, "recurse-submodules", &option_recursive,
-		    N_("initialize submodules in the clone")),
+	{ OPTION_CALLBACK, 0, "recursive", &option_recurse_submodules,
+	  N_("pathspec"), N_("initialize submodules in the clone"),
+	  PARSE_OPT_OPTARG | PARSE_OPT_HIDDEN, recurse_submodules_cb,
+	  (intptr_t)"." },
+	{ OPTION_CALLBACK, 0, "recurse-submodules", &option_recurse_submodules,
+	  N_("pathspec"), N_("initialize submodules in the clone"),
+	  PARSE_OPT_OPTARG, recurse_submodules_cb, (intptr_t)"." },
 	OPT_INTEGER('j', "jobs", &max_jobs,
 		    N_("number of submodules cloned in parallel")),
 	OPT_STRING(0, "template", &option_template, N_("template-directory"),
@@ -122,9 +131,6 @@ static struct option builtin_clone_options[] = {
 			TRANSPORT_FAMILY_IPV4),
 	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
 			TRANSPORT_FAMILY_IPV6),
-	OPT_CALLBACK(0, "submodule-spec", &submodule_spec, N_("<pathspec>"),
-			N_("clone specific submodules. Pass multiple times for complex pathspecs"),
-			submodule_spec_cb),
 	OPT_END()
 };
 
@@ -746,21 +752,13 @@ static int checkout(int submodule_progress)
 	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
 			   sha1_to_hex(sha1), "1", NULL);
 
-	if (!err && (option_recursive || submodule_spec.nr > 0)) {
+	if (!err && (option_recurse_submodules.nr > 0)) {
 		struct argv_array args = ARGV_ARRAY_INIT;
-		argv_array_pushl(&args, "submodule", "update", NULL);
-
-		if (submodule_spec.nr > 0)
-			argv_array_pushf(&args, "--init-active");
-		else
-			argv_array_pushf(&args, "--init");
+		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
 
 		if (option_shallow_submodules == 1)
 			argv_array_push(&args, "--depth=1");
 
-		if (option_recursive)
-			argv_array_pushf(&args, "--recursive");
-
 		if (max_jobs != -1)
 			argv_array_pushf(&args, "--jobs=%d", max_jobs);
 
@@ -908,17 +906,6 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		option_no_checkout = 1;
 	}
 
-	if (submodule_spec.nr > 0) {
-		struct string_list_item *item;
-		struct strbuf sb = STRBUF_INIT;
-		for_each_string_list_item(item, &submodule_spec) {
-			strbuf_addf(&sb, "submodule.active=%s",
-				    item->string);
-			string_list_append(&option_config,
-					   strbuf_detach(&sb, NULL));
-		}
-	}
-
 	if (!option_origin)
 		option_origin = "origin";
 
@@ -989,7 +976,21 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 			fprintf(stderr, _("Cloning into '%s'...\n"), dir);
 	}
 
-	if (option_recursive) {
+	if (option_recurse_submodules.nr > 0) {
+		struct string_list_item *item;
+		struct strbuf sb = STRBUF_INIT;
+
+		/* remove duplicates */
+		string_list_sort(&option_recurse_submodules);
+		string_list_remove_duplicates(&option_recurse_submodules, 0);
+
+		for_each_string_list_item(item, &option_recurse_submodules) {
+			strbuf_addf(&sb, "submodule.active=%s",
+				    item->string);
+			string_list_append(&option_config,
+					   strbuf_detach(&sb, NULL));
+		}
+
 		if (option_required_reference.nr &&
 		    option_optional_reference.nr)
 			die(_("clone --recursive is not compatible with "
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index b669ed031..a574596cb 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -437,12 +437,9 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	struct pathspec pathspec;
 	struct module_list list = MODULE_LIST_INIT;
 	int quiet = 0;
-	int active = 0;
 	int i;
 
 	struct option module_init_options[] = {
-		OPT_BOOL(0, "active", &active,
-			 N_("ensure all active submodules are properly initialized")),
 		OPT__QUIET(&quiet, N_("Suppress output for initializing a submodule")),
 		OPT_END()
 	};
@@ -455,7 +452,11 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix, module_init_options,
 			     git_submodule_helper_usage, 0);
 
-	if (active)
+	/*
+	 * If there are no path args and submodule.active is set then,
+	 * by default, only initialize 'active' modules.
+	 */
+	if (!argc && git_config_get_value_multi("submodule.active"))
 		module_list_active(&list);
 	else if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
 		return 1;
diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index eb13433d5..fc32286a4 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -1212,7 +1212,6 @@ _git_clone ()
 			--recurse-submodules
 			--no-single-branch
 			--shallow-submodules
-			--submodule-spec
 			"
 		return
 		;;
diff --git a/git-submodule.sh b/git-submodule.sh
index 2c510038d..6eca93416 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -9,7 +9,7 @@ USAGE="[--quiet] add [-b <branch>] [-f|--force] [--name <name>] [--reference <re
    or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]
    or: $dashless [--quiet] init [--] [<path>...]
    or: $dashless [--quiet] deinit [-f|--force] (--all| [--] <path>...)
-   or: $dashless [--quiet] update [--init[-active]] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference <repository>] [--recursive] [--] [<path>...]
+   or: $dashless [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--checkout|--merge|--rebase] [--[no-]recommend-shallow] [--reference <repository>] [--recursive] [--] [<path>...]
    or: $dashless [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
    or: $dashless [--quiet] foreach [--recursive] <command>
    or: $dashless [--quiet] sync [--recursive] [--] [<path>...]
@@ -285,10 +285,10 @@ or you are unsure what this means choose another name with the '--name' option."
 		# current configured pathspec, set the submodule's active flag
 		if ! git submodule--helper is-active "$sm_path"
 		then
-			git config --add submodule."$sm_name".active "true"
+			git config submodule."$sm_name".active "true"
 		fi
 	else
-		git config --add submodule."$sm_name".active "true"
+		git config submodule."$sm_name".active "true"
 	fi
 }
 
@@ -375,9 +375,6 @@ cmd_init()
 		-q|--quiet)
 			GIT_QUIET=1
 			;;
-		--active)
-			GIT_ACTIVE=1
-			;;
 		--)
 			shift
 			break
@@ -392,7 +389,7 @@ cmd_init()
 		shift
 	done
 
-	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper init ${GIT_QUIET:+--quiet} ${GIT_ACTIVE:+--active} "$@"
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper init ${GIT_QUIET:+--quiet}  "$@"
 }
 
 #
@@ -521,12 +518,7 @@ cmd_update()
 			progress="--progress"
 			;;
 		-i|--init)
-			test -z $init || test $init = by_args || die "$(gettext "Only one of --init or --init-active may be used.")"
-			init=by_args
-			;;
-		--init-active)
-			test -z $init || test $init = by_config || die "$(gettext "Only one of --init or --init-active may be used.")"
-			init=by_config
+			init=1
 			;;
 		--remote)
 			remote=1
@@ -595,17 +587,7 @@ cmd_update()
 
 	if test -n "$init"
 	then
-		if test "$init" = "by_config"
-		then
-			if test $# -gt 0
-			then
-				die "$(gettext "path arguments are incompatible with --init-active")"
-			fi
-			cmd_init "--active" || return
-		else
-			cmd_init "--" "$@" || return
-		fi
-
+		cmd_init "--" "$@" || return
 	fi
 
 	{
@@ -1119,6 +1101,13 @@ cmd_sync()
 	while read mode sha1 stage sm_path
 	do
 		die_if_unmatched "$mode" "$sha1"
+
+		# skip inactive submodules
+		if ! git submodule--helper is-active "$sm_path"
+		then
+			continue
+		fi
+
 		name=$(git submodule--helper name "$sm_path")
 		url=$(git config -f .gitmodules --get submodule."$name".url)
 
@@ -1141,27 +1130,24 @@ cmd_sync()
 			;;
 		esac
 
-		if git submodule--helper is-active "$sm_path"
+		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
+		say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
+		git config submodule."$name".url "$super_config_url"
+
+		if test -e "$sm_path"/.git
 		then
-			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-			say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
-			git config submodule."$name".url "$super_config_url"
+		(
+			sanitize_submodule_env
+			cd "$sm_path"
+			remote=$(get_default_remote)
+			git config remote."$remote".url "$sub_origin_url"
 
-			if test -e "$sm_path"/.git
+			if test -n "$recursive"
 			then
-			(
-				sanitize_submodule_env
-				cd "$sm_path"
-				remote=$(get_default_remote)
-				git config remote."$remote".url "$sub_origin_url"
-
-				if test -n "$recursive"
-				then
-					prefix="$prefix$sm_path/"
-					eval cmd_sync
-				fi
-			)
+				prefix="$prefix$sm_path/"
+				eval cmd_sync
 			fi
+		)
 		fi
 	done
 }
diff --git a/submodule.c b/submodule.c
index 2b33bd70f..ad2779ee7 100644
--- a/submodule.c
+++ b/submodule.c
@@ -212,12 +212,19 @@ void gitmodules_config_sha1(const unsigned char *commit_sha1)
 }
 
 /*
+ * NEEDSWORK: With the addition of different configuration options to determine
+ * if a submodule is of interests, the validity of this function's name comes
+ * into question.  Once the dust has settled and more concrete terminology is
+ * decided upon, come up with a more proper name for this function.  One
+ * potential candidate could be 'is_submodule_active()'.
+ *
  * Determine if a submodule has been initialized at a given 'path'
  */
 int is_submodule_initialized(const char *path)
 {
 	int ret = 0;
-	char *key;
+	char *key = NULL;
+	char *value = NULL;
 	const struct string_list *sl;
 	const struct submodule *module = submodule_from_path(null_sha1, path);
 
@@ -233,8 +240,8 @@ int is_submodule_initialized(const char *path)
 	}
 	free(key);
 
+	/* submodule.active is set */
 	sl = git_config_get_value_multi("submodule.active");
-
 	if (sl) {
 		struct pathspec ps;
 		struct argv_array args = ARGV_ARRAY_INIT;
@@ -244,21 +251,20 @@ int is_submodule_initialized(const char *path)
 			argv_array_push(&args, item->string);
 		}
 
-		parse_pathspec(&ps, 0, 0, 0, args.argv);
+		parse_pathspec(&ps, 0, 0, NULL, args.argv);
 		ret = match_pathspec(&ps, path, strlen(path), 0, NULL, 1);
 
 		argv_array_clear(&args);
 		clear_pathspec(&ps);
-	} else {
-		char *value = NULL;
-		key = xstrfmt("submodule.%s.url", module->name);
-
-		ret = !git_config_get_string(key, &value);
-
-		free(value);
-		free(key);
+		return ret;
 	}
 
+	/* fallback to checking if the URL is set */
+	key = xstrfmt("submodule.%s.url", module->name);
+	ret = !git_config_get_string(key, &value);
+
+	free(value);
+	free(key);
 	return ret;
 }
 
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index c2e198a92..cf77a3a35 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1158,19 +1158,18 @@ test_expect_success 'submodule update --init with a specification' '
 	test_cmp expect actual
 '
 
-test_expect_success 'submodule update --init-active' '
+test_expect_success 'submodule update --init with submodule.active set' '
 	test_when_finished "rm -rf multisuper_clone" &&
 	pwd=$(pwd) &&
 	git clone file://"$pwd"/multisuper multisuper_clone &&
 	git -C multisuper_clone config submodule.active "." &&
 	git -C multisuper_clone config --add submodule.active ":(exclude)sub0" &&
-	git -C multisuper_clone submodule update --init-active &&
+	git -C multisuper_clone submodule update --init &&
 	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
-	test_must_fail git -C multisuper_clone submodule update --init-active sub0 &&
 	test_cmp expect actual
 '
 
-test_expect_success 'submodule init --active and update' '
+test_expect_success 'submodule update and setting submodule.<name>.active' '
 	test_when_finished "rm -rf multisuper_clone" &&
 	pwd=$(pwd) &&
 	git clone file://"$pwd"/multisuper multisuper_clone &&
@@ -1189,7 +1188,7 @@ test_expect_success 'submodule init --active and update' '
 	test_cmp expect actual
 '
 
-test_expect_success 'clone --submodule-spec works' '
+test_expect_success 'clone --recurse-submodules with a pathspec works' '
 	test_when_finished "rm -rf multisuper_clone" &&
 	cat >expected <<-\EOF &&
 	 sub0 (test2)
@@ -1198,12 +1197,12 @@ test_expect_success 'clone --submodule-spec works' '
 	-sub3
 	EOF
 
-	git clone --recurse-submodules --submodule-spec="sub0" multisuper multisuper_clone &&
+	git clone --recurse-submodules="sub0" multisuper multisuper_clone &&
 	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
 	test_cmp actual expected
 '
 
-test_expect_success 'clone with multiple --submodule-spec options' '
+test_expect_success 'clone with multiple --recurse-submodules options' '
 	test_when_finished "rm -rf multisuper_clone" &&
 	cat >expect <<-\EOF &&
 	-sub0
@@ -1212,10 +1211,9 @@ test_expect_success 'clone with multiple --submodule-spec options' '
 	 sub3 (test2)
 	EOF
 
-	git clone --recurse-submodules \
-		  --submodule-spec="." \
-		  --submodule-spec ":(exclude)sub0" \
-		  --submodule-spec ":(exclude)sub2" \
+	git clone --recurse-submodules="." \
+		  --recurse-submodules=":(exclude)sub0" \
+		  --recurse-submodules=":(exclude)sub2" \
 		  multisuper multisuper_clone &&
 	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
 	test_cmp expect actual
@@ -1239,11 +1237,10 @@ test_expect_success 'clone and subsequent updates correctly auto-initialize subm
 	 sub5 (test2)
 	EOF
 
-	git clone --recurse-submodules \
-		  --submodule-spec="." \
-		  --submodule-spec ":(exclude)sub0" \
-		  --submodule-spec ":(exclude)sub2" \
-		  --submodule-spec ":(exclude)sub4" \
+	git clone --recurse-submodules="." \
+		  --recurse-submodules=":(exclude)sub0" \
+		  --recurse-submodules=":(exclude)sub2" \
+		  --recurse-submodules=":(exclude)sub4" \
 		  multisuper multisuper_clone &&
 
 	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
@@ -1254,9 +1251,20 @@ test_expect_success 'clone and subsequent updates correctly auto-initialize subm
 	git -C multisuper commit -m "add more submodules" &&
 	# obtain the new superproject
 	git -C multisuper_clone pull &&
-	git -C multisuper_clone submodule update --init-active &&
+	git -C multisuper_clone submodule update --init &&
 	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
 	test_cmp expect2 actual
 '
 
+test_expect_success 'init properly sets the config' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	git clone --recurse-submodules="." \
+		  --recurse-submodules=":(exclude)sub0" \
+		  multisuper multisuper_clone &&
+
+	git -C multisuper_clone submodule init -- sub0 sub1 &&
+	git -C multisuper_clone config --get submodule.sub0.active &&
+	test_must_fail git -C multisuper_clone config --get submodule.sub1.active
+'
+
 test_done
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
index 865931978..9c785b07e 100755
--- a/t/t7413-submodule-is-active.sh
+++ b/t/t7413-submodule-is-active.sh
@@ -35,58 +35,58 @@ test_expect_success 'is-active works with urls' '
 '
 
 test_expect_success 'is-active works with submodule.<name>.active config' '
+	test_when_finished "git -C super config --unset submodule.sub1.active" &&
+	test_when_finished "git -C super config submodule.sub1.URL ../sub" &&
+
 	git -C super config --bool submodule.sub1.active "false" &&
 	test_must_fail git -C super submodule--helper is-active sub1 &&
 
 	git -C super config --bool submodule.sub1.active "true" &&
 	git -C super config --unset submodule.sub1.URL &&
-	git -C super submodule--helper is-active sub1 &&
-
-	git -C super config submodule.sub1.URL ../sub &&
-	git -C super config --unset submodule.sub1.active
+	git -C super submodule--helper is-active sub1
 '
 
 test_expect_success 'is-active works with basic submodule.active config' '
+	test_when_finished "git -C super config submodule.sub1.URL ../sub" &&
+	test_when_finished "git -C super config --unset-all submodule.active" &&
+
 	git -C super config --add submodule.active "." &&
 	git -C super config --unset submodule.sub1.URL &&
 
 	git -C super submodule--helper is-active sub1 &&
-	git -C super submodule--helper is-active sub2 &&
-
-	git -C super config submodule.sub1.URL ../sub &&
-	git -C super config --unset-all submodule.active
+	git -C super submodule--helper is-active sub2
 '
 
 test_expect_success 'is-active correctly works with paths that are not submodules' '
-	test_must_fail git -C super submodule--helper is-active not-a-submodule &&
+	test_when_finished "git -C super config --unset-all submodule.active" &&
 
-	git -C super config --add submodule.active "." &&
 	test_must_fail git -C super submodule--helper is-active not-a-submodule &&
 
-	git -C super config --unset-all submodule.active
+	git -C super config --add submodule.active "." &&
+	test_must_fail git -C super submodule--helper is-active not-a-submodule
 '
 
 test_expect_success 'is-active works with exclusions in submodule.active config' '
+	test_when_finished "git -C super config --unset-all submodule.active" &&
+
 	git -C super config --add submodule.active "." &&
 	git -C super config --add submodule.active ":(exclude)sub1" &&
 
 	test_must_fail git -C super submodule--helper is-active sub1 &&
-	git -C super submodule--helper is-active sub2 &&
-
-	git -C super config --unset-all submodule.active
+	git -C super submodule--helper is-active sub2
 '
 
 test_expect_success 'is-active with submodule.active and submodule.<name>.active' '
+	test_when_finished "git -C super config --unset-all submodule.active" &&
+	test_when_finished "git -C super config --unset submodule.sub1.active" &&
+	test_when_finished "git -C super config --unset submodule.sub2.active" &&
+
 	git -C super config --add submodule.active "sub1" &&
 	git -C super config --bool submodule.sub1.active "false" &&
 	git -C super config --bool submodule.sub2.active "true" &&
 
 	test_must_fail git -C super submodule--helper is-active sub1 &&
-	git -C super submodule--helper is-active sub2 &&
-
-	git -C super config --unset-all submodule.active &&
-	git -C super config --unset submodule.sub1.active &&
-	git -C super config --unset submodule.sub2.active
+	git -C super submodule--helper is-active sub2
 '
 
 test_expect_success 'is-active, submodule.active and submodule add' '

-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v4 01/10] submodule--helper: add is-active subcommand

[Brandon Williams]

The definition of which submodules are of interest by the user
is tied to the configuration submodule.<name>.url; when it is
set to a non-empty string, it is of interest.  We'd want to be
able to later change this definition, but there are many places
that explicitly check this condition in the scripted Porcelain.

Introduce the "is-active" subcommand to "submodule--helper", so
that the exact definition of what submodule is of interest can
be centrally defined (and changed in later steps).  In a few
patches that follow, this helper is used to replace the explicit
checks of the configuration variable in scripts.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c    | 11 +++++++++++
 t/t7413-submodule-is-active.sh | 31 +++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100755 t/t7413-submodule-is-active.sh

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 15a5430c0..5fe7e23b1 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -1127,6 +1127,16 @@ static int absorb_git_dirs(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+static int is_active(int argc, const char **argv, const char *prefix)
+{
+	if (argc != 2)
+		die("submodule--helper is-active takes exactly 1 arguments");
+
+	gitmodules_config();
+
+	return !is_submodule_initialized(argv[1]);
+}
+
 #define SUPPORT_SUPER_PREFIX (1<<0)
 
 struct cmd_struct {
@@ -1146,6 +1156,7 @@ static struct cmd_struct commands[] = {
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
+	{"is-active", is_active, 0},
 };
 
 int cmd_submodule__helper(int argc, const char **argv, const char *prefix)
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
new file mode 100755
index 000000000..f18e0c925
--- /dev/null
+++ b/t/t7413-submodule-is-active.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+
+test_description='Test submodule--helper is-active
+
+This test verifies that `git submodue--helper is-active` correclty identifies
+submodules which are "active" and interesting to the user.
+'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	git init sub &&
+	test_commit -C sub initial &&
+	git init super &&
+	test_commit -C super initial &&
+	git -C super submodule add ../sub sub1 &&
+	git -C super submodule add ../sub sub2 &&
+	git -C super commit -a -m "add 2 submodules at sub{1,2}"
+'
+
+test_expect_success 'is-active works with urls' '
+	git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config --unset submodule.sub1.URL &&
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super config submodule.sub1.URL ../sub &&
+	git -C super submodule--helper is-active sub1
+'
+
+test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v4 02/10] submodule status: use submodule--helper is-active

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 136e26a2c..ab233712d 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1010,14 +1010,13 @@ cmd_status()
 	do
 		die_if_unmatched "$mode" "$sha1"
 		name=$(git submodule--helper name "$sm_path") || exit
-		url=$(git config submodule."$name".url)
 		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
 		if test "$stage" = U
 		then
 			say "U$sha1 $displaypath"
 			continue
 		fi
-		if test -z "$url" ||
+		if ! git submodule--helper is-active "$sm_path" ||
 		{
 			! test -d "$sm_path"/.git &&
 			! test -f "$sm_path"/.git
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v4 03/10] submodule sync: skip work for inactive submodules

[Brandon Williams]

Sync does some work determining what URLs should be used for a submodule
but then throws this work away if the submodule isn't active.  Instead
perform the activity check earlier and skip inactive submodule in order
to avoid doing unnecessary work.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 38 +++++++++++++++++++++-----------------
 1 file changed, 21 insertions(+), 17 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index ab233712d..577136148 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1090,6 +1090,13 @@ cmd_sync()
 	do
 		die_if_unmatched "$mode" "$sha1"
 		name=$(git submodule--helper name "$sm_path")
+
+		# skip inactive submodules
+		if ! git config "submodule.$name.url" >/dev/null 2>/dev/null
+		then
+			continue
+		fi
+
 		url=$(git config -f .gitmodules --get submodule."$name".url)
 
 		# Possibly a url relative to parent
@@ -1111,27 +1118,24 @@ cmd_sync()
 			;;
 		esac
 
-		if git config "submodule.$name.url" >/dev/null 2>/dev/null
+		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
+		say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
+		git config submodule."$name".url "$super_config_url"
+
+		if test -e "$sm_path"/.git
 		then
-			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-			say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
-			git config submodule."$name".url "$super_config_url"
+		(
+			sanitize_submodule_env
+			cd "$sm_path"
+			remote=$(get_default_remote)
+			git config remote."$remote".url "$sub_origin_url"
 
-			if test -e "$sm_path"/.git
+			if test -n "$recursive"
 			then
-			(
-				sanitize_submodule_env
-				cd "$sm_path"
-				remote=$(get_default_remote)
-				git config remote."$remote".url "$sub_origin_url"
-
-				if test -n "$recursive"
-				then
-					prefix="$prefix$sm_path/"
-					eval cmd_sync
-				fi
-			)
+				prefix="$prefix$sm_path/"
+				eval cmd_sync
 			fi
+		)
 		fi
 	done
 }
-- 
2.12.0.367.g23dc2f6d3c-goog

Re: [PATCH v4 03/10] submodule sync: skip work for inactive submodules

[Stefan Beller]

On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
> Sync does some work determining what URLs should be used for a submodule
> but then throws this work away if the submodule isn't active.  Instead
> perform the activity check earlier and skip inactive submodule in order
> to avoid doing unnecessary work.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---

> +               # skip inactive submodules
> +               if ! git config "submodule.$name.url" >/dev/null 2>/dev/null

As a stand alone patch, this would look good to me.
But with the goal of this patch series in mind (which is to add
more options to enable a submodules "active" state), this
should rather be "submodule--helper is-active" here?


> +               then
> +                       continue
> +               fi
> +

The changes below here are just de-dented, no behavioral change?

>
> -               if git config "submodule.$name.url" >/dev/null 2>/dev/null
> +               displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
> +               say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
> +               git config submodule."$name".url "$super_config_url"
> +
> +               if test -e "$sm_path"/.git
>                 then
> -                       displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
> -                       say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
> -                       git config submodule."$name".url "$super_config_url"
> +               (
> +                       sanitize_submodule_env
> +                       cd "$sm_path"
> +                       remote=$(get_default_remote)
> +                       git config remote."$remote".url "$sub_origin_url"
>
> -                       if test -e "$sm_path"/.git
> +                       if test -n "$recursive"
>                         then
> -                       (
> -                               sanitize_submodule_env
> -                               cd "$sm_path"
> -                               remote=$(get_default_remote)
> -                               git config remote."$remote".url "$sub_origin_url"
> -
> -                               if test -n "$recursive"
> -                               then
> -                                       prefix="$prefix$sm_path/"
> -                                       eval cmd_sync
> -                               fi
> -                       )
> +                               prefix="$prefix$sm_path/"
> +                               eval cmd_sync
>                         fi
> +               )

Re: [PATCH v4 03/10] submodule sync: skip work for inactive submodules

[Stefan Beller]

On Thu, Mar 16, 2017 at 4:24 PM, Stefan Beller <sbeller@google.com> wrote:
> On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
>> Sync does some work determining what URLs should be used for a submodule
>> but then throws this work away if the submodule isn't active.  Instead
>> perform the activity check earlier and skip inactive submodule in order
>> to avoid doing unnecessary work.
>>
>> Signed-off-by: Brandon Williams <bmwill@google.com>
>> ---
>
>> +               # skip inactive submodules
>> +               if ! git config "submodule.$name.url" >/dev/null 2>/dev/null
>
> As a stand alone patch, this would look good to me.
> But with the goal of this patch series in mind (which is to add
> more options to enable a submodules "active" state), this
> should rather be "submodule--helper is-active" here?
>

I spoke too early. this happens in the next patch.

ok, thanks!
Stefan

Re: [PATCH v4 03/10] submodule sync: skip work for inactive submodules

[Brandon Williams]

On 03/16, Stefan Beller wrote:
> On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
> > Sync does some work determining what URLs should be used for a submodule
> > but then throws this work away if the submodule isn't active.  Instead
> > perform the activity check earlier and skip inactive submodule in order
> > to avoid doing unnecessary work.
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> 
> > +               # skip inactive submodules
> > +               if ! git config "submodule.$name.url" >/dev/null 2>/dev/null
> 
> As a stand alone patch, this would look good to me.
> But with the goal of this patch series in mind (which is to add
> more options to enable a submodules "active" state), this
> should rather be "submodule--helper is-active" here?

Junio mentioned do this as a preparatory step to make it easier to
review.  The next patch changes that to use 'is-active' instead of the
URL.

> 
> 
> > +               then
> > +                       continue
> > +               fi
> > +
> 
> The changes below here are just de-dented, no behavioral change?

Yeah no behavioral change, just short circuiting earlier.
> 
> >
> > -               if git config "submodule.$name.url" >/dev/null 2>/dev/null
> > +               displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
> > +               say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
> > +               git config submodule."$name".url "$super_config_url"
> > +
> > +               if test -e "$sm_path"/.git
> >                 then
> > -                       displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
> > -                       say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
> > -                       git config submodule."$name".url "$super_config_url"
> > +               (
> > +                       sanitize_submodule_env
> > +                       cd "$sm_path"
> > +                       remote=$(get_default_remote)
> > +                       git config remote."$remote".url "$sub_origin_url"
> >
> > -                       if test -e "$sm_path"/.git
> > +                       if test -n "$recursive"
> >                         then
> > -                       (
> > -                               sanitize_submodule_env
> > -                               cd "$sm_path"
> > -                               remote=$(get_default_remote)
> > -                               git config remote."$remote".url "$sub_origin_url"
> > -
> > -                               if test -n "$recursive"
> > -                               then
> > -                                       prefix="$prefix$sm_path/"
> > -                                       eval cmd_sync
> > -                               fi
> > -                       )
> > +                               prefix="$prefix$sm_path/"
> > +                               eval cmd_sync
> >                         fi
> > +               )

-- 
Brandon Williams

Re: [PATCH v4 03/10] submodule sync: skip work for inactive submodules

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
>> Sync does some work determining what URLs should be used for a submodule
>> but then throws this work away if the submodule isn't active.  Instead
>> perform the activity check earlier and skip inactive submodule in order
>> to avoid doing unnecessary work.
>>
>> Signed-off-by: Brandon Williams <bmwill@google.com>
>> ---
>
>> +               # skip inactive submodules
>> +               if ! git config "submodule.$name.url" >/dev/null 2>/dev/null

Just a style thing, but ">/dev/null 2>&1" is probably shorter and
more idiomatic.

Re: [PATCH v4 03/10] submodule sync: skip work for inactive submodules

[Brandon Williams]

On 03/16, Junio C Hamano wrote:
> Stefan Beller <sbeller@google.com> writes:
> 
> > On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
> >> Sync does some work determining what URLs should be used for a submodule
> >> but then throws this work away if the submodule isn't active.  Instead
> >> perform the activity check earlier and skip inactive submodule in order
> >> to avoid doing unnecessary work.
> >>
> >> Signed-off-by: Brandon Williams <bmwill@google.com>
> >> ---
> >
> >> +               # skip inactive submodules
> >> +               if ! git config "submodule.$name.url" >/dev/null 2>/dev/null
> 
> Just a style thing, but ">/dev/null 2>&1" is probably shorter and
> more idiomatic.
> 

Yeah you're right.  I was just moving around the code that was already
there.  But this line is removed in the next patch anyways so it
probably doesn't matter much.

-- 
Brandon Williams

Re: [PATCH v4 03/10] submodule sync: skip work for inactive submodules

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> ...  But this line is removed in the next patch anyways so it
> probably doesn't matter much.

Right ;-)

[PATCH v4 04/10] submodule sync: use submodule--helper is-active

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 577136148..db94dea3b 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1089,14 +1089,14 @@ cmd_sync()
 	while read mode sha1 stage sm_path
 	do
 		die_if_unmatched "$mode" "$sha1"
-		name=$(git submodule--helper name "$sm_path")
 
 		# skip inactive submodules
-		if ! git config "submodule.$name.url" >/dev/null 2>/dev/null
+		if ! git submodule--helper is-active "$sm_path"
 		then
 			continue
 		fi
 
+		name=$(git submodule--helper name "$sm_path")
 		url=$(git config -f .gitmodules --get submodule."$name".url)
 
 		# Possibly a url relative to parent
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v4 05/10] submodule--helper clone: check for configured submodules using helper

[Brandon Williams]

Use the 'is_submodule_initialized()' helper to check for configured
submodules instead of manually checking for the submodule's URL in the
config.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 5fe7e23b1..f38e332c5 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -759,7 +759,6 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 	struct strbuf displaypath_sb = STRBUF_INIT;
 	struct strbuf sb = STRBUF_INIT;
 	const char *displaypath = NULL;
-	char *url = NULL;
 	int needs_cloning = 0;
 
 	if (ce_stage(ce)) {
@@ -793,15 +792,8 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		goto cleanup;
 	}
 
-	/*
-	 * Looking up the url in .git/config.
-	 * We must not fall back to .gitmodules as we only want
-	 * to process configured submodules.
-	 */
-	strbuf_reset(&sb);
-	strbuf_addf(&sb, "submodule.%s.url", sub->name);
-	git_config_get_string(sb.buf, &url);
-	if (!url) {
+	/* Check if the submodule has been initialized. */
+	if (!is_submodule_initialized(ce->name)) {
 		next_submodule_warn_missing(suc, out, displaypath);
 		goto cleanup;
 	}
@@ -835,7 +827,7 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		argv_array_push(&child->args, "--depth=1");
 	argv_array_pushl(&child->args, "--path", sub->path, NULL);
 	argv_array_pushl(&child->args, "--name", sub->name, NULL);
-	argv_array_pushl(&child->args, "--url", url, NULL);
+	argv_array_pushl(&child->args, "--url", sub->url, NULL);
 	if (suc->references.nr) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, &suc->references)
@@ -845,7 +837,6 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		argv_array_push(&child->args, suc->depth);
 
 cleanup:
-	free(url);
 	strbuf_reset(&displaypath_sb);
 	strbuf_reset(&sb);
 
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v4 06/10] submodule: decouple url and submodule interest

[Brandon Williams]

Currently the submodule.<name>.url config option is used to determine if
a given submodule is of interest to the user.  This ends up being
cumbersome in a world where we want to have different submodules checked
out in different worktrees or a more generalized mechanism to select
which submodules are of interest.

In a future with worktree support for submodules, there will be multiple
working trees, each of which may only need a subset of the submodules
checked out.  The URL (which is where the submodule repository can be
obtained) should not differ between different working trees.

It may also be convenient for users to more easily specify groups of
submodules they are interested in as apposed to running "git submodule
init <path>" on each submodule they want checked out in their working
tree.

To this end two config options are introduced, submodule.active and
submodule.<name>.active.  The submodule.active config holds a pathspec
that specifies which submodules should exist in the working tree.  The
submodule.<name>.active config is a boolean flag used to indicate if
that particular submodule should exist in the working tree.

Its important to note that submodule.active functions differently than
the other configuration options since it takes a pathspec.  This allows
users to adopt at least two new workflows:

  1. Submodules can be grouped with a leading directory, such that a
     pathspec e.g. 'lib/' would cover all library-ish modules to allow
     those who are interested in library-ish modules to set
     "submodule.active = lib/" just once to say any and all modules in
     'lib/' are interesting.

  2. Once the pathspec-attribute feature is invented, users can label
     submodules with attributes to group them, so that a broad pathspec
     with attribute requirements, e.g. ':(attr:lib)', can be used to say
     any and all modules with the 'lib' attribute are interesting.
     Since the .gitattributes file, just like the .gitmodules file, is
     tracked by the superproject, when a submodule moves in the
     superproject tree, the project can adjust which path gets the
     attribute in .gitattributes, just like it can adjust which path has
     the submodule in .gitmodules.

Neither of these two additional configuration options solve the problem
of wanting different submodules checked out in different worktrees
because multiple worktrees share .git/config.  Only once per-worktree
configurations become a reality can this be solved, but this is a
necessary preparatory step for that future.

Given these multiple ways to check if a submodule is of interest, the
more fine-grained submodule.<name>.active option has the highest order
of precedence followed by the pathspec check against submodule.active.
To ensure backwards compatibility, if neither of these options are set,
git falls back to checking the submodule.<name>.url option to determine
if a submodule is interesting.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/config.txt       | 15 ++++++++++--
 submodule.c                    | 50 ++++++++++++++++++++++++++++++++------
 t/t7413-submodule-is-active.sh | 55 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 110 insertions(+), 10 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 5e5c2ae5f..d2d79b9d4 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -2920,8 +2920,9 @@ submodule.<name>.url::
 	The URL for a submodule. This variable is copied from the .gitmodules
 	file to the git config via 'git submodule init'. The user can change
 	the configured URL before obtaining the submodule via 'git submodule
-	update'. After obtaining the submodule, the presence of this variable
-	is used as a sign whether the submodule is of interest to git commands.
+	update'. If neither submodule.<name>.active or submodule.active are
+	set, the presence of this variable is used as a fallback to indicate
+	whether the submodule is of interest to git commands.
 	See linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.
 
 submodule.<name>.update::
@@ -2959,6 +2960,16 @@ submodule.<name>.ignore::
 	"--ignore-submodules" option. The 'git submodule' commands are not
 	affected by this setting.
 
+submodule.<name>.active::
+	Boolean value indicating if the submodule is of interest to git
+	commands.  This config option takes precedence over the
+	submodule.active config option.
+
+submodule.active::
+	A repeated field which contains a pathspec used to match against a
+	submodule's path to determine if the submodule is of interest to git
+	commands.
+
 submodule.fetchJobs::
 	Specifies how many submodules are fetched/cloned at the same time.
 	A positive integer allows up to that number of submodules fetched
diff --git a/submodule.c b/submodule.c
index 0a2831d84..ad2779ee7 100644
--- a/submodule.c
+++ b/submodule.c
@@ -212,25 +212,59 @@ void gitmodules_config_sha1(const unsigned char *commit_sha1)
 }
 
 /*
+ * NEEDSWORK: With the addition of different configuration options to determine
+ * if a submodule is of interests, the validity of this function's name comes
+ * into question.  Once the dust has settled and more concrete terminology is
+ * decided upon, come up with a more proper name for this function.  One
+ * potential candidate could be 'is_submodule_active()'.
+ *
  * Determine if a submodule has been initialized at a given 'path'
  */
 int is_submodule_initialized(const char *path)
 {
 	int ret = 0;
-	const struct submodule *module = NULL;
+	char *key = NULL;
+	char *value = NULL;
+	const struct string_list *sl;
+	const struct submodule *module = submodule_from_path(null_sha1, path);
 
-	module = submodule_from_path(null_sha1, path);
+	/* early return if there isn't a path->module mapping */
+	if (!module)
+		return 0;
 
-	if (module) {
-		char *key = xstrfmt("submodule.%s.url", module->name);
-		char *value = NULL;
+	/* submodule.<name>.active is set */
+	key = xstrfmt("submodule.%s.active", module->name);
+	if (!git_config_get_bool(key, &ret)) {
+		free(key);
+		return ret;
+	}
+	free(key);
 
-		ret = !git_config_get_string(key, &value);
+	/* submodule.active is set */
+	sl = git_config_get_value_multi("submodule.active");
+	if (sl) {
+		struct pathspec ps;
+		struct argv_array args = ARGV_ARRAY_INIT;
+		const struct string_list_item *item;
 
-		free(value);
-		free(key);
+		for_each_string_list_item(item, sl) {
+			argv_array_push(&args, item->string);
+		}
+
+		parse_pathspec(&ps, 0, 0, NULL, args.argv);
+		ret = match_pathspec(&ps, path, strlen(path), 0, NULL, 1);
+
+		argv_array_clear(&args);
+		clear_pathspec(&ps);
+		return ret;
 	}
 
+	/* fallback to checking if the URL is set */
+	key = xstrfmt("submodule.%s.url", module->name);
+	ret = !git_config_get_string(key, &value);
+
+	free(value);
+	free(key);
 	return ret;
 }
 
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
index f18e0c925..ea1644b58 100755
--- a/t/t7413-submodule-is-active.sh
+++ b/t/t7413-submodule-is-active.sh
@@ -28,4 +28,59 @@ test_expect_success 'is-active works with urls' '
 	git -C super submodule--helper is-active sub1
 '
 
+test_expect_success 'is-active works with submodule.<name>.active config' '
+	test_when_finished "git -C super config --unset submodule.sub1.active" &&
+	test_when_finished "git -C super config submodule.sub1.URL ../sub" &&
+
+	git -C super config --bool submodule.sub1.active "false" &&
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+
+	git -C super config --bool submodule.sub1.active "true" &&
+	git -C super config --unset submodule.sub1.URL &&
+	git -C super submodule--helper is-active sub1
+'
+
+test_expect_success 'is-active works with basic submodule.active config' '
+	test_when_finished "git -C super config submodule.sub1.URL ../sub" &&
+	test_when_finished "git -C super config --unset-all submodule.active" &&
+
+	git -C super config --add submodule.active "." &&
+	git -C super config --unset submodule.sub1.URL &&
+
+	git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2
+'
+
+test_expect_success 'is-active correctly works with paths that are not submodules' '
+	test_when_finished "git -C super config --unset-all submodule.active" &&
+
+	test_must_fail git -C super submodule--helper is-active not-a-submodule &&
+
+	git -C super config --add submodule.active "." &&
+	test_must_fail git -C super submodule--helper is-active not-a-submodule
+'
+
+test_expect_success 'is-active works with exclusions in submodule.active config' '
+	test_when_finished "git -C super config --unset-all submodule.active" &&
+
+	git -C super config --add submodule.active "." &&
+	git -C super config --add submodule.active ":(exclude)sub1" &&
+
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2
+'
+
+test_expect_success 'is-active with submodule.active and submodule.<name>.active' '
+	test_when_finished "git -C super config --unset-all submodule.active" &&
+	test_when_finished "git -C super config --unset submodule.sub1.active" &&
+	test_when_finished "git -C super config --unset submodule.sub2.active" &&
+
+	git -C super config --add submodule.active "sub1" &&
+	git -C super config --bool submodule.sub1.active "false" &&
+	git -C super config --bool submodule.sub2.active "true" &&
+
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2
+'
+
 test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

Re: [PATCH v4 06/10] submodule: decouple url and submodule interest

[Stefan Beller]

On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
> Currently the submodule.<name>.url config option is used to determine if
> a given submodule is of interest to the user.  This ends up being
> cumbersome in a world where we want to have different submodules checked
> out in different worktrees or a more generalized mechanism to select
> which submodules are of interest.
>
> In a future with worktree support for submodules, there will be multiple
> working trees, each of which may only need a subset of the submodules
> checked out.  The URL (which is where the submodule repository can be
> obtained) should not differ between different working trees.
>
> It may also be convenient for users to more easily specify groups of
> submodules they are interested in as apposed to running "git submodule

Did you mean opposed?

> init <path>" on each submodule they want checked out in their working
> tree.
>
> To this end two config options are introduced, submodule.active and
> submodule.<name>.active.  The submodule.active config holds a pathspec
> that specifies which submodules should exist in the working tree.  The
> submodule.<name>.active config is a boolean flag used to indicate if
> that particular submodule should exist in the working tree.
>
> Its important to note that submodule.active functions differently than
> the other configuration options since it takes a pathspec.  This allows
> users to adopt at least two new workflows:
>
>   1. Submodules can be grouped with a leading directory, such that a
>      pathspec e.g. 'lib/' would cover all library-ish modules to allow
>      those who are interested in library-ish modules to set
>      "submodule.active = lib/" just once to say any and all modules in
>      'lib/' are interesting.
>
>   2. Once the pathspec-attribute feature is invented, users can label
>      submodules with attributes to group them, so that a broad pathspec
>      with attribute requirements, e.g. ':(attr:lib)', can be used to say
>      any and all modules with the 'lib' attribute are interesting.
>      Since the .gitattributes file, just like the .gitmodules file, is
>      tracked by the superproject, when a submodule moves in the
>      superproject tree, the project can adjust which path gets the
>      attribute in .gitattributes, just like it can adjust which path has
>      the submodule in .gitmodules.
>
> Neither of these two additional configuration options solve the problem
> of wanting different submodules checked out in different worktrees
> because multiple worktrees share .git/config.  Only once per-worktree
> configurations become a reality can this be solved, but this is a
> necessary preparatory step for that future.
>
> Given these multiple ways to check if a submodule is of interest, the
> more fine-grained submodule.<name>.active option has the highest order
> of precedence followed by the pathspec check against submodule.active.
> To ensure backwards compatibility, if neither of these options are set,
> git falls back to checking the submodule.<name>.url option to determine
> if a submodule is interesting.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  Documentation/config.txt       | 15 ++++++++++--
>  submodule.c                    | 50 ++++++++++++++++++++++++++++++++------
>  t/t7413-submodule-is-active.sh | 55 ++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 110 insertions(+), 10 deletions(-)
>
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 5e5c2ae5f..d2d79b9d4 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -2920,8 +2920,9 @@ submodule.<name>.url::
>         The URL for a submodule. This variable is copied from the .gitmodules
>         file to the git config via 'git submodule init'. The user can change
>         the configured URL before obtaining the submodule via 'git submodule
> -       update'. After obtaining the submodule, the presence of this variable
> -       is used as a sign whether the submodule is of interest to git commands.
> +       update'. If neither submodule.<name>.active or submodule.active are
> +       set, the presence of this variable is used as a fallback to indicate
> +       whether the submodule is of interest to git commands.
>         See linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.
>
>  submodule.<name>.update::
> @@ -2959,6 +2960,16 @@ submodule.<name>.ignore::
>         "--ignore-submodules" option. The 'git submodule' commands are not
>         affected by this setting.
>
> +submodule.<name>.active::
> +       Boolean value indicating if the submodule is of interest to git
> +       commands.  This config option takes precedence over the
> +       submodule.active config option.
> +
> +submodule.active::
> +       A repeated field which contains a pathspec used to match against a
> +       submodule's path to determine if the submodule is of interest to git
> +       commands.
> +
>  submodule.fetchJobs::
>         Specifies how many submodules are fetched/cloned at the same time.
>         A positive integer allows up to that number of submodules fetched
> diff --git a/submodule.c b/submodule.c
> index 0a2831d84..ad2779ee7 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -212,25 +212,59 @@ void gitmodules_config_sha1(const unsigned char *commit_sha1)
>  }
>
>  /*
> + * NEEDSWORK: With the addition of different configuration options to determine
> + * if a submodule is of interests, the validity of this function's name comes
> + * into question.  Once the dust has settled and more concrete terminology is
> + * decided upon, come up with a more proper name for this function.  One
> + * potential candidate could be 'is_submodule_active()'.
> + *
>   * Determine if a submodule has been initialized at a given 'path'
>   */
>  int is_submodule_initialized(const char *path)
>  {
>         int ret = 0;
> -       const struct submodule *module = NULL;
> +       char *key = NULL;
> +       char *value = NULL;
> +       const struct string_list *sl;
> +       const struct submodule *module = submodule_from_path(null_sha1, path);
>
> -       module = submodule_from_path(null_sha1, path);
> +       /* early return if there isn't a path->module mapping */
> +       if (!module)
> +               return 0;
>
> -       if (module) {
> -               char *key = xstrfmt("submodule.%s.url", module->name);
> -               char *value = NULL;
> +       /* submodule.<name>.active is set */
> +       key = xstrfmt("submodule.%s.active", module->name);
> +       if (!git_config_get_bool(key, &ret)) {
> +               free(key);
> +               return ret;
> +       }
> +       free(key);
>
> -               ret = !git_config_get_string(key, &value);
> +       /* submodule.active is set */
> +       sl = git_config_get_value_multi("submodule.active");
> +       if (sl) {
> +               struct pathspec ps;
> +               struct argv_array args = ARGV_ARRAY_INIT;
> +               const struct string_list_item *item;
>
> -               free(value);
> -               free(key);
> +               for_each_string_list_item(item, sl) {
> +                       argv_array_push(&args, item->string);
> +               }
> +
> +               parse_pathspec(&ps, 0, 0, NULL, args.argv);
> +               ret = match_pathspec(&ps, path, strlen(path), 0, NULL, 1);
> +
> +               argv_array_clear(&args);
> +               clear_pathspec(&ps);
> +               return ret;
>         }
>
> +       /* fallback to checking if the URL is set */
> +       key = xstrfmt("submodule.%s.url", module->name);
> +       ret = !git_config_get_string(key, &value);
> +
> +       free(value);
> +       free(key);
>         return ret;
>  }
>
> diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
> index f18e0c925..ea1644b58 100755
> --- a/t/t7413-submodule-is-active.sh
> +++ b/t/t7413-submodule-is-active.sh
> @@ -28,4 +28,59 @@ test_expect_success 'is-active works with urls' '
>         git -C super submodule--helper is-active sub1
>  '
>
> +test_expect_success 'is-active works with submodule.<name>.active config' '
> +       test_when_finished "git -C super config --unset submodule.sub1.active" &&

Sorry for late notice: This can be solved more elegantly with
test_config[_global].
If this is the only nit, we can fix it later, though.

> +       test_when_finished "git -C super config submodule.sub1.URL ../sub" &&

All tests look like they are structured very similarly; we also want
to test all combinations,
so maybe it is easier to make a function

test_active_case()
{
    on_submodule=$1
    expectation=$2
    with_URL=$3
    with_active_flag=$4
    with_active_pathspec=$5
    test_expect_success 'is_active for $1 $2 $3 $4 $5' '
      # do the actual test
    '
}

# here put all combinations for a submodule + the non-submodule case

I am not sure though, it would be a more disciplined approach, but not
necessarily
be covering the corner cases just as nice, i.e. having multiple
pathspecs, including 'exclude's.

Just food for thought.

Thanks,
Stefan

[PATCH v4 07/10] submodule init: initialize active submodules

[Brandon Williams]

Teach `submodule init` to initialize submodules which have been
configured to be active by setting 'submodule.active' with a pathspec.

Now if no path arguments are given and 'submodule.active' is configured,
`init` will initialize all submodules which have been configured to be
active.  If no path arguments are given and 'submodule.active' is not
configured, then `init` will retain the old behavior of initializing all
submodules.

This allows users to record more complex patterns as it saves retyping
them whenever you invoke update.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/git-submodule.txt |  4 ++-
 builtin/submodule--helper.c     | 36 +++++++++++++++++++++++++-
 t/t7400-submodule-basic.sh      | 57 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 95 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index e05d0cdde..74bc6200d 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -129,7 +129,9 @@ init [--] [<path>...]::
 	repository will be assumed to be upstream.
 +
 Optional <path> arguments limit which submodules will be initialized.
-If no path is specified, all submodules are initialized.
+If no path is specified and submodule.active has been configured, submodules
+configured to be active will be initialized, otherwise all submodules are
+initialized.
 +
 When present, it will also copy the value of `submodule.$name.update`.
 This command does not alter existing information in .git/config.
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index f38e332c5..e95738b42 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -270,6 +270,34 @@ static int module_list_compute(int argc, const char **argv,
 	return result;
 }
 
+static void module_list_active(struct module_list *list)
+{
+	int i;
+
+	if (read_cache() < 0)
+		die(_("index file corrupt"));
+
+	gitmodules_config();
+
+	for (i = 0; i < active_nr; i++) {
+		const struct cache_entry *ce = active_cache[i];
+
+		if (!S_ISGITLINK(ce->ce_mode) ||
+		    !is_submodule_initialized(ce->name))
+			continue;
+
+		ALLOC_GROW(list->entries, list->nr + 1, list->alloc);
+		list->entries[list->nr++] = ce;
+		while (i + 1 < active_nr &&
+		       !strcmp(ce->name, active_cache[i + 1]->name))
+			/*
+			 * Skip entries with the same name in different stages
+			 * to make sure an entry is returned only once.
+			 */
+			i++;
+	}
+}
+
 static int module_list(int argc, const char **argv, const char *prefix)
 {
 	int i;
@@ -417,7 +445,13 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix, module_init_options,
 			     git_submodule_helper_usage, 0);
 
-	if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
+	/*
+	 * If there are no path args and submodule.active is set then,
+	 * by default, only initialize 'active' modules.
+	 */
+	if (!argc && git_config_get_value_multi("submodule.active"))
+		module_list_active(&list);
+	else if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
 		return 1;
 
 	for (i = 0; i < list.nr; i++)
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index c09ce0d4c..fbbe932d1 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1130,5 +1130,62 @@ test_expect_success 'submodule helper list is not confused by common prefixes' '
 	test_cmp expect actual
 '
 
+test_expect_success 'setup superproject with submodules' '
+	git init sub1 &&
+	test_commit -C sub1 test &&
+	test_commit -C sub1 test2 &&
+	git init multisuper &&
+	git -C multisuper submodule add ../sub1 sub0 &&
+	git -C multisuper submodule add ../sub1 sub1 &&
+	git -C multisuper submodule add ../sub1 sub2 &&
+	git -C multisuper submodule add ../sub1 sub3 &&
+	git -C multisuper commit -m "add some submodules"
+'
+
+cat >expect <<-EOF
+-sub0
+ sub1 (test2)
+ sub2 (test2)
+ sub3 (test2)
+EOF
+
+test_expect_success 'submodule update --init with a specification' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone submodule update --init . ":(exclude)sub0" &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'submodule update --init with submodule.active set' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone config submodule.active "." &&
+	git -C multisuper_clone config --add submodule.active ":(exclude)sub0" &&
+	git -C multisuper_clone submodule update --init &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'submodule update and setting submodule.<name>.active' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone config --bool submodule.sub0.active "true" &&
+	git -C multisuper_clone config --bool submodule.sub1.active "false" &&
+	git -C multisuper_clone config --bool submodule.sub2.active "true" &&
+
+	cat >expect <<-\EOF &&
+	 sub0 (test2)
+	-sub1
+	 sub2 (test2)
+	-sub3
+	EOF
+	git -C multisuper_clone submodule update &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_cmp expect actual
+'
 
 test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

Re: [PATCH v4 07/10] submodule init: initialize active submodules

[Stefan Beller]

On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
> Teach `submodule init` to initialize submodules which have been
> configured to be active by setting 'submodule.active' with a pathspec.
>
> Now if no path arguments are given and 'submodule.active' is configured,
> `init` will initialize all submodules which have been configured to be
> active.  If no path arguments are given and 'submodule.active' is not
> configured, then `init` will retain the old behavior of initializing all
> submodules.
>
> This allows users to record more complex patterns as it saves retyping
> them whenever you invoke update.
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---


> @@ -417,7 +445,13 @@ static int module_init(int argc, const char **argv, const char *prefix)
>         argc = parse_options(argc, argv, prefix, module_init_options,
>                              git_submodule_helper_usage, 0);
>
> -       if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
> +       /*
> +        * If there are no path args and submodule.active is set then,
> +        * by default, only initialize 'active' modules.
> +        */
> +       if (!argc && git_config_get_value_multi("submodule.active"))
> +               module_list_active(&list);
> +       else if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
>                 return 1;

I would rather reuse module_list_compute and then post-process the list
to filter out inactive submodules iff "submodule.active" is set as that seems
cleaner and performance is not a pressing issue here?

>
> +static void module_list_active(struct module_list *list)
> +{
> +       int i;
> +
> +       if (read_cache() < 0)
> +               die(_("index file corrupt"));
> +
> +       gitmodules_config();

Here we also need to have

  git_config(submodule_config, NULL);

such that is_submodule_initialized works correctly,
I would assume?

> +
> +       for (i = 0; i < active_nr; i++) {
> +               const struct cache_entry *ce = active_cache[i];
> +
> +               if (!S_ISGITLINK(ce->ce_mode) ||
> +                   !is_submodule_initialized(ce->name))
> +                       continue;
> +
> +               ALLOC_GROW(list->entries, list->nr + 1, list->alloc);
> +               list->entries[list->nr++] = ce;
> +               while (i + 1 < active_nr &&
> +                      !strcmp(ce->name, active_cache[i + 1]->name))
> +                       /*
> +                        * Skip entries with the same name in different stages
> +                        * to make sure an entry is returned only once.
> +                        */
> +                       i++;
> +       }
> +}

Re: [PATCH v4 07/10] submodule init: initialize active submodules

[Brandon Williams]

On 03/17, Stefan Beller wrote:
> On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
> > Teach `submodule init` to initialize submodules which have been
> > configured to be active by setting 'submodule.active' with a pathspec.
> >
> > Now if no path arguments are given and 'submodule.active' is configured,
> > `init` will initialize all submodules which have been configured to be
> > active.  If no path arguments are given and 'submodule.active' is not
> > configured, then `init` will retain the old behavior of initializing all
> > submodules.
> >
> > This allows users to record more complex patterns as it saves retyping
> > them whenever you invoke update.
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> 
> 
> > @@ -417,7 +445,13 @@ static int module_init(int argc, const char **argv, const char *prefix)
> >         argc = parse_options(argc, argv, prefix, module_init_options,
> >                              git_submodule_helper_usage, 0);
> >
> > -       if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
> > +       /*
> > +        * If there are no path args and submodule.active is set then,
> > +        * by default, only initialize 'active' modules.
> > +        */
> > +       if (!argc && git_config_get_value_multi("submodule.active"))
> > +               module_list_active(&list);
> > +       else if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
> >                 return 1;
> 
> I would rather reuse module_list_compute and then post-process the list
> to filter out inactive submodules iff "submodule.active" is set as that seems
> cleaner and performance is not a pressing issue here?

Ok, Can do.  Shouldn't be very hard to do that.

> 
> >
> > +static void module_list_active(struct module_list *list)
> > +{
> > +       int i;
> > +
> > +       if (read_cache() < 0)
> > +               die(_("index file corrupt"));
> > +
> > +       gitmodules_config();
> 
> Here we also need to have
> 
>   git_config(submodule_config, NULL);
> 
> such that is_submodule_initialized works correctly,
> I would assume?

No I don't think so.  is_submodule_initialized doesn't need them to be
overlayed, it just needs the .gitmodules mappings.

> 
> > +
> > +       for (i = 0; i < active_nr; i++) {
> > +               const struct cache_entry *ce = active_cache[i];
> > +
> > +               if (!S_ISGITLINK(ce->ce_mode) ||
> > +                   !is_submodule_initialized(ce->name))
> > +                       continue;
> > +
> > +               ALLOC_GROW(list->entries, list->nr + 1, list->alloc);
> > +               list->entries[list->nr++] = ce;
> > +               while (i + 1 < active_nr &&
> > +                      !strcmp(ce->name, active_cache[i + 1]->name))
> > +                       /*
> > +                        * Skip entries with the same name in different stages
> > +                        * to make sure an entry is returned only once.
> > +                        */
> > +                       i++;
> > +       }
> > +}

-- 
Brandon Williams

Re: [PATCH v4 07/10] submodule init: initialize active submodules

[Stefan Beller]

>> Here we also need to have
>>
>>   git_config(submodule_config, NULL);
>>
>> such that is_submodule_initialized works correctly,
>> I would assume?
>
> No I don't think so.  is_submodule_initialized doesn't need them to be
> overlayed, it just needs the .gitmodules mappings.

ok, thanks! I assumed it would need it later on, when e.g. looking at
"submodule.active" from the config.

Thanks,
Stefan

[PATCH v4 08/10] clone: teach --recurse-submodules to optionally take a pathspec

[Brandon Williams]

Teach clone --recurse-submodules to optionally take a pathspec argument
which describes which submodules should be recursively initialized and
cloned.  If no pathspec is provided, --recurse-submodules will
recursively initialize and clone all submodules by using a default
pathspec of ".".  In order to construct more complex pathspecs,
--recurse-submodules can be given multiple times.

Additionally this configures the 'submodule.active' configuration option
to be the given pathspec, such that any future invocation of `git
submodule update` will keep up with the pathspec.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/git-clone.txt | 14 ++++++----
 builtin/clone.c             | 47 ++++++++++++++++++++++++++-----
 t/t7400-submodule-basic.sh  | 68 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 117 insertions(+), 12 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 35cc34b2f..30052cce4 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -14,7 +14,7 @@ SYNOPSIS
 	  [-o <name>] [-b <name>] [-u <upload-pack>] [--reference <repository>]
 	  [--dissociate] [--separate-git-dir <git dir>]
 	  [--depth <depth>] [--[no-]single-branch]
-	  [--recursive | --recurse-submodules] [--[no-]shallow-submodules]
+	  [--recurse-submodules] [--[no-]shallow-submodules]
 	  [--jobs <n>] [--] <repository> [<directory>]
 
 DESCRIPTION
@@ -215,10 +215,14 @@ objects from the source repository into a pack in the cloned repository.
 	branch when `--single-branch` clone was made, no remote-tracking
 	branch is created.
 
---recursive::
---recurse-submodules::
-	After the clone is created, initialize all submodules within,
-	using their default settings. This is equivalent to running
+--recurse-submodules[=<pathspec]::
+	After the clone is created, initialize and clone submodules
+	within based on the provided pathspec.  If no pathspec is
+	provided, all submodules are initialized and cloned.
+	Submodules are initialized and cloned using their default
+	settings.  The resulting clone has `submodule.active` set to
+	the provided pathspec, or "." (meaning all submodules) if no
+	pathspec is provided.  This is equivalent to running
 	`git submodule update --init --recursive` immediately after
 	the clone is finished. This option is ignored if the cloned
 	repository does not have a worktree/checkout (i.e. if any of
diff --git a/builtin/clone.c b/builtin/clone.c
index 3f63edbbf..3dc8faac5 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -39,7 +39,7 @@ static const char * const builtin_clone_usage[] = {
 };
 
 static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
-static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
+static int option_local = -1, option_no_hardlinks, option_shared;
 static int option_shallow_submodules;
 static int deepen;
 static char *option_template, *option_depth, *option_since;
@@ -56,6 +56,22 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;
 static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;
 static int option_dissociate;
 static int max_jobs = -1;
+static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;
+
+static int recurse_submodules_cb(const struct option *opt,
+				 const char *arg, int unset)
+{
+	if (unset)
+		return -1;
+
+	if (arg)
+		string_list_append((struct string_list *)opt->value, arg);
+	else
+		string_list_append((struct string_list *)opt->value,
+				   (const char *)opt->defval);
+
+	return 0;
+}
 
 static struct option builtin_clone_options[] = {
 	OPT__VERBOSITY(&option_verbosity),
@@ -74,10 +90,13 @@ static struct option builtin_clone_options[] = {
 		    N_("don't use local hardlinks, always copy")),
 	OPT_BOOL('s', "shared", &option_shared,
 		    N_("setup as shared repository")),
-	OPT_BOOL(0, "recursive", &option_recursive,
-		    N_("initialize submodules in the clone")),
-	OPT_BOOL(0, "recurse-submodules", &option_recursive,
-		    N_("initialize submodules in the clone")),
+	{ OPTION_CALLBACK, 0, "recursive", &option_recurse_submodules,
+	  N_("pathspec"), N_("initialize submodules in the clone"),
+	  PARSE_OPT_OPTARG | PARSE_OPT_HIDDEN, recurse_submodules_cb,
+	  (intptr_t)"." },
+	{ OPTION_CALLBACK, 0, "recurse-submodules", &option_recurse_submodules,
+	  N_("pathspec"), N_("initialize submodules in the clone"),
+	  PARSE_OPT_OPTARG, recurse_submodules_cb, (intptr_t)"." },
 	OPT_INTEGER('j', "jobs", &max_jobs,
 		    N_("number of submodules cloned in parallel")),
 	OPT_STRING(0, "template", &option_template, N_("template-directory"),
@@ -733,7 +752,7 @@ static int checkout(int submodule_progress)
 	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
 			   sha1_to_hex(sha1), "1", NULL);
 
-	if (!err && option_recursive) {
+	if (!err && (option_recurse_submodules.nr > 0)) {
 		struct argv_array args = ARGV_ARRAY_INIT;
 		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
 
@@ -957,7 +976,21 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 			fprintf(stderr, _("Cloning into '%s'...\n"), dir);
 	}
 
-	if (option_recursive) {
+	if (option_recurse_submodules.nr > 0) {
+		struct string_list_item *item;
+		struct strbuf sb = STRBUF_INIT;
+
+		/* remove duplicates */
+		string_list_sort(&option_recurse_submodules);
+		string_list_remove_duplicates(&option_recurse_submodules, 0);
+
+		for_each_string_list_item(item, &option_recurse_submodules) {
+			strbuf_addf(&sb, "submodule.active=%s",
+				    item->string);
+			string_list_append(&option_config,
+					   strbuf_detach(&sb, NULL));
+		}
+
 		if (option_required_reference.nr &&
 		    option_optional_reference.nr)
 			die(_("clone --recursive is not compatible with "
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index fbbe932d1..3af1c00ff 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1188,4 +1188,72 @@ test_expect_success 'submodule update and setting submodule.<name>.active' '
 	test_cmp expect actual
 '
 
+test_expect_success 'clone --recurse-submodules with a pathspec works' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat >expected <<-\EOF &&
+	 sub0 (test2)
+	-sub1
+	-sub2
+	-sub3
+	EOF
+
+	git clone --recurse-submodules="sub0" multisuper multisuper_clone &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp actual expected
+'
+
+test_expect_success 'clone with multiple --recurse-submodules options' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat >expect <<-\EOF &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	EOF
+
+	git clone --recurse-submodules="." \
+		  --recurse-submodules=":(exclude)sub0" \
+		  --recurse-submodules=":(exclude)sub2" \
+		  multisuper multisuper_clone &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'clone and subsequent updates correctly auto-initialize submodules' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat <<-\EOF >expect &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	EOF
+
+	cat <<-\EOF >expect2 &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	-sub4
+	 sub5 (test2)
+	EOF
+
+	git clone --recurse-submodules="." \
+		  --recurse-submodules=":(exclude)sub0" \
+		  --recurse-submodules=":(exclude)sub2" \
+		  --recurse-submodules=":(exclude)sub4" \
+		  multisuper multisuper_clone &&
+
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect actual &&
+
+	git -C multisuper submodule add ../sub1 sub4 &&
+	git -C multisuper submodule add ../sub1 sub5 &&
+	git -C multisuper commit -m "add more submodules" &&
+	# obtain the new superproject
+	git -C multisuper_clone pull &&
+	git -C multisuper_clone submodule update --init &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect2 actual
+'
+
 test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

Re: [PATCH v4 08/10] clone: teach --recurse-submodules to optionally take a pathspec

[Stefan Beller]

On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
> Teach clone --recurse-submodules to optionally take a pathspec argument
> which describes which submodules should be recursively initialized and
> cloned.  If no pathspec is provided, --recurse-submodules will
> recursively initialize and clone all submodules by using a default
> pathspec of ".".  In order to construct more complex pathspecs,
> --recurse-submodules can be given multiple times.
>
> Additionally this configures the 'submodule.active' configuration option
> to be the given pathspec, such that any future invocation of `git
> submodule update` will keep up with the pathspec.

  Additionally the switch '--recurse' is removed from the Documentation
  as well as marked hidden in the options array, to streamline the options
  for submodules.  A simple '--recurse' doesn't convey what is being
  recursed, e.g. it could mean directories or trees (c.f. ls-tree)
  In a lot of other commands we already have '--recurse-submodules'
  to mean recursing into submodules, so advertise this spelling here
  as the genuine option.

would also be worth mentioning?


>  static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
> -static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
> +static int option_local = -1, option_no_hardlinks, option_shared;
>  static int option_shallow_submodules;
>  static int deepen;
>  static char *option_template, *option_depth, *option_since;
> @@ -56,6 +56,22 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;
>  static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;
>  static int option_dissociate;
>  static int max_jobs = -1;
> +static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;
> +
> +static int recurse_submodules_cb(const struct option *opt,
> +                                const char *arg, int unset)
> +{
> +       if (unset)
> +               return -1;
> +
> +       if (arg)
> +               string_list_append((struct string_list *)opt->value, arg);
> +       else

in this case I'd rather set the removed (int) option_recursive, because, then
we would not need to sort and remove duplicates later on.
Instead we can pass the string list literally to the config setter.
(and in case option_recursive is set, we add an additional single
"." then)

> +               string_list_append((struct string_list *)opt->value,
> +                                  (const char *)opt->defval);
> +
> +       return 0;
> +}
>

>
> -       if (!err && option_recursive) {
> +       if (!err && (option_recurse_submodules.nr > 0)) {

Well, checks like these would become more tangled.
So maybe we could set option_recursive unconditionally
in the callback (unless unset was given, then we reset it to 0)
and later have a check if we need to add "." (when the string list
is empty).

Speaking of unset, this seems like a regression here, as the callback
would error out to "git clone --no-recurse", which is a valid use case
currently? (Searching for "git clone --no-recurse" yields no results
via Google, so maybe this use case is not so valid)

To get the behavior as is for unset we could just clear the string
list instead of returning -1.

Thanks,
Stefan

Re: [PATCH v4 08/10] clone: teach --recurse-submodules to optionally take a pathspec

[Brandon Williams]

On 03/17, Stefan Beller wrote:
> On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
> > Teach clone --recurse-submodules to optionally take a pathspec argument
> > which describes which submodules should be recursively initialized and
> > cloned.  If no pathspec is provided, --recurse-submodules will
> > recursively initialize and clone all submodules by using a default
> > pathspec of ".".  In order to construct more complex pathspecs,
> > --recurse-submodules can be given multiple times.
> >
> > Additionally this configures the 'submodule.active' configuration option
> > to be the given pathspec, such that any future invocation of `git
> > submodule update` will keep up with the pathspec.
> 
>   Additionally the switch '--recurse' is removed from the Documentation
>   as well as marked hidden in the options array, to streamline the options
>   for submodules.  A simple '--recurse' doesn't convey what is being
>   recursed, e.g. it could mean directories or trees (c.f. ls-tree)
>   In a lot of other commands we already have '--recurse-submodules'
>   to mean recursing into submodules, so advertise this spelling here
>   as the genuine option.
> 
> would also be worth mentioning?
> 

Yeah I can just add this into the commit msg.

> 
> >  static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
> > -static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
> > +static int option_local = -1, option_no_hardlinks, option_shared;
> >  static int option_shallow_submodules;
> >  static int deepen;
> >  static char *option_template, *option_depth, *option_since;
> > @@ -56,6 +56,22 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;
> >  static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;
> >  static int option_dissociate;
> >  static int max_jobs = -1;
> > +static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;
> > +
> > +static int recurse_submodules_cb(const struct option *opt,
> > +                                const char *arg, int unset)
> > +{
> > +       if (unset)
> > +               return -1;
> > +
> > +       if (arg)
> > +               string_list_append((struct string_list *)opt->value, arg);
> > +       else
> 
> in this case I'd rather set the removed (int) option_recursive, because, then
> we would not need to sort and remove duplicates later on.
> Instead we can pass the string list literally to the config setter.
> (and in case option_recursive is set, we add an additional single
> "." then)

That's just one more thing to worry about though.  This felt a little
bit cleaner than doing more special casing.

> 
> > +               string_list_append((struct string_list *)opt->value,
> > +                                  (const char *)opt->defval);
> > +
> > +       return 0;
> > +}
> >
> 
> >
> > -       if (!err && option_recursive) {
> > +       if (!err && (option_recurse_submodules.nr > 0)) {
> 
> Well, checks like these would become more tangled.
> So maybe we could set option_recursive unconditionally
> in the callback (unless unset was given, then we reset it to 0)
> and later have a check if we need to add "." (when the string list
> is empty).
> 
> Speaking of unset, this seems like a regression here, as the callback
> would error out to "git clone --no-recurse", which is a valid use case
> currently? (Searching for "git clone --no-recurse" yields no results
> via Google, so maybe this use case is not so valid)
> 
> To get the behavior as is for unset we could just clear the string
> list instead of returning -1.

Yeah I can do that.

> 
> Thanks,
> Stefan

-- 
Brandon Williams

[PATCH v4 09/10] submodule--helper init: set submodule.<name>.active

[Brandon Williams]

When initializing a submodule set the submodule.<name>.active config to
true if the module hasn't already been configured to be active by some
other means (e.g. a pathspec set in submodule.active).

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c |  7 +++++++
 t/t7400-submodule-basic.sh  | 11 +++++++++++
 2 files changed, 18 insertions(+)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index e95738b42..a574596cb 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -361,6 +361,13 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 		die(_("No url found for submodule path '%s' in .gitmodules"),
 			displaypath);
 
+	/* Set active flag for the submodule being initialized */
+	if (!is_submodule_initialized(path)) {
+		strbuf_reset(&sb);
+		strbuf_addf(&sb, "submodule.%s.active", sub->name);
+		git_config_set_gently(sb.buf, "true");
+	}
+
 	/*
 	 * Copy url setting when it is not set yet.
 	 * To look up the url in .git/config, we must not fall back to
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 3af1c00ff..cf77a3a35 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1256,4 +1256,15 @@ test_expect_success 'clone and subsequent updates correctly auto-initialize subm
 	test_cmp expect2 actual
 '
 
+test_expect_success 'init properly sets the config' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	git clone --recurse-submodules="." \
+		  --recurse-submodules=":(exclude)sub0" \
+		  multisuper multisuper_clone &&
+
+	git -C multisuper_clone submodule init -- sub0 sub1 &&
+	git -C multisuper_clone config --get submodule.sub0.active &&
+	test_must_fail git -C multisuper_clone config --get submodule.sub1.active
+'
+
 test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

Re: [PATCH v4 09/10] submodule--helper init: set submodule.<name>.active

[Stefan Beller]

On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
> When initializing a submodule set the submodule.<name>.active config to
> true if the module hasn't already been configured to be active by some
> other means (e.g. a pathspec set in submodule.active).
>
> Signed-off-by: Brandon Williams <bmwill@google.com>
> ---
>  builtin/submodule--helper.c |  7 +++++++
>  t/t7400-submodule-basic.sh  | 11 +++++++++++
>  2 files changed, 18 insertions(+)
>
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index e95738b42..a574596cb 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -361,6 +361,13 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
>                 die(_("No url found for submodule path '%s' in .gitmodules"),
>                         displaypath);
>
> +       /* Set active flag for the submodule being initialized */
> +       if (!is_submodule_initialized(path)) {
> +               strbuf_reset(&sb);
> +               strbuf_addf(&sb, "submodule.%s.active", sub->name);

In case a reroll is needed, you could mark this location with

    /*
     * NEEDSWORK: in a multi-working-tree world we need to set
     * this per-worktree here.
     */

Re: [PATCH v4 09/10] submodule--helper init: set submodule.<name>.active

[Brandon Williams]

On 03/17, Stefan Beller wrote:
> On Thu, Mar 16, 2017 at 3:29 PM, Brandon Williams <bmwill@google.com> wrote:
> > When initializing a submodule set the submodule.<name>.active config to
> > true if the module hasn't already been configured to be active by some
> > other means (e.g. a pathspec set in submodule.active).
> >
> > Signed-off-by: Brandon Williams <bmwill@google.com>
> > ---
> >  builtin/submodule--helper.c |  7 +++++++
> >  t/t7400-submodule-basic.sh  | 11 +++++++++++
> >  2 files changed, 18 insertions(+)
> >
> > diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> > index e95738b42..a574596cb 100644
> > --- a/builtin/submodule--helper.c
> > +++ b/builtin/submodule--helper.c
> > @@ -361,6 +361,13 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
> >                 die(_("No url found for submodule path '%s' in .gitmodules"),
> >                         displaypath);
> >
> > +       /* Set active flag for the submodule being initialized */
> > +       if (!is_submodule_initialized(path)) {
> > +               strbuf_reset(&sb);
> > +               strbuf_addf(&sb, "submodule.%s.active", sub->name);
> 
> In case a reroll is needed, you could mark this location with
> 
>     /*
>      * NEEDSWORK: in a multi-working-tree world we need to set
>      * this per-worktree here.
>      */

I'll add this here and where this happens in clone.

-- 
Brandon Williams

[PATCH v4 10/10] submodule add: respect submodule.active and submodule.<name>.active

[Brandon Williams]

In addition to adding submodule.<name>.url to the config, set
submodule.<name>.active to true unless submodule.active is configured
and the submodule's path matches the configured pathspec.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh               | 12 ++++++++++++
 t/t7413-submodule-is-active.sh | 21 +++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/git-submodule.sh b/git-submodule.sh
index db94dea3b..6eca93416 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -278,6 +278,18 @@ or you are unsure what this means choose another name with the '--name' option."
 	fi &&
 	git add --force .gitmodules ||
 	die "$(eval_gettext "Failed to register submodule '\$sm_path'")"
+
+	if git config --get submodule.active >/dev/null
+	then
+		# If the submodule being adding isn't already covered by the
+		# current configured pathspec, set the submodule's active flag
+		if ! git submodule--helper is-active "$sm_path"
+		then
+			git config submodule."$sm_name".active "true"
+		fi
+	else
+		git config submodule."$sm_name".active "true"
+	fi
 }
 
 #
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
index ea1644b58..9c785b07e 100755
--- a/t/t7413-submodule-is-active.sh
+++ b/t/t7413-submodule-is-active.sh
@@ -15,6 +15,12 @@ test_expect_success 'setup' '
 	test_commit -C super initial &&
 	git -C super submodule add ../sub sub1 &&
 	git -C super submodule add ../sub sub2 &&
+
+	# Remove submodule.<name>.active entries in order to test in an
+	# environment where only URLs are present in the conifg
+	git -C super config --unset submodule.sub1.active &&
+	git -C super config --unset submodule.sub2.active &&
+
 	git -C super commit -a -m "add 2 submodules at sub{1,2}"
 '
 
@@ -83,4 +89,19 @@ test_expect_success 'is-active with submodule.active and submodule.<name>.active
 	git -C super submodule--helper is-active sub2
 '
 
+test_expect_success 'is-active, submodule.active and submodule add' '
+	test_when_finished "rm -rf super2" &&
+	git init super2 &&
+	test_commit -C super2 initial &&
+	git -C super2 config --add submodule.active "sub*" &&
+
+	# submodule add should only add submodule.<name>.active
+	# to the config if not matched by the pathspec
+	git -C super2 submodule add ../sub sub1 &&
+	test_must_fail git -C super2 config --get submodule.sub1.active &&
+
+	git -C super2 submodule add ../sub mod &&
+	git -C super2 config --get submodule.mod.active
+'
+
 test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v5 00/10] decoupling url and submodule interest

[Brandon Williams]

Changes in v5:
  * Add "NEEDSWORK" comments to indicate where attention is needed once
    per-worktree config is a reality
  * --no-recurse now works by clearing the string list of paths.
  * module_list_active() now does post-processing instead of duplicating code.

Brandon Williams (10):
  submodule--helper: add is-active subcommand
  submodule status: use submodule--helper is-active
  submodule sync: skip work for inactive submodules
  submodule sync: use submodule--helper is-active
  submodule--helper clone: check for configured submodules using helper
  submodule: decouple url and submodule interest
  submodule init: initialize active submodules
  clone: teach --recurse-submodules to optionally take a pathspec
  submodule--helper init: set submodule.<name>.active
  submodule add: respect submodule.active and submodule.<name>.active

 Documentation/config.txt        |  15 ++++-
 Documentation/git-clone.txt     |  14 +++--
 Documentation/git-submodule.txt |   4 +-
 builtin/clone.c                 |  50 ++++++++++++---
 builtin/submodule--helper.c     |  68 ++++++++++++++++----
 git-submodule.sh                |  55 ++++++++++------
 submodule.c                     |  50 ++++++++++++---
 t/t7400-submodule-basic.sh      | 136 ++++++++++++++++++++++++++++++++++++++++
 t/t7413-submodule-is-active.sh  | 107 +++++++++++++++++++++++++++++++
 9 files changed, 445 insertions(+), 54 deletions(-)
 create mode 100755 t/t7413-submodule-is-active.sh

---interdiff with 'origin/bw/submodule-is-active'


diff --git a/builtin/clone.c b/builtin/clone.c
index 3dc8faac5..a7be61d6b 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -62,9 +62,8 @@ static int recurse_submodules_cb(const struct option *opt,
 				 const char *arg, int unset)
 {
 	if (unset)
-		return -1;
-
-	if (arg)
+		string_list_clear((struct string_list *)opt->value, 0);
+	else if (arg)
 		string_list_append((struct string_list *)opt->value, arg);
 	else
 		string_list_append((struct string_list *)opt->value,
@@ -984,6 +983,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		string_list_sort(&option_recurse_submodules);
 		string_list_remove_duplicates(&option_recurse_submodules, 0);

+		/*
+		 * NEEDSWORK: In a multi-working-tree world, this needs to be
+		 * set in the per-worktree config.
+		 */
 		for_each_string_list_item(item, &option_recurse_submodules) {
 			strbuf_addf(&sb, "submodule.active=%s",
 				    item->string);
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index a574596cb..7700d8948 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -273,29 +273,24 @@ static int module_list_compute(int argc, const char **argv,
 static void module_list_active(struct module_list *list)
 {
 	int i;
-
-	if (read_cache() < 0)
-		die(_("index file corrupt"));
+	struct module_list active_modules = MODULE_LIST_INIT;

 	gitmodules_config();

-	for (i = 0; i < active_nr; i++) {
-		const struct cache_entry *ce = active_cache[i];
+	for (i = 0; i < list->nr; i++) {
+		const struct cache_entry *ce = list->entries[i];

-		if (!S_ISGITLINK(ce->ce_mode) ||
-		    !is_submodule_initialized(ce->name))
+		if (!is_submodule_initialized(ce->name))
 			continue;

-		ALLOC_GROW(list->entries, list->nr + 1, list->alloc);
-		list->entries[list->nr++] = ce;
-		while (i + 1 < active_nr &&
-		       !strcmp(ce->name, active_cache[i + 1]->name))
-			/*
-			 * Skip entries with the same name in different stages
-			 * to make sure an entry is returned only once.
-			 */
-			i++;
+		ALLOC_GROW(active_modules.entries,
+			   active_modules.nr + 1,
+			   active_modules.alloc);
+		active_modules.entries[active_modules.nr++] = ce;
 	}
+
+	free(list->entries);
+	*list = active_modules;
 }

 static int module_list(int argc, const char **argv, const char *prefix)
@@ -361,7 +356,12 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 		die(_("No url found for submodule path '%s' in .gitmodules"),
 			displaypath);

-	/* Set active flag for the submodule being initialized */
+	/*
+	 * NEEDSWORK: In a multi-working-tree world, this needs to be
+	 * set in the per-worktree config.
+	 *
+	 * Set active flag for the submodule being initialized
+	 */
 	if (!is_submodule_initialized(path)) {
 		strbuf_reset(&sb);
 		strbuf_addf(&sb, "submodule.%s.active", sub->name);
@@ -452,14 +452,15 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix, module_init_options,
 			     git_submodule_helper_usage, 0);

+	if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
+		return 1;
+
 	/*
 	 * If there are no path args and submodule.active is set then,
 	 * by default, only initialize 'active' modules.
 	 */
 	if (!argc && git_config_get_value_multi("submodule.active"))
 		module_list_active(&list);
-	else if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
-		return 1;

 	for (i = 0; i < list.nr; i++)
 		init_submodule(list.entries[i]->name, prefix, quiet);
diff --git a/git-submodule.sh b/git-submodule.sh
index 6eca93416..6ec35e5fc 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -279,6 +279,8 @@ or you are unsure what this means choose another name with the '--name' option."
 	git add --force .gitmodules ||
 	die "$(eval_gettext "Failed to register submodule '\$sm_path'")"

+	# NEEDSWORK: In a multi-working-tree world, this needs to be
+	# set in the per-worktree config.
 	if git config --get submodule.active >/dev/null
 	then
 		# If the submodule being adding isn't already covered by the

--
2.12.0.367.g23dc2f6d3c-goog

[PATCH v5 01/10] submodule--helper: add is-active subcommand

[Brandon Williams]

The definition of which submodules are of interest by the user
is tied to the configuration submodule.<name>.url; when it is
set to a non-empty string, it is of interest.  We'd want to be
able to later change this definition, but there are many places
that explicitly check this condition in the scripted Porcelain.

Introduce the "is-active" subcommand to "submodule--helper", so
that the exact definition of what submodule is of interest can
be centrally defined (and changed in later steps).  In a few
patches that follow, this helper is used to replace the explicit
checks of the configuration variable in scripts.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c    | 11 +++++++++++
 t/t7413-submodule-is-active.sh | 31 +++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+)
 create mode 100755 t/t7413-submodule-is-active.sh

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 15a5430c0..5fe7e23b1 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -1127,6 +1127,16 @@ static int absorb_git_dirs(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
+static int is_active(int argc, const char **argv, const char *prefix)
+{
+	if (argc != 2)
+		die("submodule--helper is-active takes exactly 1 arguments");
+
+	gitmodules_config();
+
+	return !is_submodule_initialized(argv[1]);
+}
+
 #define SUPPORT_SUPER_PREFIX (1<<0)
 
 struct cmd_struct {
@@ -1146,6 +1156,7 @@ static struct cmd_struct commands[] = {
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"remote-branch", resolve_remote_submodule_branch, 0},
 	{"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
+	{"is-active", is_active, 0},
 };
 
 int cmd_submodule__helper(int argc, const char **argv, const char *prefix)
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
new file mode 100755
index 000000000..f18e0c925
--- /dev/null
+++ b/t/t7413-submodule-is-active.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+
+test_description='Test submodule--helper is-active
+
+This test verifies that `git submodue--helper is-active` correclty identifies
+submodules which are "active" and interesting to the user.
+'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	git init sub &&
+	test_commit -C sub initial &&
+	git init super &&
+	test_commit -C super initial &&
+	git -C super submodule add ../sub sub1 &&
+	git -C super submodule add ../sub sub2 &&
+	git -C super commit -a -m "add 2 submodules at sub{1,2}"
+'
+
+test_expect_success 'is-active works with urls' '
+	git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2 &&
+
+	git -C super config --unset submodule.sub1.URL &&
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super config submodule.sub1.URL ../sub &&
+	git -C super submodule--helper is-active sub1
+'
+
+test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v5 02/10] submodule status: use submodule--helper is-active

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 136e26a2c..ab233712d 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1010,14 +1010,13 @@ cmd_status()
 	do
 		die_if_unmatched "$mode" "$sha1"
 		name=$(git submodule--helper name "$sm_path") || exit
-		url=$(git config submodule."$name".url)
 		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
 		if test "$stage" = U
 		then
 			say "U$sha1 $displaypath"
 			continue
 		fi
-		if test -z "$url" ||
+		if ! git submodule--helper is-active "$sm_path" ||
 		{
 			! test -d "$sm_path"/.git &&
 			! test -f "$sm_path"/.git
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v5 03/10] submodule sync: skip work for inactive submodules

[Brandon Williams]

Sync does some work determining what URLs should be used for a submodule
but then throws this work away if the submodule isn't active.  Instead
perform the activity check earlier and skip inactive submodule in order
to avoid doing unnecessary work.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 38 +++++++++++++++++++++-----------------
 1 file changed, 21 insertions(+), 17 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index ab233712d..577136148 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1090,6 +1090,13 @@ cmd_sync()
 	do
 		die_if_unmatched "$mode" "$sha1"
 		name=$(git submodule--helper name "$sm_path")
+
+		# skip inactive submodules
+		if ! git config "submodule.$name.url" >/dev/null 2>/dev/null
+		then
+			continue
+		fi
+
 		url=$(git config -f .gitmodules --get submodule."$name".url)
 
 		# Possibly a url relative to parent
@@ -1111,27 +1118,24 @@ cmd_sync()
 			;;
 		esac
 
-		if git config "submodule.$name.url" >/dev/null 2>/dev/null
+		displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
+		say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
+		git config submodule."$name".url "$super_config_url"
+
+		if test -e "$sm_path"/.git
 		then
-			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-			say "$(eval_gettext "Synchronizing submodule url for '\$displaypath'")"
-			git config submodule."$name".url "$super_config_url"
+		(
+			sanitize_submodule_env
+			cd "$sm_path"
+			remote=$(get_default_remote)
+			git config remote."$remote".url "$sub_origin_url"
 
-			if test -e "$sm_path"/.git
+			if test -n "$recursive"
 			then
-			(
-				sanitize_submodule_env
-				cd "$sm_path"
-				remote=$(get_default_remote)
-				git config remote."$remote".url "$sub_origin_url"
-
-				if test -n "$recursive"
-				then
-					prefix="$prefix$sm_path/"
-					eval cmd_sync
-				fi
-			)
+				prefix="$prefix$sm_path/"
+				eval cmd_sync
 			fi
+		)
 		fi
 	done
 }
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v5 04/10] submodule sync: use submodule--helper is-active

[Brandon Williams]

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 577136148..db94dea3b 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -1089,14 +1089,14 @@ cmd_sync()
 	while read mode sha1 stage sm_path
 	do
 		die_if_unmatched "$mode" "$sha1"
-		name=$(git submodule--helper name "$sm_path")
 
 		# skip inactive submodules
-		if ! git config "submodule.$name.url" >/dev/null 2>/dev/null
+		if ! git submodule--helper is-active "$sm_path"
 		then
 			continue
 		fi
 
+		name=$(git submodule--helper name "$sm_path")
 		url=$(git config -f .gitmodules --get submodule."$name".url)
 
 		# Possibly a url relative to parent
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v5 05/10] submodule--helper clone: check for configured submodules using helper

[Brandon Williams]

Use the 'is_submodule_initialized()' helper to check for configured
submodules instead of manually checking for the submodule's URL in the
config.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 5fe7e23b1..f38e332c5 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -759,7 +759,6 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 	struct strbuf displaypath_sb = STRBUF_INIT;
 	struct strbuf sb = STRBUF_INIT;
 	const char *displaypath = NULL;
-	char *url = NULL;
 	int needs_cloning = 0;
 
 	if (ce_stage(ce)) {
@@ -793,15 +792,8 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		goto cleanup;
 	}
 
-	/*
-	 * Looking up the url in .git/config.
-	 * We must not fall back to .gitmodules as we only want
-	 * to process configured submodules.
-	 */
-	strbuf_reset(&sb);
-	strbuf_addf(&sb, "submodule.%s.url", sub->name);
-	git_config_get_string(sb.buf, &url);
-	if (!url) {
+	/* Check if the submodule has been initialized. */
+	if (!is_submodule_initialized(ce->name)) {
 		next_submodule_warn_missing(suc, out, displaypath);
 		goto cleanup;
 	}
@@ -835,7 +827,7 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		argv_array_push(&child->args, "--depth=1");
 	argv_array_pushl(&child->args, "--path", sub->path, NULL);
 	argv_array_pushl(&child->args, "--name", sub->name, NULL);
-	argv_array_pushl(&child->args, "--url", url, NULL);
+	argv_array_pushl(&child->args, "--url", sub->url, NULL);
 	if (suc->references.nr) {
 		struct string_list_item *item;
 		for_each_string_list_item(item, &suc->references)
@@ -845,7 +837,6 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		argv_array_push(&child->args, suc->depth);
 
 cleanup:
-	free(url);
 	strbuf_reset(&displaypath_sb);
 	strbuf_reset(&sb);
 
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v5 06/10] submodule: decouple url and submodule interest

[Brandon Williams]

Currently the submodule.<name>.url config option is used to determine if
a given submodule is of interest to the user.  This ends up being
cumbersome in a world where we want to have different submodules checked
out in different worktrees or a more generalized mechanism to select
which submodules are of interest.

In a future with worktree support for submodules, there will be multiple
working trees, each of which may only need a subset of the submodules
checked out.  The URL (which is where the submodule repository can be
obtained) should not differ between different working trees.

It may also be convenient for users to more easily specify groups of
submodules they are interested in as opposed to running "git submodule
init <path>" on each submodule they want checked out in their working
tree.

To this end two config options are introduced, submodule.active and
submodule.<name>.active.  The submodule.active config holds a pathspec
that specifies which submodules should exist in the working tree.  The
submodule.<name>.active config is a boolean flag used to indicate if
that particular submodule should exist in the working tree.

Its important to note that submodule.active functions differently than
the other configuration options since it takes a pathspec.  This allows
users to adopt at least two new workflows:

  1. Submodules can be grouped with a leading directory, such that a
     pathspec e.g. 'lib/' would cover all library-ish modules to allow
     those who are interested in library-ish modules to set
     "submodule.active = lib/" just once to say any and all modules in
     'lib/' are interesting.

  2. Once the pathspec-attribute feature is invented, users can label
     submodules with attributes to group them, so that a broad pathspec
     with attribute requirements, e.g. ':(attr:lib)', can be used to say
     any and all modules with the 'lib' attribute are interesting.
     Since the .gitattributes file, just like the .gitmodules file, is
     tracked by the superproject, when a submodule moves in the
     superproject tree, the project can adjust which path gets the
     attribute in .gitattributes, just like it can adjust which path has
     the submodule in .gitmodules.

Neither of these two additional configuration options solve the problem
of wanting different submodules checked out in different worktrees
because multiple worktrees share .git/config.  Only once per-worktree
configurations become a reality can this be solved, but this is a
necessary preparatory step for that future.

Given these multiple ways to check if a submodule is of interest, the
more fine-grained submodule.<name>.active option has the highest order
of precedence followed by the pathspec check against submodule.active.
To ensure backwards compatibility, if neither of these options are set,
git falls back to checking the submodule.<name>.url option to determine
if a submodule is interesting.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/config.txt       | 15 ++++++++++--
 submodule.c                    | 50 ++++++++++++++++++++++++++++++++------
 t/t7413-submodule-is-active.sh | 55 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 110 insertions(+), 10 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 5e5c2ae5f..d2d79b9d4 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -2920,8 +2920,9 @@ submodule.<name>.url::
 	The URL for a submodule. This variable is copied from the .gitmodules
 	file to the git config via 'git submodule init'. The user can change
 	the configured URL before obtaining the submodule via 'git submodule
-	update'. After obtaining the submodule, the presence of this variable
-	is used as a sign whether the submodule is of interest to git commands.
+	update'. If neither submodule.<name>.active or submodule.active are
+	set, the presence of this variable is used as a fallback to indicate
+	whether the submodule is of interest to git commands.
 	See linkgit:git-submodule[1] and linkgit:gitmodules[5] for details.
 
 submodule.<name>.update::
@@ -2959,6 +2960,16 @@ submodule.<name>.ignore::
 	"--ignore-submodules" option. The 'git submodule' commands are not
 	affected by this setting.
 
+submodule.<name>.active::
+	Boolean value indicating if the submodule is of interest to git
+	commands.  This config option takes precedence over the
+	submodule.active config option.
+
+submodule.active::
+	A repeated field which contains a pathspec used to match against a
+	submodule's path to determine if the submodule is of interest to git
+	commands.
+
 submodule.fetchJobs::
 	Specifies how many submodules are fetched/cloned at the same time.
 	A positive integer allows up to that number of submodules fetched
diff --git a/submodule.c b/submodule.c
index 0a2831d84..ad2779ee7 100644
--- a/submodule.c
+++ b/submodule.c
@@ -212,25 +212,59 @@ void gitmodules_config_sha1(const unsigned char *commit_sha1)
 }
 
 /*
+ * NEEDSWORK: With the addition of different configuration options to determine
+ * if a submodule is of interests, the validity of this function's name comes
+ * into question.  Once the dust has settled and more concrete terminology is
+ * decided upon, come up with a more proper name for this function.  One
+ * potential candidate could be 'is_submodule_active()'.
+ *
  * Determine if a submodule has been initialized at a given 'path'
  */
 int is_submodule_initialized(const char *path)
 {
 	int ret = 0;
-	const struct submodule *module = NULL;
+	char *key = NULL;
+	char *value = NULL;
+	const struct string_list *sl;
+	const struct submodule *module = submodule_from_path(null_sha1, path);
 
-	module = submodule_from_path(null_sha1, path);
+	/* early return if there isn't a path->module mapping */
+	if (!module)
+		return 0;
 
-	if (module) {
-		char *key = xstrfmt("submodule.%s.url", module->name);
-		char *value = NULL;
+	/* submodule.<name>.active is set */
+	key = xstrfmt("submodule.%s.active", module->name);
+	if (!git_config_get_bool(key, &ret)) {
+		free(key);
+		return ret;
+	}
+	free(key);
 
-		ret = !git_config_get_string(key, &value);
+	/* submodule.active is set */
+	sl = git_config_get_value_multi("submodule.active");
+	if (sl) {
+		struct pathspec ps;
+		struct argv_array args = ARGV_ARRAY_INIT;
+		const struct string_list_item *item;
 
-		free(value);
-		free(key);
+		for_each_string_list_item(item, sl) {
+			argv_array_push(&args, item->string);
+		}
+
+		parse_pathspec(&ps, 0, 0, NULL, args.argv);
+		ret = match_pathspec(&ps, path, strlen(path), 0, NULL, 1);
+
+		argv_array_clear(&args);
+		clear_pathspec(&ps);
+		return ret;
 	}
 
+	/* fallback to checking if the URL is set */
+	key = xstrfmt("submodule.%s.url", module->name);
+	ret = !git_config_get_string(key, &value);
+
+	free(value);
+	free(key);
 	return ret;
 }
 
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
index f18e0c925..ea1644b58 100755
--- a/t/t7413-submodule-is-active.sh
+++ b/t/t7413-submodule-is-active.sh
@@ -28,4 +28,59 @@ test_expect_success 'is-active works with urls' '
 	git -C super submodule--helper is-active sub1
 '
 
+test_expect_success 'is-active works with submodule.<name>.active config' '
+	test_when_finished "git -C super config --unset submodule.sub1.active" &&
+	test_when_finished "git -C super config submodule.sub1.URL ../sub" &&
+
+	git -C super config --bool submodule.sub1.active "false" &&
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+
+	git -C super config --bool submodule.sub1.active "true" &&
+	git -C super config --unset submodule.sub1.URL &&
+	git -C super submodule--helper is-active sub1
+'
+
+test_expect_success 'is-active works with basic submodule.active config' '
+	test_when_finished "git -C super config submodule.sub1.URL ../sub" &&
+	test_when_finished "git -C super config --unset-all submodule.active" &&
+
+	git -C super config --add submodule.active "." &&
+	git -C super config --unset submodule.sub1.URL &&
+
+	git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2
+'
+
+test_expect_success 'is-active correctly works with paths that are not submodules' '
+	test_when_finished "git -C super config --unset-all submodule.active" &&
+
+	test_must_fail git -C super submodule--helper is-active not-a-submodule &&
+
+	git -C super config --add submodule.active "." &&
+	test_must_fail git -C super submodule--helper is-active not-a-submodule
+'
+
+test_expect_success 'is-active works with exclusions in submodule.active config' '
+	test_when_finished "git -C super config --unset-all submodule.active" &&
+
+	git -C super config --add submodule.active "." &&
+	git -C super config --add submodule.active ":(exclude)sub1" &&
+
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2
+'
+
+test_expect_success 'is-active with submodule.active and submodule.<name>.active' '
+	test_when_finished "git -C super config --unset-all submodule.active" &&
+	test_when_finished "git -C super config --unset submodule.sub1.active" &&
+	test_when_finished "git -C super config --unset submodule.sub2.active" &&
+
+	git -C super config --add submodule.active "sub1" &&
+	git -C super config --bool submodule.sub1.active "false" &&
+	git -C super config --bool submodule.sub2.active "true" &&
+
+	test_must_fail git -C super submodule--helper is-active sub1 &&
+	git -C super submodule--helper is-active sub2
+'
+
 test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v5 07/10] submodule init: initialize active submodules

[Brandon Williams]

Teach `submodule init` to initialize submodules which have been
configured to be active by setting 'submodule.active' with a pathspec.

Now if no path arguments are given and 'submodule.active' is configured,
`init` will initialize all submodules which have been configured to be
active.  If no path arguments are given and 'submodule.active' is not
configured, then `init` will retain the old behavior of initializing all
submodules.

This allows users to record more complex patterns as it saves retyping
them whenever you invoke update.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/git-submodule.txt |  4 ++-
 builtin/submodule--helper.c     | 30 ++++++++++++++++++++++
 t/t7400-submodule-basic.sh      | 57 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 90 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index e05d0cdde..74bc6200d 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -129,7 +129,9 @@ init [--] [<path>...]::
 	repository will be assumed to be upstream.
 +
 Optional <path> arguments limit which submodules will be initialized.
-If no path is specified, all submodules are initialized.
+If no path is specified and submodule.active has been configured, submodules
+configured to be active will be initialized, otherwise all submodules are
+initialized.
 +
 When present, it will also copy the value of `submodule.$name.update`.
 This command does not alter existing information in .git/config.
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index f38e332c5..65208faa7 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -270,6 +270,29 @@ static int module_list_compute(int argc, const char **argv,
 	return result;
 }
 
+static void module_list_active(struct module_list *list)
+{
+	int i;
+	struct module_list active_modules = MODULE_LIST_INIT;
+
+	gitmodules_config();
+
+	for (i = 0; i < list->nr; i++) {
+		const struct cache_entry *ce = list->entries[i];
+
+		if (!is_submodule_initialized(ce->name))
+			continue;
+
+		ALLOC_GROW(active_modules.entries,
+			   active_modules.nr + 1,
+			   active_modules.alloc);
+		active_modules.entries[active_modules.nr++] = ce;
+	}
+
+	free(list->entries);
+	*list = active_modules;
+}
+
 static int module_list(int argc, const char **argv, const char *prefix)
 {
 	int i;
@@ -420,6 +443,13 @@ static int module_init(int argc, const char **argv, const char *prefix)
 	if (module_list_compute(argc, argv, prefix, &pathspec, &list) < 0)
 		return 1;
 
+	/*
+	 * If there are no path args and submodule.active is set then,
+	 * by default, only initialize 'active' modules.
+	 */
+	if (!argc && git_config_get_value_multi("submodule.active"))
+		module_list_active(&list);
+
 	for (i = 0; i < list.nr; i++)
 		init_submodule(list.entries[i]->name, prefix, quiet);
 
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index c09ce0d4c..fbbe932d1 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1130,5 +1130,62 @@ test_expect_success 'submodule helper list is not confused by common prefixes' '
 	test_cmp expect actual
 '
 
+test_expect_success 'setup superproject with submodules' '
+	git init sub1 &&
+	test_commit -C sub1 test &&
+	test_commit -C sub1 test2 &&
+	git init multisuper &&
+	git -C multisuper submodule add ../sub1 sub0 &&
+	git -C multisuper submodule add ../sub1 sub1 &&
+	git -C multisuper submodule add ../sub1 sub2 &&
+	git -C multisuper submodule add ../sub1 sub3 &&
+	git -C multisuper commit -m "add some submodules"
+'
+
+cat >expect <<-EOF
+-sub0
+ sub1 (test2)
+ sub2 (test2)
+ sub3 (test2)
+EOF
+
+test_expect_success 'submodule update --init with a specification' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone submodule update --init . ":(exclude)sub0" &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'submodule update --init with submodule.active set' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone config submodule.active "." &&
+	git -C multisuper_clone config --add submodule.active ":(exclude)sub0" &&
+	git -C multisuper_clone submodule update --init &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'submodule update and setting submodule.<name>.active' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	pwd=$(pwd) &&
+	git clone file://"$pwd"/multisuper multisuper_clone &&
+	git -C multisuper_clone config --bool submodule.sub0.active "true" &&
+	git -C multisuper_clone config --bool submodule.sub1.active "false" &&
+	git -C multisuper_clone config --bool submodule.sub2.active "true" &&
+
+	cat >expect <<-\EOF &&
+	 sub0 (test2)
+	-sub1
+	 sub2 (test2)
+	-sub3
+	EOF
+	git -C multisuper_clone submodule update &&
+	git -C multisuper_clone submodule status |cut -c 1,43- >actual &&
+	test_cmp expect actual
+'
 
 test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v5 08/10] clone: teach --recurse-submodules to optionally take a pathspec

[Brandon Williams]

Teach clone --recurse-submodules to optionally take a pathspec argument
which describes which submodules should be recursively initialized and
cloned.  If no pathspec is provided, --recurse-submodules will
recursively initialize and clone all submodules by using a default
pathspec of ".".  In order to construct more complex pathspecs,
--recurse-submodules can be given multiple times.

This also configures the 'submodule.active' configuration option to be
the given pathspec, such that any future invocation of `git submodule
update` will keep up with the pathspec.

Additionally the switch '--recurse' is removed from the Documentation as
well as marked hidden in the options array, to streamline the options
for submodules.  A simple '--recurse' doesn't convey what is being
recursed, e.g. it could mean directories or trees (c.f. ls-tree) In a
lot of other commands we already have '--recurse-submodules' to mean
recursing into submodules, so advertise this spelling here as the
genuine option.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 Documentation/git-clone.txt | 14 ++++++----
 builtin/clone.c             | 50 ++++++++++++++++++++++++++++-----
 t/t7400-submodule-basic.sh  | 68 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 120 insertions(+), 12 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 35cc34b2f..30052cce4 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -14,7 +14,7 @@ SYNOPSIS
 	  [-o <name>] [-b <name>] [-u <upload-pack>] [--reference <repository>]
 	  [--dissociate] [--separate-git-dir <git dir>]
 	  [--depth <depth>] [--[no-]single-branch]
-	  [--recursive | --recurse-submodules] [--[no-]shallow-submodules]
+	  [--recurse-submodules] [--[no-]shallow-submodules]
 	  [--jobs <n>] [--] <repository> [<directory>]
 
 DESCRIPTION
@@ -215,10 +215,14 @@ objects from the source repository into a pack in the cloned repository.
 	branch when `--single-branch` clone was made, no remote-tracking
 	branch is created.
 
---recursive::
---recurse-submodules::
-	After the clone is created, initialize all submodules within,
-	using their default settings. This is equivalent to running
+--recurse-submodules[=<pathspec]::
+	After the clone is created, initialize and clone submodules
+	within based on the provided pathspec.  If no pathspec is
+	provided, all submodules are initialized and cloned.
+	Submodules are initialized and cloned using their default
+	settings.  The resulting clone has `submodule.active` set to
+	the provided pathspec, or "." (meaning all submodules) if no
+	pathspec is provided.  This is equivalent to running
 	`git submodule update --init --recursive` immediately after
 	the clone is finished. This option is ignored if the cloned
 	repository does not have a worktree/checkout (i.e. if any of
diff --git a/builtin/clone.c b/builtin/clone.c
index 3f63edbbf..a7be61d6b 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -39,7 +39,7 @@ static const char * const builtin_clone_usage[] = {
 };
 
 static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
-static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
+static int option_local = -1, option_no_hardlinks, option_shared;
 static int option_shallow_submodules;
 static int deepen;
 static char *option_template, *option_depth, *option_since;
@@ -56,6 +56,21 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;
 static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;
 static int option_dissociate;
 static int max_jobs = -1;
+static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;
+
+static int recurse_submodules_cb(const struct option *opt,
+				 const char *arg, int unset)
+{
+	if (unset)
+		string_list_clear((struct string_list *)opt->value, 0);
+	else if (arg)
+		string_list_append((struct string_list *)opt->value, arg);
+	else
+		string_list_append((struct string_list *)opt->value,
+				   (const char *)opt->defval);
+
+	return 0;
+}
 
 static struct option builtin_clone_options[] = {
 	OPT__VERBOSITY(&option_verbosity),
@@ -74,10 +89,13 @@ static struct option builtin_clone_options[] = {
 		    N_("don't use local hardlinks, always copy")),
 	OPT_BOOL('s', "shared", &option_shared,
 		    N_("setup as shared repository")),
-	OPT_BOOL(0, "recursive", &option_recursive,
-		    N_("initialize submodules in the clone")),
-	OPT_BOOL(0, "recurse-submodules", &option_recursive,
-		    N_("initialize submodules in the clone")),
+	{ OPTION_CALLBACK, 0, "recursive", &option_recurse_submodules,
+	  N_("pathspec"), N_("initialize submodules in the clone"),
+	  PARSE_OPT_OPTARG | PARSE_OPT_HIDDEN, recurse_submodules_cb,
+	  (intptr_t)"." },
+	{ OPTION_CALLBACK, 0, "recurse-submodules", &option_recurse_submodules,
+	  N_("pathspec"), N_("initialize submodules in the clone"),
+	  PARSE_OPT_OPTARG, recurse_submodules_cb, (intptr_t)"." },
 	OPT_INTEGER('j', "jobs", &max_jobs,
 		    N_("number of submodules cloned in parallel")),
 	OPT_STRING(0, "template", &option_template, N_("template-directory"),
@@ -733,7 +751,7 @@ static int checkout(int submodule_progress)
 	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
 			   sha1_to_hex(sha1), "1", NULL);
 
-	if (!err && option_recursive) {
+	if (!err && (option_recurse_submodules.nr > 0)) {
 		struct argv_array args = ARGV_ARRAY_INIT;
 		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
 
@@ -957,7 +975,25 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 			fprintf(stderr, _("Cloning into '%s'...\n"), dir);
 	}
 
-	if (option_recursive) {
+	if (option_recurse_submodules.nr > 0) {
+		struct string_list_item *item;
+		struct strbuf sb = STRBUF_INIT;
+
+		/* remove duplicates */
+		string_list_sort(&option_recurse_submodules);
+		string_list_remove_duplicates(&option_recurse_submodules, 0);
+
+		/*
+		 * NEEDSWORK: In a multi-working-tree world, this needs to be
+		 * set in the per-worktree config.
+		 */
+		for_each_string_list_item(item, &option_recurse_submodules) {
+			strbuf_addf(&sb, "submodule.active=%s",
+				    item->string);
+			string_list_append(&option_config,
+					   strbuf_detach(&sb, NULL));
+		}
+
 		if (option_required_reference.nr &&
 		    option_optional_reference.nr)
 			die(_("clone --recursive is not compatible with "
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index fbbe932d1..3af1c00ff 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1188,4 +1188,72 @@ test_expect_success 'submodule update and setting submodule.<name>.active' '
 	test_cmp expect actual
 '
 
+test_expect_success 'clone --recurse-submodules with a pathspec works' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat >expected <<-\EOF &&
+	 sub0 (test2)
+	-sub1
+	-sub2
+	-sub3
+	EOF
+
+	git clone --recurse-submodules="sub0" multisuper multisuper_clone &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp actual expected
+'
+
+test_expect_success 'clone with multiple --recurse-submodules options' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat >expect <<-\EOF &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	EOF
+
+	git clone --recurse-submodules="." \
+		  --recurse-submodules=":(exclude)sub0" \
+		  --recurse-submodules=":(exclude)sub2" \
+		  multisuper multisuper_clone &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'clone and subsequent updates correctly auto-initialize submodules' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	cat <<-\EOF >expect &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	EOF
+
+	cat <<-\EOF >expect2 &&
+	-sub0
+	 sub1 (test2)
+	-sub2
+	 sub3 (test2)
+	-sub4
+	 sub5 (test2)
+	EOF
+
+	git clone --recurse-submodules="." \
+		  --recurse-submodules=":(exclude)sub0" \
+		  --recurse-submodules=":(exclude)sub2" \
+		  --recurse-submodules=":(exclude)sub4" \
+		  multisuper multisuper_clone &&
+
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect actual &&
+
+	git -C multisuper submodule add ../sub1 sub4 &&
+	git -C multisuper submodule add ../sub1 sub5 &&
+	git -C multisuper commit -m "add more submodules" &&
+	# obtain the new superproject
+	git -C multisuper_clone pull &&
+	git -C multisuper_clone submodule update --init &&
+	git -C multisuper_clone submodule status |cut -c1,43- >actual &&
+	test_cmp expect2 actual
+'
+
 test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v5 09/10] submodule--helper init: set submodule.<name>.active

[Brandon Williams]

When initializing a submodule set the submodule.<name>.active config to
true if the module hasn't already been configured to be active by some
other means (e.g. a pathspec set in submodule.active).

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 builtin/submodule--helper.c | 12 ++++++++++++
 t/t7400-submodule-basic.sh  | 11 +++++++++++
 2 files changed, 23 insertions(+)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 65208faa7..7700d8948 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -357,6 +357,18 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 			displaypath);
 
 	/*
+	 * NEEDSWORK: In a multi-working-tree world, this needs to be
+	 * set in the per-worktree config.
+	 *
+	 * Set active flag for the submodule being initialized
+	 */
+	if (!is_submodule_initialized(path)) {
+		strbuf_reset(&sb);
+		strbuf_addf(&sb, "submodule.%s.active", sub->name);
+		git_config_set_gently(sb.buf, "true");
+	}
+
+	/*
 	 * Copy url setting when it is not set yet.
 	 * To look up the url in .git/config, we must not fall back to
 	 * .gitmodules, so look it up directly.
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 3af1c00ff..cf77a3a35 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1256,4 +1256,15 @@ test_expect_success 'clone and subsequent updates correctly auto-initialize subm
 	test_cmp expect2 actual
 '
 
+test_expect_success 'init properly sets the config' '
+	test_when_finished "rm -rf multisuper_clone" &&
+	git clone --recurse-submodules="." \
+		  --recurse-submodules=":(exclude)sub0" \
+		  multisuper multisuper_clone &&
+
+	git -C multisuper_clone submodule init -- sub0 sub1 &&
+	git -C multisuper_clone config --get submodule.sub0.active &&
+	test_must_fail git -C multisuper_clone config --get submodule.sub1.active
+'
+
 test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

[PATCH v5 10/10] submodule add: respect submodule.active and submodule.<name>.active

[Brandon Williams]

In addition to adding submodule.<name>.url to the config, set
submodule.<name>.active to true unless submodule.active is configured
and the submodule's path matches the configured pathspec.

Signed-off-by: Brandon Williams <bmwill@google.com>
---
 git-submodule.sh               | 14 ++++++++++++++
 t/t7413-submodule-is-active.sh | 21 +++++++++++++++++++++
 2 files changed, 35 insertions(+)

diff --git a/git-submodule.sh b/git-submodule.sh
index db94dea3b..6ec35e5fc 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -278,6 +278,20 @@ or you are unsure what this means choose another name with the '--name' option."
 	fi &&
 	git add --force .gitmodules ||
 	die "$(eval_gettext "Failed to register submodule '\$sm_path'")"
+
+	# NEEDSWORK: In a multi-working-tree world, this needs to be
+	# set in the per-worktree config.
+	if git config --get submodule.active >/dev/null
+	then
+		# If the submodule being adding isn't already covered by the
+		# current configured pathspec, set the submodule's active flag
+		if ! git submodule--helper is-active "$sm_path"
+		then
+			git config submodule."$sm_name".active "true"
+		fi
+	else
+		git config submodule."$sm_name".active "true"
+	fi
 }
 
 #
diff --git a/t/t7413-submodule-is-active.sh b/t/t7413-submodule-is-active.sh
index ea1644b58..9c785b07e 100755
--- a/t/t7413-submodule-is-active.sh
+++ b/t/t7413-submodule-is-active.sh
@@ -15,6 +15,12 @@ test_expect_success 'setup' '
 	test_commit -C super initial &&
 	git -C super submodule add ../sub sub1 &&
 	git -C super submodule add ../sub sub2 &&
+
+	# Remove submodule.<name>.active entries in order to test in an
+	# environment where only URLs are present in the conifg
+	git -C super config --unset submodule.sub1.active &&
+	git -C super config --unset submodule.sub2.active &&
+
 	git -C super commit -a -m "add 2 submodules at sub{1,2}"
 '
 
@@ -83,4 +89,19 @@ test_expect_success 'is-active with submodule.active and submodule.<name>.active
 	git -C super submodule--helper is-active sub2
 '
 
+test_expect_success 'is-active, submodule.active and submodule add' '
+	test_when_finished "rm -rf super2" &&
+	git init super2 &&
+	test_commit -C super2 initial &&
+	git -C super2 config --add submodule.active "sub*" &&
+
+	# submodule add should only add submodule.<name>.active
+	# to the config if not matched by the pathspec
+	git -C super2 submodule add ../sub sub1 &&
+	test_must_fail git -C super2 config --get submodule.sub1.active &&
+
+	git -C super2 submodule add ../sub mod &&
+	git -C super2 config --get submodule.mod.active
+'
+
 test_done
-- 
2.12.0.367.g23dc2f6d3c-goog

Re: [PATCH v5 00/10] decoupling url and submodule interest

[Stefan Beller]

On Fri, Mar 17, 2017 at 3:37 PM, Brandon Williams <bmwill@google.com> wrote:
> Changes in v5:
>   * Add "NEEDSWORK" comments to indicate where attention is needed once
>     per-worktree config is a reality
>   * --no-recurse now works by clearing the string list of paths.
>   * module_list_active() now does post-processing instead of duplicating code.

Skimming these patches, they all look good to me.

Thanks,
Stefan

Re: [PATCH v5 00/10] decoupling url and submodule interest

[Junio C Hamano]

Brandon Williams <bmwill@google.com> writes:

> Changes in v5:
>   * Add "NEEDSWORK" comments to indicate where attention is needed once
>     per-worktree config is a reality
>   * --no-recurse now works by clearing the string list of paths.
>   * module_list_active() now does post-processing instead of duplicating code.
>
> Brandon Williams (10):
>   submodule--helper: add is-active subcommand
>   submodule status: use submodule--helper is-active
>   submodule sync: skip work for inactive submodules
>   submodule sync: use submodule--helper is-active
>   submodule--helper clone: check for configured submodules using helper
>   submodule: decouple url and submodule interest
>   submodule init: initialize active submodules
>   clone: teach --recurse-submodules to optionally take a pathspec
>   submodule--helper init: set submodule.<name>.active
>   submodule add: respect submodule.active and submodule.<name>.active

Replaced what has been queued with this and read it over
again. Looked sensible.

Thanks.