[RFC/PATCH 0/5] [WAS: Submodule Groups] Labels and submodule.autoInitialize

[RFC/PATCH 0/5] [WAS: Submodule Groups] Labels and submodule.autoInitialize

[Stefan Beller]

This series introduces labels which you can attach to submodules like so:

    $ cat .gitmodules
    [submodule "gcc"]
        path = gcc
        url = git://...
        label = default
        label = devel
    [submodule "linux"]
        path = linux
        url = git://...
        label = default

    $ git submodule add --name emacs --label "editor" --label default git://...
    
    # If upstream has submodules properly labeled, you can make use of them:
    
    $ git config --add submodule.autoInitialize "*default"
    $ git config --add submodule.autoInitialize ":name"
    $ git config --add submodule.autoInitialize "./by/path"
    
    # The prefix * denotes a label as found in .gitmodules
    # : goes before names 
    # path are prefixed ./ currently
    # both path and names need work
    
    # no --init necessary, partially initializes submodules (only those which
    # were specified by label, name or path)
    $ git submodule update
    
    # time passes, upstream may have added new submodules and we get them without
    # extra commands!
    $ git submodule update
    
    # The above configuration can be given to git clone directly via:
    $ git clone --init-submodule=*labelA ...
    
Why?
====
If you have lots of submodules, you probably don't need all of them at once,
but you have functional units. Some submodules are absolutely required,
some are optional and only for very specific purposes.

Changes to the previous version with groups:
====
* it's called labels now (it's easier to imagine to attach more than one
  label to a submodule than having it in more groups)
* In .git/config we have another name, too ("submodule.autoInitialize")
* Support for more than just groups, but also names and paths are supported
  (in a very rudimentary way though)
  
This series applies on top of sb/submodule-init, or can be found at
https://github.com/stefanbeller/git/tree/submodule-groups-v3
  
  Thanks,
  Stefan

Stefan Beller (5):
  submodule init: Write submodule registration to stderr
  git submodule: Teach add to label submodules
  submodule-config: keep labels around
  submodule update: respect submodule.autoInitialize
  builtin/clone: Configure submodule.autoInitialize via --init-submodule

 Documentation/git-clone.txt     |   6 ++
 Documentation/git-submodule.txt |   5 +-
 builtin/clone.c                 |  40 +++++++++-
 builtin/submodule--helper.c     |  58 +++++++++++++-
 git-submodule.sh                |  14 +++-
 submodule-config.c              |  15 ++++
 submodule-config.h              |   2 +
 t/t7400-submodule-basic.sh      | 165 ++++++++++++++++++++++++++++++++++++++++
 8 files changed, 298 insertions(+), 7 deletions(-)

-- 
2.7.0.rc0.42.g77a36b9.dirty

[PATCH 1/5] submodule init: Write submodule registration to stderr

[Stefan Beller]

The registration of the submodule will be reported to stderr, as that is
consistent with the rest of progress reporting within Git.

This helps us in a later patch when we want to reuse the
init_submodule function in update_clone whose stdout will be piped
to shell which reads parameters off stdout in a very specific way.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/submodule--helper.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index c9b0c05..05c18a3 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -267,7 +267,7 @@ static void init_submodule(const char *path, const char *prefix, int quiet)
 			die(_("Failed to register url for submodule path '%s'"),
 			    displaypath);
 		if (!quiet)
-			printf(_("Submodule '%s' (%s) registered for path '%s'\n"),
+			fprintf(stderr, _("Submodule '%s' (%s) registered for path '%s'\n"),
 				sub->name, url, displaypath);
 		free(url);
 	}
-- 
2.7.0.rc0.42.g77a36b9.dirty

[PATCH 2/5] git submodule: Teach add to label submodules

[Stefan Beller]

When adding new submodules, you can specify the
label(s) the submodule belongs to by giving one or more
--label arguments. This will record each label in the
.gitmodules file as a value of the key
"submodule.$NAME.label".

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/git-submodule.txt |  5 ++++-
 git-submodule.sh                | 14 +++++++++++++-
 t/t7400-submodule-basic.sh      | 32 ++++++++++++++++++++++++++++++++
 3 files changed, 49 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 13adebf..c0744eb 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -9,7 +9,7 @@ git-submodule - Initialize, update or inspect submodules
 SYNOPSIS
 --------
 [verse]
-'git submodule' [--quiet] add [-b <branch>] [-f|--force] [--name <name>]
+'git submodule' [--quiet] add [-b <branch>] [-f|--force] [-l|--label <label>]
 	      [--reference <repository>] [--depth <depth>] [--] <repository> [<path>]
 'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]
 'git submodule' [--quiet] init [--] [<path>...]
@@ -101,6 +101,9 @@ is the superproject and submodule repositories will be kept
 together in the same relative location, and only the
 superproject's URL needs to be provided: git-submodule will correctly
 locate the submodule using the relative URL in .gitmodules.
++
+If at least one label argument was given, all labels are recorded in the
+.gitmodules file in the label fields.
 
 status::
 	Show the status of the submodules. This will print the SHA-1 of the
diff --git a/git-submodule.sh b/git-submodule.sh
index 6fce0dc..65b740c 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -5,7 +5,7 @@
 # Copyright (c) 2007 Lars Hjemli
 
 dashless=$(basename "$0" | sed -e 's/-/ /')
-USAGE="[--quiet] add [-b <branch>] [-f|--force] [--name <name>] [--reference <repository>] [--] <repository> [<path>]
+USAGE="[--quiet] add [-b <branch>] [-f|--force] [--name <name>] [--reference <repository>] [-l|--label <label>][--] <repository> [<path>]
    or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]
    or: $dashless [--quiet] init [--] [<path>...]
    or: $dashless [--quiet] deinit [-f|--force] [--] <path>...
@@ -130,6 +130,7 @@ cmd_add()
 {
 	# parse $args after "submodule ... add".
 	reference_path=
+	labels=
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -165,6 +166,13 @@ cmd_add()
 		--depth=*)
 			depth=$1
 			;;
+		-l|--label)
+			labels="${labels} $2"
+			shift
+			;;
+		--label=*)
+			labels="${labels} ${1#--label=}"
+			;;
 		--)
 			shift
 			break
@@ -292,6 +300,10 @@ Use -f if you really want to add it." >&2
 
 	git config -f .gitmodules submodule."$sm_name".path "$sm_path" &&
 	git config -f .gitmodules submodule."$sm_name".url "$repo" &&
+	for label in $labels
+	do
+		git config --add -f .gitmodules submodule."$sm_name".label "${label}"
+	done &&
 	if test -n "$branch"
 	then
 		git config -f .gitmodules submodule."$sm_name".branch "$branch"
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 5991e3c..01d54e3 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -986,6 +986,7 @@ test_expect_success 'submodule with UTF-8 name' '
 '
 
 test_expect_success 'submodule add clone shallow submodule' '
+	test_when_finished "rm -rf super" &&
 	mkdir super &&
 	pwd=$(pwd) &&
 	(
@@ -999,5 +1000,36 @@ test_expect_success 'submodule add clone shallow submodule' '
 	)
 '
 
+test_expect_success 'submodule add records a label' '
+	test_when_finished "rm -rf super" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --label labelA file://"$pwd"/example2 submodule &&
+		git config -f .gitmodules submodule."submodule".label >actual &&
+		echo labelA >expected &&
+		test_cmp expected actual
+	)
+'
+
+cat >expected <<-EOF
+labelA
+labelB
+EOF
+
+test_expect_success 'submodule add records multiple lables' '
+	test_when_finished "rm -rf super" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --label=labelA -l labelB file://"$pwd"/example2 submodule &&
+		git config --get-all -f .gitmodules submodule."submodule".label >../actual
+	) &&
+	test_cmp expected actual
+'
 
 test_done
-- 
2.7.0.rc0.42.g77a36b9.dirty

[PATCH 3/5] submodule-config: keep labels around

[Stefan Beller]

We need the submodule groups in a later patch.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule-config.c | 15 +++++++++++++++
 submodule-config.h |  2 ++
 2 files changed, 17 insertions(+)

diff --git a/submodule-config.c b/submodule-config.c
index a32259e..245a0f6 100644
--- a/submodule-config.c
+++ b/submodule-config.c
@@ -60,6 +60,10 @@ static void free_one_config(struct submodule_entry *entry)
 {
 	free((void *) entry->config->path);
 	free((void *) entry->config->name);
+	if (entry->config->labels) {
+		string_list_clear(entry->config->labels, 0);
+		free(entry->config->labels);
+	}
 	free(entry->config);
 }
 
@@ -184,6 +188,7 @@ static struct submodule *lookup_or_create_by_name(struct submodule_cache *cache,
 	submodule->update = NULL;
 	submodule->fetch_recurse = RECURSE_SUBMODULES_NONE;
 	submodule->ignore = NULL;
+	submodule->labels = NULL;
 
 	hashcpy(submodule->gitmodules_sha1, gitmodules_sha1);
 
@@ -324,6 +329,16 @@ static int parse_specific_submodule_config(const char *subsection, int subsectio
 			free((void *) submodule->update);
 			submodule->update = xstrdup(value);
 		}
+	} else if (!strcmp(key, "label")) {
+		if (!value)
+			ret = config_error_nonbool(var);
+		else {
+			if (!submodule->labels) {
+				submodule->labels = xmalloc(sizeof(*submodule->labels));
+				string_list_init(submodule->labels, 1);
+			}
+			string_list_insert(submodule->labels, value);
+		}
 	}
 
 	return ret;
diff --git a/submodule-config.h b/submodule-config.h
index d9bbf9a..df73fd7 100644
--- a/submodule-config.h
+++ b/submodule-config.h
@@ -17,6 +17,8 @@ struct submodule {
 	const char *update;
 	/* the sha1 blob id of the responsible .gitmodules file */
 	unsigned char gitmodules_sha1[20];
+	/* sorted, not as on disk */
+	struct string_list *labels;
 };
 
 int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg);
-- 
2.7.0.rc0.42.g77a36b9.dirty

[PATCH 4/5] submodule update: respect submodule.autoInitialize

[Stefan Beller]

All submodules which are selected via submodule.autoInitialize
are initialized if they were not initialized before updating
the submodules.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/submodule--helper.c | 56 ++++++++++++++++++++++++++++++++++++++++++++-
 t/t7400-submodule-basic.sh  | 32 ++++++++++++++++++++++++++
 2 files changed, 87 insertions(+), 1 deletion(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 05c18a3..61e2488 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -579,9 +579,10 @@ struct submodule_update_clone {
 	const char *prefix;
 	struct module_list list;
 	struct string_list projectlines;
+	struct string_list *initialize;
 	struct pathspec pathspec;
 };
-#define SUBMODULE_UPDATE_CLONE_INIT {0, 0, 0, NULL, NULL, NULL, NULL, NULL, MODULE_LIST_INIT, STRING_LIST_INIT_DUP}
+#define SUBMODULE_UPDATE_CLONE_INIT {0, 0, 0, NULL, NULL, NULL, NULL, NULL, MODULE_LIST_INIT, STRING_LIST_INIT_DUP, NULL}
 
 static void fill_clone_command(struct child_process *cp, int quiet,
 			       const char *prefix, const char *path,
@@ -624,6 +625,7 @@ static int update_clone_get_next_task(struct child_process *cp,
 		const char *update_module = NULL;
 		char *url = NULL;
 		int needs_cloning = 0;
+		int auto_init = 0;
 
 		if (ce_stage(ce)) {
 			if (pp->recursive_prefix)
@@ -667,6 +669,49 @@ static int update_clone_get_next_task(struct child_process *cp,
 		strbuf_reset(&sb);
 		strbuf_addf(&sb, "submodule.%s.url", sub->name);
 		git_config_get_string(sb.buf, &url);
+		if (pp->initialize) {
+			if (sub->labels) {
+				struct string_list_item *item;
+				for_each_string_list_item(item, sub->labels) {
+					strbuf_reset(&sb);
+					strbuf_addf(&sb, "*%s", item->string);
+					if (string_list_has_string(
+					    pp->initialize, sb.buf)) {
+						auto_init = 1;
+						break;
+					}
+				}
+			}
+			if (sub->path) {
+				/*
+				 * NEEDSWORK: This currently works only for
+				 * exact paths, but we want to enable
+				 * inexact matches such wildcards.
+				 */
+				strbuf_reset(&sb);
+				strbuf_addf(&sb, "./%s", sub->path);
+				if (string_list_has_string(pp->initialize, sb.buf)) {
+					auto_init = 1;
+				}
+			}
+			if (sub->name) {
+				/*
+				 * NEEDSWORK: Same as with path. Do we want to
+				 * support wildcards or such?
+				 */
+				strbuf_reset(&sb);
+				strbuf_addf(&sb, ":%s", sub->name);
+				if (string_list_has_string(pp->initialize, sb.buf)) {
+					auto_init = 1;
+				}
+			}
+			if (auto_init) {
+				if (!url) {
+					init_submodule(sub->path, pp->prefix, pp->quiet);
+					url = xstrdup(sub->url);
+				}
+			}
+		}
 		if (!url) {
 			/*
 			 * Only mention uninitialized submodules when its
@@ -733,6 +778,7 @@ static int update_clone_task_finished(int result,
 static int update_clone(int argc, const char **argv, const char *prefix)
 {
 	int max_jobs = -1;
+	const struct string_list *list;
 	struct string_list_item *item;
 	struct submodule_update_clone pp = SUBMODULE_UPDATE_CLONE_INIT;
 
@@ -777,6 +823,14 @@ static int update_clone(int argc, const char **argv, const char *prefix)
 	/* Overlay the parsed .gitmodules file with .git/config */
 	git_config(git_submodule_config, NULL);
 
+	list = git_config_get_value_multi("submodule.autoInitialize");
+	if (list) {
+		pp.initialize = xmalloc(sizeof(*pp.initialize));
+		string_list_init(pp.initialize, 1);
+		for_each_string_list_item(item, list)
+			string_list_insert(pp.initialize, item->string);
+	}
+
 	if (max_jobs < 0)
 		max_jobs = config_parallel_submodules();
 	if (max_jobs < 0)
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 01d54e3..e1ade1e 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1032,4 +1032,36 @@ test_expect_success 'submodule add records multiple lables' '
 	test_cmp expected actual
 '
 
+cat <<EOF > expected
+submodule
+submodule1
+submodule2
+-submodule3
+EOF
+
+test_expect_success 'submodule update auto-initializes submodules' '
+	test_when_finished "rm -rf super super_clone" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --label labelA file://"$pwd"/example2 submodule &&
+		git submodule add --name specialSnowflake file://"$pwd"/example2 submodule1 &&
+		git submodule add file://"$pwd"/example2 submodule2 &&
+		git submodule add file://"$pwd"/example2 submodule3 &&
+		git commit -a -m "create repository with 4 submodules, one is labeled"
+	) &&
+	git clone super super_clone &&
+	(
+		cd super_clone &&
+		git config submodule.autoInitialize \*labelA &&
+		git config --add submodule.autoInitialize :specialSnowflake &&
+		git config --add submodule.autoInitialize ./submodule2 &&
+		git submodule update &&
+		git submodule status |cut -c1,42-52 | tr -d " " >../actual
+	) &&
+	test_cmp actual expected
+'
+
 test_done
-- 
2.7.0.rc0.42.g77a36b9.dirty

[PATCH 5/5] builtin/clone: Configure submodule.autoInitialize via --init-submodule

[Stefan Beller]

When invoking clone with the --init-submodule option, the choice will be
recorded in the submodule.autoInitialize config option.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/git-clone.txt |   6 +++
 builtin/clone.c             |  40 ++++++++++++++++--
 t/t7400-submodule-basic.sh  | 101 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 144 insertions(+), 3 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 6db7b6d..4baf444 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -214,6 +214,12 @@ objects from the source repository into a pack in the cloned repository.
 	repository does not have a worktree/checkout (i.e. if any of
 	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
 
+--init-submodule::
+	After the repository is cloned, specified submodules are cloned.
+	It is possible to give multiple specifications by repeating the
+	argument. This option will be recorded in the repository config
+	as `submodule.autoInitialize`.
+
 --separate-git-dir=<git dir>::
 	Instead of placing the cloned repository where it is supposed
 	to be, place the cloned repository at the specified directory,
diff --git a/builtin/clone.c b/builtin/clone.c
index b004fb4..3c37c3d 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -51,6 +51,22 @@ static struct string_list option_config;
 static struct string_list option_reference;
 static int option_dissociate;
 static int max_jobs = -1;
+static struct string_list init_submodules;
+
+static int init_submodules_cb(const struct option *opt, const char *arg, int unset)
+{
+	struct string_list_item *item;
+	struct string_list sl = STRING_LIST_INIT_DUP;
+
+	if (unset)
+		return -1;
+
+	string_list_split(&sl, arg, ',', -1);
+	for_each_string_list_item(item, &sl)
+		string_list_append((struct string_list *)opt->value, item->string);
+
+	return 0;
+}
 
 static struct option builtin_clone_options[] = {
 	OPT__VERBOSITY(&option_verbosity),
@@ -95,6 +111,8 @@ static struct option builtin_clone_options[] = {
 		   N_("separate git dir from working tree")),
 	OPT_STRING_LIST('c', "config", &option_config, N_("key=value"),
 			N_("set config inside the new repository")),
+	OPT_CALLBACK(0, "init-submodule", &init_submodules, N_("string"),
+			N_("clone specific submodules"), init_submodules_cb),
 	OPT_END()
 };
 
@@ -723,9 +741,17 @@ static int checkout(void)
 	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
 			   sha1_to_hex(sha1), "1", NULL);
 
-	if (!err && option_recursive) {
+	if (err)
+		goto out;
+
+	if (option_recursive || init_submodules.nr > 0) {
 		struct argv_array args = ARGV_ARRAY_INIT;
-		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
+		argv_array_pushl(&args, "submodule", "update", NULL);
+
+		if (option_recursive) {
+			argv_array_pushf(&args, "--init");
+			argv_array_pushf(&args, "--recursive");
+		}
 
 		if (max_jobs != -1)
 			argv_array_pushf(&args, "--jobs=%d", max_jobs);
@@ -733,7 +759,7 @@ static int checkout(void)
 		err = run_command_v_opt(args.argv, RUN_GIT_CMD);
 		argv_array_clear(&args);
 	}
-
+out:
 	return err;
 }
 
@@ -867,6 +893,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 			die(_("--bare and --separate-git-dir are incompatible."));
 		option_no_checkout = 1;
 	}
+	if (init_submodules.nr > 0) {
+		struct string_list_item *item;
+		struct strbuf sb = STRBUF_INIT;
+		for_each_string_list_item(item, &init_submodules) {
+			strbuf_addf(&sb, "submodule.autoInitialize=%s", item->string);
+			string_list_append(&option_config, strbuf_detach(&sb, 0));
+		}
+	}
 
 	if (!option_origin)
 		option_origin = "origin";
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index e1ade1e..e83f403 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -1064,4 +1064,105 @@ test_expect_success 'submodule update auto-initializes submodules' '
 	test_cmp actual expected
 '
 
+cat <<EOF > expected
+submodule
+-submodule1
+EOF
+
+test_expect_success 'clone --init-submodule works' '
+	test_when_finished "rm -rf super super_clone" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --label labelA file://"$pwd"/example2 submodule &&
+		git submodule add file://"$pwd"/example2 submodule1 &&
+		git commit -a -m "create repository with 2 submodules, one is in a group"
+	) &&
+	git clone --init-submodule \*labelA super super_clone &&
+	(
+		cd super_clone &&
+		git submodule status |cut -c1,42-52 | tr -d " " >../actual
+	) &&
+	test_cmp actual expected
+'
+
+cat <<EOF > expected
+-submodule1
+submoduleA
+-submoduleB
+submoduleC
+-submoduleD
+submoduleE
+EOF
+
+test_expect_success 'clone initializes submodules correctly with more than one --init-submodule option' '
+	test_when_finished "rm -rf super super_clone" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --label groupA file://"$pwd"/example2 submoduleA &&
+		git submodule add --label groupB file://"$pwd"/example2 submoduleB &&
+		git submodule add --label groupC file://"$pwd"/example2 submoduleC &&
+		git submodule add --label groupD --name submoduleE file://"$pwd"/example2 submoduleD &&
+		git submodule add --label groupE --name submoduleD file://"$pwd"/example2 submoduleE &&
+		git submodule add file://"$pwd"/example2 submodule1 &&
+		git commit -a -m "create repository with submodules groups"
+	) &&
+	git clone --init-submodule=\*groupA --init-submodule ./submoduleC --init-submodule :submoduleD super super_clone &&
+	(
+		cd super_clone &&
+		git submodule status |cut -c1,42-52 | tr -d " " >../actual
+	) &&
+	test_cmp actual expected
+'
+
+cat <<EOF > expected1
+submoduleA
+-submoduleB
+EOF
+
+cat <<EOF > expected2
+submoduleA
+-submoduleB
+submoduleC
+EOF
+
+test_expect_success 'clone and subsequent updates correctly auto-initialize submodules' '
+	test_when_finished "rm -rf super super_clone" &&
+	mkdir super &&
+	pwd=$(pwd) &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --label LA file://"$pwd"/example2 submoduleA &&
+		git submodule add file://"$pwd"/example2 submoduleB &&
+		git commit -a -m "create repository with submodules groups"
+	) &&
+	git clone --init-submodule=\*LA super super_clone &&
+	(
+		cd super_clone &&
+		git submodule status |cut -c1,42-52 | tr -d " " >../actual
+	) &&
+	test_cmp actual expected1 &&
+	(
+		cd super &&
+		git init &&
+		git submodule add --label LA file://"$pwd"/example2 submoduleC &&
+		git commit -a -m "add another labled submodule"
+	) &&
+	(
+		cd super_clone &&
+		# obtain the new superproject
+		git pull &&
+		# submoduleC should just appear as it has the label LA
+		# which was configured to autoInitialize in git clone
+		git submodule update &&
+		git submodule status |cut -c1,42-52 | tr -d " " >../actual
+	) &&
+	test_cmp actual expected2
+'
 test_done
-- 
2.7.0.rc0.42.g77a36b9.dirty

Re: [PATCH 3/5] submodule-config: keep labels around

[Sebastian Schuberth]

On Sat, Jan 23, 2016 at 1:31 AM, Stefan Beller <sbeller@google.com> wrote:

> We need the submodule groups in a later patch.

The commit message should now say "labels", too, I guess.

-- 
Sebastian Schuberth

Re: [RFC/PATCH 0/5] [WAS: Submodule Groups] Labels and submodule.autoInitialize

[Jens Lehmann]

Disclaimer: Due to my currently very limited Git time budget I only
glanced over the recent discussion and patches. If you think I missed
something already discussed, I'd be happy being pointed to the relevant
discussion so I can catch up and avoid wasting everybody's time.

Am 23.01.2016 um 01:31 schrieb Stefan Beller:
> This series introduces labels which you can attach to submodules like so:
>
>      $ cat .gitmodules
>      [submodule "gcc"]
>          path = gcc
>          url = git://...
>          label = default
>          label = devel
>      [submodule "linux"]
>          path = linux
>          url = git://...
>          label = default
>
>      $ git submodule add --name emacs --label "editor" --label default git://...
>
>      # If upstream has submodules properly labeled, you can make use of them:

Cool. Without having looked at the code I assume you also can label
submodules yourself in .git/config (or your global config) to override
upstream's settings (or use your own labels if .gitmodules does not
contain any)?

>      $ git config --add submodule.autoInitialize "*default"
>      $ git config --add submodule.autoInitialize ":name"
>      $ git config --add submodule.autoInitialize "./by/path"

Ok. Though we might wanna call it submodule.autoUpdate, as initializing
it is only the prerequisite for automatically updating submodules. And
I believe automatically updating is the thing we're after here, right?

I'll try to explain why I believe we should be generous in initializing
submodules: If a submodule in one branch has a label configured to be
automatically updated and hasn't got the same label in another branch,
we still need to initialize the submodule even when we are on the latter
branch in case the user switches to the first branch, right? And the
fetch command needs to fetch submodule changes too when they happen in
a branch where this submodule is part of a label group configured to be
updated automatically, no matter what is currently found in the work
tree.

So I'd propose to:

*) Initialize every submodule present on clone or newly fetched when
    the autoUpdate config is set.

*) Automatically fetch only those submodules that are changed in
    a commit where they have a label configured (in the commit's
    .gitmodules or locally) that is to be checked out.

*) Let "git submodule update" update only those submodules that
    have an autoupdate label configured.

That will make switching between branches with different label
configurations work fine. Or am I missing something here?

And we need to teach diff and status to complain about empty work
trees and missing initialization of submodules that are to be
automatically updated too.

>      # The prefix * denotes a label as found in .gitmodules
>      # : goes before names
>      # path are prefixed ./ currently
>      # both path and names need work

Question: how do I configure all submodules to be automatically
initialized without having to give them a label? "./*"? Or just
setting the option without a specific value?

>      # no --init necessary, partially initializes submodules (only those which
>      # were specified by label, name or path)
>      $ git submodule update

Yup. Just like they will be fetched if they haven't been yet they
should be initialized if they haven't been yet but are configured
to be automatically updated.

>      # time passes, upstream may have added new submodules and we get them without
>      # extra commands!
>      $ git submodule update
>
>      # The above configuration can be given to git clone directly via:
>      $ git clone --init-submodule=*labelA ...

Ok. Expecially nice is the ability to also give names and paths to
"--init-submodule". (but maybe that option should be called
"--autoupdate-submodule" for the reasons stated above?)

> Why?
> ====
> If you have lots of submodules, you probably don't need all of them at once,
> but you have functional units. Some submodules are absolutely required,
> some are optional and only for very specific purposes.

I'd rather like to see this as a special case of sparse checkout, as
this will be a submodule-specific solution to a more generic problem.
But as I understand configuring sparse checkout won't materialize for
quite some time, so I suspect we'll have to add labels first to make
current submodule users happy in the near future. Sparse configuration
can then re-use the infrastructure added here.

> Changes to the previous version with groups:
> ====
> * it's called labels now (it's easier to imagine to attach more than one
>    label to a submodule than having it in more groups)
> * In .git/config we have another name, too ("submodule.autoInitialize")
> * Support for more than just groups, but also names and paths are supported
>    (in a very rudimentary way though)
>
> This series applies on top of sb/submodule-init, or can be found at
> https://github.com/stefanbeller/git/tree/submodule-groups-v3
>
>    Thanks,
>    Stefan
>
> Stefan Beller (5):
>    submodule init: Write submodule registration to stderr
>    git submodule: Teach add to label submodules
>    submodule-config: keep labels around
>    submodule update: respect submodule.autoInitialize
>    builtin/clone: Configure submodule.autoInitialize via --init-submodule
>
>   Documentation/git-clone.txt     |   6 ++
>   Documentation/git-submodule.txt |   5 +-
>   builtin/clone.c                 |  40 +++++++++-
>   builtin/submodule--helper.c     |  58 +++++++++++++-
>   git-submodule.sh                |  14 +++-
>   submodule-config.c              |  15 ++++
>   submodule-config.h              |   2 +
>   t/t7400-submodule-basic.sh      | 165 ++++++++++++++++++++++++++++++++++++++++
>   8 files changed, 298 insertions(+), 7 deletions(-)
>

Re: [PATCH 3/5] submodule-config: keep labels around

[Stefan Beller]

On Sun, Jan 24, 2016 at 10:06 AM, Sebastian Schuberth
<sschuberth@gmail.com> wrote:
> On Sat, Jan 23, 2016 at 1:31 AM, Stefan Beller <sbeller@google.com> wrote:
>
>> We need the submodule groups in a later patch.
>
> The commit message should now say "labels", too, I guess.

Sure, thanks for catching!

>
> --
> Sebastian Schuberth

Re: [RFC/PATCH 0/5] [WAS: Submodule Groups] Labels and submodule.autoInitialize

[Stefan Beller]

On Sun, Jan 24, 2016 at 11:38 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Disclaimer: Due to my currently very limited Git time budget I only
> glanced over the recent discussion and patches. If you think I missed
> something already discussed, I'd be happy being pointed to the relevant
> discussion so I can catch up and avoid wasting everybody's time.
>
> Am 23.01.2016 um 01:31 schrieb Stefan Beller:
>>
>> This series introduces labels which you can attach to submodules like so:
>>
>>      $ cat .gitmodules
>>      [submodule "gcc"]
>>          path = gcc
>>          url = git://...
>>          label = default
>>          label = devel
>>      [submodule "linux"]
>>          path = linux
>>          url = git://...
>>          label = default
>>
>>      $ git submodule add --name emacs --label "editor" --label default
>> git://...
>>
>>      # If upstream has submodules properly labeled, you can make use of
>> them:
>
>
> Cool. Without having looked at the code I assume you also can label
> submodules yourself in .git/config (or your global config) to override
> upstream's settings (or use your own labels if .gitmodules does not
> contain any)?

I am not sure. I'll add a test for that in a reroll and make sure it passes.

>
>>      $ git config --add submodule.autoInitialize "*default"
>>      $ git config --add submodule.autoInitialize ":name"
>>      $ git config --add submodule.autoInitialize "./by/path"
>
>
> Ok. Though we might wanna call it submodule.autoUpdate, as initializing
> it is only the prerequisite for automatically updating submodules. And
> I believe automatically updating is the thing we're after here, right?

I am not sure here, too. I would not mind an occasional "git submodule update"
for whenever I want upstream to come down on my disk. However that's what I
do with "git pull" in the non-submodule case, so you'd expect git pull to
also run the update strategies for all submodules which are configured to
autoUpdate?

That makes sense to me. Though I never use "git pull" to begin with.
I always use fetch and see how to go from there (merge or rebase
after inspecting the code I fetched). That would mean we want to
add the autoUpdate strategy to merge/rebase and the fetching of
submodules to the fetch command?

>
> I'll try to explain why I believe we should be generous in initializing
> submodules: If a submodule in one branch has a label configured to be
> automatically updated and hasn't got the same label in another branch,
> we still need to initialize the submodule even when we are on the latter
> branch in case the user switches to the first branch, right?

No. "git checkout" ought to autoInitalize the submodule in question when
switching branches. I don't want to see initialized, but unused submodules
around (neither empty dirs nor in the .git/config ideally)?


> And the
> fetch command needs to fetch submodule changes too when they happen in
> a branch where this submodule is part of a label group configured to be
> updated automatically, no matter what is currently found in the work
> tree.

Right, as said above fetch needs to fetch all the submodules as well. I wonder
if it needs to fetch all submodule sha1s only or just try to get as
much from the
submodule as possible.

>
> So I'd propose to:
>
> *) Initialize every submodule present on clone or newly fetched when
>    the autoUpdate config is set.

What if you clone branch A and then switch to B ? B has a submodule which
was not initialized in A. I do not think initializing on clone/fetch
is the right thing
to do, but rather the branch switching command (checkout) shall make sure
all its autoUpdate/autoInitialze submodules are setup properly, no?

>
> *) Automatically fetch only those submodules that are changed in
>    a commit where they have a label configured (in the commit's
>    .gitmodules or locally) that is to be checked out.

Not sure I follow here.

>
> *) Let "git submodule update" update only those submodules that
>    have an autoupdate label configured.

Why not update all initialized submodules? (In my imagination
"all initialized submodules" are equal to "all submodules the user is
interested in", i.e. when going from branch A to B, the checkout will
(de-)init submodules as necessary.

>
> That will make switching between branches with different label
> configurations work fine. Or am I missing something here?
>
> And we need to teach diff and status to complain about empty work
> trees and missing initialization of submodules that are to be
> automatically updated too.

What about empty work trees?

I'll add "git status" complaining about missing initialized submodules.

>
>>      # The prefix * denotes a label as found in .gitmodules
>>      # : goes before names
>>      # path are prefixed ./ currently
>>      # both path and names need work
>
>
> Question: how do I configure all submodules to be automatically
> initialized without having to give them a label? "./*"? Or just
> setting the option without a specific value?

I'd guess ./* should do. Path wildcard patterns are not supported in this
series, but I think it would be a viable way.

>
>>      # no --init necessary, partially initializes submodules (only those
>> which
>>      # were specified by label, name or path)
>>      $ git submodule update
>
>
> Yup. Just like they will be fetched if they haven't been yet they
> should be initialized if they haven't been yet but are configured
> to be automatically updated.
>
>>      # time passes, upstream may have added new submodules and we get them
>> without
>>      # extra commands!
>>      $ git submodule update
>>
>>      # The above configuration can be given to git clone directly via:
>>      $ git clone --init-submodule=*labelA ...
>
>
> Ok. Expecially nice is the ability to also give names and paths to
> "--init-submodule". (but maybe that option should be called
> "--autoupdate-submodule" for the reasons stated above?)

If I can understand the discussion above a bit further, I'd be happy
to rename the option.

I think we have some different opinions on when
submodules are initialized (the invariant of what an initalized submodules
means), and resulting from that we also have different opinions on when
to do the (de-)init.

>
>> Why?
>> ====
>> If you have lots of submodules, you probably don't need all of them at
>> once,
>> but you have functional units. Some submodules are absolutely required,
>> some are optional and only for very specific purposes.
>
>
> I'd rather like to see this as a special case of sparse checkout, as
> this will be a submodule-specific solution to a more generic problem.

That makes sense.

> But as I understand configuring sparse checkout won't materialize for
> quite some time, so I suspect we'll have to add labels first to make
> current submodule users happy in the near future. Sparse configuration
> can then re-use the infrastructure added here.
>
>


Thanks,
Stefan

Re: [RFC/PATCH 0/5] [WAS: Submodule Groups] Labels and submodule.autoInitialize

[Jens Lehmann]

Am 25.01.2016 um 19:59 schrieb Stefan Beller:
> On Sun, Jan 24, 2016 at 11:38 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> Am 23.01.2016 um 01:31 schrieb Stefan Beller:
>>>
>>> This series introduces labels which you can attach to submodules like so:
>>>
>>>       $ cat .gitmodules
>>>       [submodule "gcc"]
>>>           path = gcc
>>>           url = git://...
>>>           label = default
>>>           label = devel
>>>       [submodule "linux"]
>>>           path = linux
>>>           url = git://...
>>>           label = default
>>>
>>>       $ git submodule add --name emacs --label "editor" --label default
>>> git://...
>>>
>>>       # If upstream has submodules properly labeled, you can make use of
>>> them:
>>
>>
>> Cool. Without having looked at the code I assume you also can label
>> submodules yourself in .git/config (or your global config) to override
>> upstream's settings (or use your own labels if .gitmodules does not
>> contain any)?
>
> I am not sure. I'll add a test for that in a reroll and make sure it passes.

Thanks.

>>
>>>       $ git config --add submodule.autoInitialize "*default"
>>>       $ git config --add submodule.autoInitialize ":name"
>>>       $ git config --add submodule.autoInitialize "./by/path"
>>
>>
>> Ok. Though we might wanna call it submodule.autoUpdate, as initializing
>> it is only the prerequisite for automatically updating submodules. And
>> I believe automatically updating is the thing we're after here, right?
>
> I am not sure here, too. I would not mind an occasional "git submodule update"
> for whenever I want upstream to come down on my disk.
 >
> However that's what I
> do with "git pull" in the non-submodule case, so you'd expect git pull to
> also run the update strategies for all submodules which are configured to
> autoUpdate?
>
> That makes sense to me. Though I never use "git pull" to begin with.
> I always use fetch and see how to go from there (merge or rebase
> after inspecting the code I fetched). That would mean we want to
> add the autoUpdate strategy to merge/rebase and the fetching of
> submodules to the fetch command?

Hmm, maybe autoUpdate promises too much. After all this config is
just about which submodules are chosen to be updated on clone and
submodule update, not on all the other work tree manipulating
commands.

And it's similar to what sparse does. So what about calling that
"submodule.updateSparse"? Or maybe "submodule.sparseCheckout"?
Suggestions welcome.

>> I'll try to explain why I believe we should be generous in initializing
>> submodules: If a submodule in one branch has a label configured to be
>> automatically updated and hasn't got the same label in another branch,
>> we still need to initialize the submodule even when we are on the latter
>> branch in case the user switches to the first branch, right?
>
> No. "git checkout" ought to autoInitalize the submodule in question when
> switching branches. I don't want to see initialized, but unused submodules
> around (neither empty dirs nor in the .git/config ideally)?

Why not? Empty dirs is what unpopulated submodules look like from day
one (and they make the user aware she cannot add a file of the same
name). And you'll see initialized, but unused submodules around every
time you switch to a branch that doesn't contain this submodule at all.

And keeping them initialized even if they aren't currently checked out
is the only way they can keep their settings when the user is switching
between branches.

>> And the
>> fetch command needs to fetch submodule changes too when they happen in
>> a branch where this submodule is part of a label group configured to be
>> updated automatically, no matter what is currently found in the work
>> tree.
>
> Right, as said above fetch needs to fetch all the submodules as well. I wonder
> if it needs to fetch all submodule sha1s only or just try to get as
> much from the
> submodule as possible.

Right now we just do a simple fetch, but only fetching the SHA-1s could
be an optimization useful for busy submodules later on.

>> So I'd propose to:
>>
>> *) Initialize every submodule present on clone or newly fetched when
>>     the autoUpdate config is set.
>
> What if you clone branch A and then switch to B ? B has a submodule which
> was not initialized in A. I do not think initializing on clone/fetch
> is the right thing
> to do, but rather the branch switching command (checkout) shall make sure
> all its autoUpdate/autoInitialze submodules are setup properly, no?

I disagree. If you init all submodules on clone/fetch you might need
to change the upstream URL right after that. You can't do that on a
subsequent branch switch which needs to initialize the submodule again,
as the former deinit did nuke that configuration.

>> *) Automatically fetch only those submodules that are changed in
>>     a commit where they have a label configured (in the commit's
>>     .gitmodules or locally) that is to be checked out.
>
> Not sure I follow here.

We could restrict fetch to not fetch everything but just those changes
needed for sparse submodule update. To be able to do that it would
have to examine the fetched superproject commits if a submodule changed
and if it is configured to be automatically updated in that commit.

>> *) Let "git submodule update" update only those submodules that
>>     have an autoupdate label configured.
>
> Why not update all initialized submodules? (In my imagination
> "all initialized submodules" are equal to "all submodules the user is
> interested in", i.e. when going from branch A to B, the checkout will
> (de-)init submodules as necessary.

And throw away any customization the user did (to the URL or other
configurations)?

Without this sparse/label/group functionality, init is the way the
user tells us he is interested in a submodule. But when configuring
a label/name/path to update, the old meaning of init is obsolete
and superseded by the new mechanism.

>> That will make switching between branches with different label
>> configurations work fine. Or am I missing something here?
>>
>> And we need to teach diff and status to complain about empty work
>> trees and missing initialization of submodules that are to be
>> automatically updated too.
>
> What about empty work trees?
>
> I'll add "git status" complaining about missing initialized submodules.

If they are to be updated on the next "git submodule update" ;-)

>>
>>>       # The prefix * denotes a label as found in .gitmodules
>>>       # : goes before names
>>>       # path are prefixed ./ currently
>>>       # both path and names need work
>>
>>
>> Question: how do I configure all submodules to be automatically
>> initialized without having to give them a label? "./*"? Or just
>> setting the option without a specific value?
>
> I'd guess ./* should do. Path wildcard patterns are not supported in this
> series, but I think it would be a viable way.

Ok.

>>>       # no --init necessary, partially initializes submodules (only those
>>> which
>>>       # were specified by label, name or path)
>>>       $ git submodule update
>>
>>
>> Yup. Just like they will be fetched if they haven't been yet they
>> should be initialized if they haven't been yet but are configured
>> to be automatically updated.
>>
>>>       # time passes, upstream may have added new submodules and we get them
>>> without
>>>       # extra commands!
>>>       $ git submodule update
>>>
>>>       # The above configuration can be given to git clone directly via:
>>>       $ git clone --init-submodule=*labelA ...
>>
>>
>> Ok. Expecially nice is the ability to also give names and paths to
>> "--init-submodule". (but maybe that option should be called
>> "--autoupdate-submodule" for the reasons stated above?)
>
> If I can understand the discussion above a bit further, I'd be happy
> to rename the option.
>
> I think we have some different opinions on when
> submodules are initialized (the invariant of what an initalized submodules
> means), and resulting from that we also have different opinions on when
> to do the (de-)init.

Yes. But I hope my arguments will convince you ;-)

Re: [RFC/PATCH 0/5] [WAS: Submodule Groups] Labels and submodule.autoInitialize

[Stefan Beller]

On Tue, Jan 26, 2016 at 12:59 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:

>>> Ok. Though we might wanna call it submodule.autoUpdate, as initializing
>>> it is only the prerequisite for automatically updating submodules. And
>>> I believe automatically updating is the thing we're after here, right?
>>
>>
>> I am not sure here, too. I would not mind an occasional "git submodule
>> update"
>> for whenever I want upstream to come down on my disk.
>
>>
>>
>> However that's what I
>> do with "git pull" in the non-submodule case, so you'd expect git pull to
>> also run the update strategies for all submodules which are configured to
>> autoUpdate?
>>
>> That makes sense to me. Though I never use "git pull" to begin with.
>> I always use fetch and see how to go from there (merge or rebase
>> after inspecting the code I fetched). That would mean we want to
>> add the autoUpdate strategy to merge/rebase and the fetching of
>> submodules to the fetch command?
>
>
> Hmm, maybe autoUpdate promises too much.

Yes, this very much. I feel like I get burned whenever I send a large
patch series.
So I want to have this first feature be the smallest "operational
unit" that makes sense.

> After all this config is
> just about which submodules are chosen to be updated on clone and
> submodule update, not on all the other work tree manipulating
> commands.

So you'd imagine that "git submodule update" would remove the
submodule and setup an empty directory in case that submodule is
not configured ? (after switching branches or when just having cloned
that thing.)

>
> And it's similar to what sparse does. So what about calling that
> "submodule.updateSparse"? Or maybe "submodule.sparseCheckout"?
> Suggestions welcome.

I'd only suggest when it's clear to me what that option actually does. :)

>
>>> I'll try to explain why I believe we should be generous in initializing
>>> submodules: If a submodule in one branch has a label configured to be
>>> automatically updated and hasn't got the same label in another branch,
>>> we still need to initialize the submodule even when we are on the latter
>>> branch in case the user switches to the first branch, right?
>>
>>
>> No. "git checkout" ought to autoInitalize the submodule in question when
>> switching branches. I don't want to see initialized, but unused submodules
>> around (neither empty dirs nor in the .git/config ideally)?
>
>
> Why not? Empty dirs is what unpopulated submodules look like from day
> one (and they make the user aware she cannot add a file of the same
> name). And you'll see initialized, but unused submodules around every
> time you switch to a branch that doesn't contain this submodule at all.

I see. We need to keep the dir around to block that file name from being
added. That makes sense.

Mentally I am still stuck in the "very large superproject tracking all the
modules of an operating system" thing, where I'd find it annoying if you'd have
100 top level directories, but you have one populated for your work.

But the reason for name blocking makes sense, so I'll follow with that.

>
> And keeping them initialized even if they aren't currently checked out
> is the only way they can keep their settings when the user is switching
> between branches.

Right, but we could invent some non-initialized, but we keep the
settings somewhere
mode. But as I agree on having them initialized, I'll take this an an
extra point to
follow your reasoning.

>
>>> And the
>>> fetch command needs to fetch submodule changes too when they happen in
>>> a branch where this submodule is part of a label group configured to be
>>> updated automatically, no matter what is currently found in the work
>>> tree.
>>
>>
>> Right, as said above fetch needs to fetch all the submodules as well. I
>> wonder
>> if it needs to fetch all submodule sha1s only or just try to get as
>> much from the
>> submodule as possible.
>
>
> Right now we just do a simple fetch, but only fetching the SHA-1s could
> be an optimization useful for busy submodules later on.

I'd rather not call it optimisation, but a correctness thing. What if you
force-pushed other content to the submodule (the sha1 is gone and
maybe should not be reachable) or the other case where you want to
clone the submodule with depth 1 (that is a serious case, which currently
breaks). In the shallow submodule case you need to have the exact sha1
for cloning, otherwise it doesn't work correctly.

>
>>> So I'd propose to:
>>>
>>> *) Initialize every submodule present on clone or newly fetched when
>>>     the autoUpdate config is set.
>>
>>
>> What if you clone branch A and then switch to B ? B has a submodule which
>> was not initialized in A. I do not think initializing on clone/fetch
>> is the right thing
>> to do, but rather the branch switching command (checkout) shall make sure
>> all its autoUpdate/autoInitialze submodules are setup properly, no?
>
>
> I disagree. If you init all submodules on clone/fetch you might need
> to change the upstream URL right after that. You can't do that on a
> subsequent branch switch which needs to initialize the submodule again,
> as the former deinit did nuke that configuration.

So we need to keep the information around, which we do by keeping
all the modules initialized all the time.

>
>>> *) Automatically fetch only those submodules that are changed in
>>>     a commit where they have a label configured (in the commit's
>>>     .gitmodules or locally) that is to be checked out.
>>
>>
>> Not sure I follow here.
>
>
> We could restrict fetch to not fetch everything but just those changes
> needed for sparse submodule update. To be able to do that it would
> have to examine the fetched superproject commits if a submodule changed
> and if it is configured to be automatically updated in that commit.

ok, that's an optimisation for later? (not strictly needed for the first series)

>
>>> *) Let "git submodule update" update only those submodules that
>>>     have an autoupdate label configured.
>>
>>
>> Why not update all initialized submodules? (In my imagination
>> "all initialized submodules" are equal to "all submodules the user is
>> interested in", i.e. when going from branch A to B, the checkout will
>> (de-)init submodules as necessary.
>
>
> And throw away any customization the user did (to the URL or other
> configurations)?
>
> Without this sparse/label/group functionality, init is the way the
> user tells us he is interested in a submodule. But when configuring
> a label/name/path to update, the old meaning of init is obsolete
> and superseded by the new mechanism.

Or if we keep it at "--initSubmodule" only, which only initializes
a subset of new submodules, the meaning is not superseded.

By having the initSubmodule thing set, the user tells us "I am interested
in all currently initialized submodules plus some more in the future, but
these have not arrived yet. To know which submodules I mean in the future
apply this pattern."

Let's take the simplest case:

A user is interested in all the submodules. So currently they clone
and initialize all of them. When upstream adds a new submodule, their
expectation is broken that all submodules are there and checked out.
by having the autoInit option, we'd just initialize any new submodule
and the user assumption "I have all the submodules" is true after
any "submodule update".

By that point of view, we would not need to keep all submodules initialized,
but only those the user is interested in. No need to have complicated
branch switching rules, but just as now "plus some futureproof rules
to declare my interest of submodules".

>
>>> That will make switching between branches with different label
>>> configurations work fine. Or am I missing something here?
>>>
>>> And we need to teach diff and status to complain about empty work
>>> trees and missing initialization of submodules that are to be
>>> automatically updated too.
>>
>>
>> What about empty work trees?
>>
>> I'll add "git status" complaining about missing initialized submodules.
>
>
> If they are to be updated on the next "git submodule update" ;-)

right.

>
>>>
>>>>       # The prefix * denotes a label as found in .gitmodules
>>>>       # : goes before names
>>>>       # path are prefixed ./ currently
>>>>       # both path and names need work
>>>
>>>
>>>
>>> Question: how do I configure all submodules to be automatically
>>> initialized without having to give them a label? "./*"? Or just
>>> setting the option without a specific value?
>>
>>
>> I'd guess ./* should do. Path wildcard patterns are not supported in this
>> series, but I think it would be a viable way.
>
>
> Ok.
>
>>>>       # no --init necessary, partially initializes submodules (only
>>>> those
>>>> which
>>>>       # were specified by label, name or path)
>>>>       $ git submodule update
>>>
>>>
>>>
>>> Yup. Just like they will be fetched if they haven't been yet they
>>> should be initialized if they haven't been yet but are configured
>>> to be automatically updated.
>>>
>>>>       # time passes, upstream may have added new submodules and we get
>>>> them
>>>> without
>>>>       # extra commands!
>>>>       $ git submodule update
>>>>
>>>>       # The above configuration can be given to git clone directly via:
>>>>       $ git clone --init-submodule=*labelA ...
>>>
>>>
>>>
>>> Ok. Expecially nice is the ability to also give names and paths to
>>> "--init-submodule". (but maybe that option should be called
>>> "--autoupdate-submodule" for the reasons stated above?)
>>
>>
>> If I can understand the discussion above a bit further, I'd be happy
>> to rename the option.
>>
>> I think we have some different opinions on when
>> submodules are initialized (the invariant of what an initalized submodules
>> means), and resulting from that we also have different opinions on when
>> to do the (de-)init.
>
>
> Yes. But I hope my arguments will convince you ;-)

--autoupdate-submodule seems to be one step ahead of my current understanding?

Thanks,
Stefan

Re: [RFC/PATCH 0/5] [WAS: Submodule Groups] Labels and submodule.autoInitialize

[Jens Lehmann]

Am 26.01.2016 um 22:50 schrieb Stefan Beller:
> On Tue, Jan 26, 2016 at 12:59 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>
>>>> Ok. Though we might wanna call it submodule.autoUpdate, as initializing
>>>> it is only the prerequisite for automatically updating submodules. And
>>>> I believe automatically updating is the thing we're after here, right?
>>>
>>>
>>> I am not sure here, too. I would not mind an occasional "git submodule
>>> update"
>>> for whenever I want upstream to come down on my disk.
>>
>>>
>>>
>>> However that's what I
>>> do with "git pull" in the non-submodule case, so you'd expect git pull to
>>> also run the update strategies for all submodules which are configured to
>>> autoUpdate?
>>>
>>> That makes sense to me. Though I never use "git pull" to begin with.
>>> I always use fetch and see how to go from there (merge or rebase
>>> after inspecting the code I fetched). That would mean we want to
>>> add the autoUpdate strategy to merge/rebase and the fetching of
>>> submodules to the fetch command?
>>
>>
>> Hmm, maybe autoUpdate promises too much.
>
> Yes, this very much. I feel like I get burned whenever I send a large
> patch series.
> So I want to have this first feature be the smallest "operational
> unit" that makes sense.

Agreed.

>> After all this config is
>> just about which submodules are chosen to be updated on clone and
>> submodule update, not on all the other work tree manipulating
>> commands.
>
> So you'd imagine that "git submodule update" would remove the
> submodule and setup an empty directory in case that submodule is
> not configured ? (after switching branches or when just having cloned
> that thing.)

Not as a result of the label feature we are talking about here,
I think that should just do what currently happens to removed
submodules: they are left populated but are ignored by status,
diff and submodule update. Removing the content is part of the
recursive submodule update topic.

>> And it's similar to what sparse does. So what about calling that
>> "submodule.updateSparse"? Or maybe "submodule.sparseCheckout"?
>> Suggestions welcome.
>
> I'd only suggest when it's clear to me what that option actually does. :)

Fair enough! ;-)

>>>> And the
>>>> fetch command needs to fetch submodule changes too when they happen in
>>>> a branch where this submodule is part of a label group configured to be
>>>> updated automatically, no matter what is currently found in the work
>>>> tree.
>>>
>>>
>>> Right, as said above fetch needs to fetch all the submodules as well. I
>>> wonder
>>> if it needs to fetch all submodule sha1s only or just try to get as
>>> much from the
>>> submodule as possible.
>>
>>
>> Right now we just do a simple fetch, but only fetching the SHA-1s could
>> be an optimization useful for busy submodules later on.
>
> I'd rather not call it optimisation, but a correctness thing. What if you
> force-pushed other content to the submodule (the sha1 is gone and
> maybe should not be reachable)or the other case where you want to
> clone the submodule with depth 1 (that is a serious case, which currently
> breaks). In the shallow submodule case you need to have the exact sha1
> for cloning, otherwise it doesn't work correctly.

I'm convinced. Correctness it is! :-)

>>>> So I'd propose to:
>>>>
>>>> *) Initialize every submodule present on clone or newly fetched when
>>>>      the autoUpdate config is set.
>>>
>>>
>>> What if you clone branch A and then switch to B ? B has a submodule which
>>> was not initialized in A. I do not think initializing on clone/fetch
>>> is the right thing
>>> to do, but rather the branch switching command (checkout) shall make sure
>>> all its autoUpdate/autoInitialze submodules are setup properly, no?
>>
>>
>> I disagree. If you init all submodules on clone/fetch you might need
>> to change the upstream URL right after that. You can't do that on a
>> subsequent branch switch which needs to initialize the submodule again,
>> as the former deinit did nuke that configuration.
>
> So we need to keep the information around, which we do by keeping
> all the modules initialized all the time.

Yup.

>>>> *) Automatically fetch only those submodules that are changed in
>>>>      a commit where they have a label configured (in the commit's
>>>>      .gitmodules or locally) that is to be checked out.
>>>
>>>
>>> Not sure I follow here.
>>
>>
>> We could restrict fetch to not fetch everything but just those changes
>> needed for sparse submodule update. To be able to do that it would
>> have to examine the fetched superproject commits if a submodule changed
>> and if it is configured to be automatically updated in that commit.
>
> ok, that's an optimisation for later? (not strictly needed for the first series)

Definitely.

>>>> *) Let "git submodule update" update only those submodules that
>>>>      have an autoupdate label configured.
>>>
>>>
>>> Why not update all initialized submodules? (In my imagination
>>> "all initialized submodules" are equal to "all submodules the user is
>>> interested in", i.e. when going from branch A to B, the checkout will
>>> (de-)init submodules as necessary.
>>
>>
>> And throw away any customization the user did (to the URL or other
>> configurations)?
>>
>> Without this sparse/label/group functionality, init is the way the
>> user tells us he is interested in a submodule. But when configuring
>> a label/name/path to update, the old meaning of init is obsolete
>> and superseded by the new mechanism.
>
> Or if we keep it at "--initSubmodule" only, which only initializes
> a subset of new submodules, the meaning is not superseded.
>
> By having the initSubmodule thing set, the user tells us "I am interested
> in all currently initialized submodules plus some more in the future, but
> these have not arrived yet. To know which submodules I mean in the future
> apply this pattern."
>
> Let's take the simplest case:
>
> A user is interested in all the submodules. So currently they clone
> and initialize all of them. When upstream adds a new submodule, their
> expectation is broken that all submodules are there and checked out.
> by having the autoInit option, we'd just initialize any new submodule
> and the user assumption "I have all the submodules" is true after
> any "submodule update".
>
> By that point of view, we would not need to keep all submodules initialized,
> but only those the user is interested in. No need to have complicated
> branch switching rules, but just as now "plus some futureproof rules
> to declare my interest of submodules".

I'm not sure what "complicated branch switching rules" you are
referring to here, as far as I can see these only happen when we do
not automatically initialize all submodules. What am I missing?

You'd have to deal with initialized but not to be updated submodules
anyway (due to the user choosing a different label or upstream
changing label assigments). So it looks to me like the approach to
initialize them all as soon as they appear is easier to grok. And
update, diff and status will just skip all submodules that don't
match the configured label(s).

Additionally this will make it easier to e.g. change the upstream
URL of the submodules in one go, as this has to be done after they
have been initialized. If you clone the android repo from a local
mirror it'd be great to just update all URL settings once right
after clone instead of having to do that again each time you choose
a different group.

So I'm not per se against a lazy submodule init like you seem to
propose it, but I believe it'd be better to just init them all as
soon as they appear.

>>>>>        # The prefix * denotes a label as found in .gitmodules
>>>>>        # : goes before names
>>>>>        # path are prefixed ./ currently
>>>>>        # both path and names need work
>>>>
>>>>
>>>>
>>>> Question: how do I configure all submodules to be automatically
>>>> initialized without having to give them a label? "./*"? Or just
>>>> setting the option without a specific value?
>>>
>>>
>>> I'd guess ./* should do. Path wildcard patterns are not supported in this
>>> series, but I think it would be a viable way.
>>
>>
>> Ok.
>>
>>>>>        # no --init necessary, partially initializes submodules (only
>>>>> those
>>>>> which
>>>>>        # were specified by label, name or path)
>>>>>        $ git submodule update
>>>>
>>>>
>>>>
>>>> Yup. Just like they will be fetched if they haven't been yet they
>>>> should be initialized if they haven't been yet but are configured
>>>> to be automatically updated.
>>>>
>>>>>        # time passes, upstream may have added new submodules and we get
>>>>> them
>>>>> without
>>>>>        # extra commands!
>>>>>        $ git submodule update
>>>>>
>>>>>        # The above configuration can be given to git clone directly via:
>>>>>        $ git clone --init-submodule=*labelA ...
>>>>
>>>>
>>>>
>>>> Ok. Expecially nice is the ability to also give names and paths to
>>>> "--init-submodule". (but maybe that option should be called
>>>> "--autoupdate-submodule" for the reasons stated above?)
>>>
>>>
>>> If I can understand the discussion above a bit further, I'd be happy
>>> to rename the option.
>>>
>>> I think we have some different opinions on when
>>> submodules are initialized (the invariant of what an initalized submodules
>>> means), and resulting from that we also have different opinions on when
>>> to do the (de-)init.
>>
>>
>> Yes. But I hope my arguments will convince you ;-)
>
> --autoupdate-submodule seems to be one step ahead of my current understanding?

Yes, sorry for the confusion.

Re: [RFC/PATCH 0/5] [WAS: Submodule Groups] Labels and submodule.autoInitialize

[Stefan Beller]

On Sun, Jan 31, 2016 at 1:09 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>> After all this config is
>>> just about which submodules are chosen to be updated on clone and
>>> submodule update, not on all the other work tree manipulating
>>> commands.
>>
>>
>> So you'd imagine that "git submodule update" would remove the
>> submodule and setup an empty directory in case that submodule is
>> not configured ? (after switching branches or when just having cloned
>> that thing.)
>
>
> Not as a result of the label feature we are talking about here,
> I think that should just do what currently happens to removed
> submodules: they are left populated but are ignored by status,
> diff and submodule update. Removing the content is part of the
> recursive submodule update topic.

That is our second next big difference in understanding.
When having an autoInitialize option, we would not go further into
making rules for when to update submodules. "git submodule update"
would just update all initialzed submodules. (This thought ties well
together with only initializing those submodules that you want instead
of all of them.)

Now that we want to init all the submodules, it makes sense to only
update those we are interested in.

So from what I understand now:

(If labels are configured, then)

 * init a submodule at the earliest time possible, (i.e. while
   cloning or when upstream added a new submodule and
   we fetch it? Though then the init would be performed not in
   clone/fetch, but the "submodule update" step)

 * never deinit submodules automatically

 * submodule update is picky what to update by default.
   It will only update by label selection or directly specified submodules
   (git submodule update foo will update foo no matter if foo is in the
   selected group, a plain "git submodule update" however will only
   update group/label selected modules.)

>>>
>>>
>>> And throw away any customization the user did (to the URL or other
>>> configurations)?
>>>
>>> Without this sparse/label/group functionality, init is the way the
>>> user tells us he is interested in a submodule. But when configuring
>>> a label/name/path to update, the old meaning of init is obsolete
>>> and superseded by the new mechanism.
>>
>>
>> Or if we keep it at "--initSubmodule" only, which only initializes
>> a subset of new submodules, the meaning is not superseded.
>>
>> By having the initSubmodule thing set, the user tells us "I am interested
>> in all currently initialized submodules plus some more in the future, but
>> these have not arrived yet. To know which submodules I mean in the future
>> apply this pattern."
>>
>> Let's take the simplest case:
>>
>> A user is interested in all the submodules. So currently they clone
>> and initialize all of them. When upstream adds a new submodule, their
>> expectation is broken that all submodules are there and checked out.
>> by having the autoInit option, we'd just initialize any new submodule
>> and the user assumption "I have all the submodules" is true after
>> any "submodule update".
>>
>> By that point of view, we would not need to keep all submodules
>> initialized,
>> but only those the user is interested in. No need to have complicated
>> branch switching rules, but just as now "plus some futureproof rules
>> to declare my interest of submodules".
>
>
> I'm not sure what "complicated branch switching rules" you are
> referring to here, as far as I can see these only happen when we do
> not automatically initialize all submodules. What am I missing?

I was assuming we need to delete the submodules and preserve
their state somewhere.

So what I understand now:

$ git checkout <anotherbranch> # will not touch submodules, however
$ git submodule update # may update a different selection as the
$ # selection changed by a different .gitmodules file

>
> You'd have to deal with initialized but not to be updated submodules
> anyway (due to the user choosing a different label or upstream
> changing label assigments). So it looks to me like the approach to
> initialize them all as soon as they appear is easier to grok. And
> update, diff and status will just skip all submodules that don't
> match the configured label(s).

We can teach update, diff and status to ignore those non selected
submodules just fine, but I still think it is somewhat ugly.
(Consider the build system, which now needs to somehow know
which submodules to build. The non selected modules may
error out when building as their dependencies are wrong/not met;
also humans don't like wading through a ton of empty directories
to find their pet project)

>
> Additionally this will make it easier to e.g. change the upstream
> URL of the submodules in one go, as this has to be done after they
> have been initialized. If you clone the android repo from a local
> mirror it'd be great to just update all URL settings once right
> after clone instead of having to do that again each time you choose
> a different group.
>
> So I'm not per se against a lazy submodule init like you seem to
> propose it, but I believe it'd be better to just init them all as
> soon as they appear.

ok, I'll prepare a patch series for that.

>>> Yes. But I hope my arguments will convince you ;-)
>>
>>
>> --autoupdate-submodule seems to be one step ahead of my current
>> understanding?
>
>
> Yes, sorry for the confusion.

So now is time to start bikeshedding (aka naming things) again ;)

The main feature I'd understand is to only apply submodule update, diff, status
(and maybe more) to the selected submodules. So maybe

    $ git clone --apply-actions-to-label <label/path/name pattern>
[may be repeated]

    $ git submodule update # no additional arguments

    $ git submodule add --label <label name>

In the .gitmodules file we can still have "submodule.$foo.label" but in the
.git/config we may want to have "submodule.action-on-label" instead ?

Thanks,
Stefan