[RFC/PATCH 0/4] Add submodule test harness

[RFC/PATCH 0/4] Add submodule test harness

[Jens Lehmann]

So here is my proposal for a single test harness for submodule updates. It is
intended to be easily applicable to all work tree manipulating commands. The
current version tests the status quo (and it documents two bugs in current
Git). This framework will be extended to cover recursive submodule update too
in a later series.

The first patch adds a simple helper function to the test lib which makes
it easier to test for an empty submodule directory.

The second patch contains the heavy lifting, it adds the test framework for
switching submodules. Currently only transitions without merge conflicts are
tested for, I intend to add others producing merge conflicts in a follow-up
series.

The third and forth patch show how to apply this framework to a simple
command (checkout) and a more complicated case where a helper function is
used to execute a preparing command (diff) which produces the input for the
then to-be-tested command (apply).

I'm currently working on adding this harness to more commands. read-tree and
reset are finished, merge and pull still show some oddities when used with
the --no-ff option, also cherry-pick and checkout-index are not working as
expected yet. And then there are am, apply, bisect, rebase, revert & stash
apply which still need to be covered.

Jens Lehmann (4):
  test-lib: add test_dir_is_empty()
  Submodules: Add the lib-submodule-update.sh test library
  checkout: call the new submodule update test framework
  apply: add t4137 for submodule updates

 t/lib-submodule-update.sh     | 627 ++++++++++++++++++++++++++++++++++++++++++
 t/t2013-checkout-submodule.sh |   5 +
 t/t4137-apply-submodule.sh    |  20 ++
 t/test-lib-functions.sh       |  11 +
 4 files changed, 663 insertions(+)
 create mode 100755 t/lib-submodule-update.sh
 create mode 100755 t/t4137-apply-submodule.sh

-- 
1.9.1.327.g3d8d896

[RFC/PATCH 1/4] test-lib: add test_dir_is_empty()

[Jens Lehmann]

For the upcoming submodule test framework we often need to assert that an
empty directory exists in the work tree. Add the test_dir_is_empty()
function which asserts that the given argument is an empty directory.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---

I believe this one is pretty straightforward (unless I missed that this
functionality already exists someplace I forgot to look ;-).

 t/test-lib-functions.sh | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index 158e10a..93d10cd 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -489,6 +489,17 @@ test_path_is_dir () {
 	fi
 }

+# Check if the directory exists and is empty as expected, barf otherwise.
+test_dir_is_empty () {
+	test_path_is_dir "$1" &&
+	if test $(ls -a1 "$1" | wc -l) != 2
+	then
+		echo "Directory '$1' is not empty, it contains:"
+		ls -la "$1"
+		return 1
+	fi
+}
+
 test_path_is_missing () {
 	if [ -e "$1" ]
 	then
-- 
1.9.1.327.g3d8d896

[RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

[Jens Lehmann]

Add this test library to simplify covering all combinations of submodule
update scenarios without having to add those to a test of each work tree
manipulating command over and over again.

The functions test_submodule_switch() and test_submodule_forced_switch()
are intended to be called from a test script with a single argument. This
argument is either a work tree manipulating command (including any command
line options) or a function (when more than a single git command is needed
to switch work trees from the current HEAD to another commit). This
command (or function) is passed a target branch as argument. The two new
functions check that each submodule transition is handled as expected,
which currently means that submodule work trees are not affected until
"git submodule update" is called. The "forced" variant is for commands
using their '-f' or '--hard' option and expects them to overwrite local
modifications as a result. Each of these two functions contains 14
tests_expect_* calls.

Calling one of these test functions the first time creates a repository
named "submodule_update_repo". At first it contains two files, then a
single submodule is added in another commit followed by commits covering
all relevant submodule modifications. This repository is newly cloned into
the "submodule_update" for each test_expect_* to avoid interference
between different parts of the test functions (some to-be-tested commands
also manipulate refs along with the work tree, e.g. "git reset").

Follow-up commits will then call these two test functions for all work
tree manipulating commands (with a combination of all their options
relevant to what they do with the work tree) making sure they work as
expected. Later this test library will be extended to cover merges
resulting in conflicts too. Also it is intended to be easily extendable
for the recursive update functionality, where even more combinations of
submodule modifications have to be tested for.

This version documents two bugs in current Git with expected failures:

*) When a submodule is replaced with a tracked file of the same name the
   submodule work tree including any local modifications (and even the
   whole history if it uses a .git directory instead of a gitfile!) is
   simply removed.

*) Forced work tree updates happily manipulate files in the directory of a
   submodule that has just been removed in the superproject (but is of
   course still present in the work tree due to the way submodules are
   currently handled). This becomes dangerous when files in the submodule
   directory are overwritten by files from the new superproject commit, as
   any modifications to the submodule files will be lost) and is expected
   to also destroy history in the - admittedly unlikely case - the new
   commit adds a file named ".git" to the submodule directory.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---

I think the first bug really needs to be fixed, as that behavior is extremely
nasty. We should always protect work tree modifications (unless forced) and
*never* remove a .git directory (even when forced).

I'm not so sure about the second one. Even though I believe the current
behavior is not correct (switching commits should never mess around in a
submodule directory) it may be that people who committed a directory =>
submodule transition (or vice versa) are depending on the current
behavior. Fixing the bug would forbid to repeat the submodule => directory
transition and only allow that after the user removed the submodule directory
manually. And as the potential fallout of this bug is not as disastrous as
for the first bug, I might be convinced we should not fix it.

 t/lib-submodule-update.sh | 627 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 627 insertions(+)
 create mode 100755 t/lib-submodule-update.sh

diff --git a/t/lib-submodule-update.sh b/t/lib-submodule-update.sh
new file mode 100755
index 0000000..de5d54e
--- /dev/null
+++ b/t/lib-submodule-update.sh
@@ -0,0 +1,627 @@
+# Create a submodule layout used for all tests below.
+#
+# The following use cases are covered:
+# - New submodule (no_submodule => add_sub1)
+# - Removed submodule (add_sub1 => remove_sub1)
+# - Updated submodule (add_sub1 => modify_sub1)
+# - Submodule updated to invalid commit (add_sub1 => invalid_sub1)
+# - Submodule updated from invalid commit (invalid_sub1 => valid_sub1)
+# - Submodule replaced by tracked files in directory (add_sub1 =>
+#   replace_sub1_with_directory)
+# - Directory containing tracked files replaced by submodule
+#   (replace_sub1_with_directory => replace_directory_with_sub1)
+# - Submodule replaced by tracked file with the same name (add_sub1 =>
+#   replace_sub1_with_file)
+# - Tracked file replaced by submodule (replace_sub1_with_file =>
+#   replace_file_with_sub1)
+#
+#                   --O-----O
+#                  /  ^     replace_directory_with_sub1
+#                 /   replace_sub1_with_directory
+#                /----O
+#               /     ^
+#              /      modify_sub1
+#      O------O-------O
+#      ^      ^\      ^
+#      |      | \     remove_sub1
+#      |      |  -----O-----O
+#      |      |   \   ^     replace_file_with_sub1
+#      |      |    \  replace_sub1_with_file
+#      |   add_sub1 --O-----O
+# no_submodule        ^     valid_sub1
+#                     invalid_sub1
+#
+create_lib_submodule_repo () {
+	git init submodule_update_repo &&
+	(
+		cd submodule_update_repo &&
+		echo "expect" >>.gitignore &&
+		echo "actual" >>.gitignore &&
+		echo "x" >file1 &&
+		echo "y" >file2 &&
+		git add .gitignore file1 file2 &&
+		git commit -m "Base" &&
+		git branch "no_submodule" &&
+
+		git checkout -b "add_sub1" &&
+		git submodule add ./. sub1 &&
+		git commit -m "Add sub1" &&
+		git checkout -b remove_sub1 &&
+		git revert HEAD &&
+
+		git checkout -b "modify_sub1" "add_sub1" &&
+		git submodule update &&
+		(
+			cd sub1 &&
+			git fetch &&
+			git checkout -b "modifications" &&
+			echo "z" >file2 &&
+			echo "x" >file3 &&
+			git add file2 file3 &&
+			git commit -m "modified file2 and added file3" &&
+			git push origin modifications
+		) &&
+		git add sub1 &&
+		git commit -m "Modify sub1" &&
+
+		git checkout -b "replace_sub1_with_directory" "add_sub1" &&
+		git submodule update &&
+		(
+			cd sub1 &&
+			git checkout modifications
+		) &&
+		git rm --cached sub1 &&
+		rm sub1/.git* &&
+		git config -f .gitmodules --remove-section "submodule.sub1" &&
+		git add .gitmodules sub1/* &&
+		git commit -m "Replace sub1 with directory" &&
+		git checkout -b replace_directory_with_sub1 &&
+		git revert HEAD &&
+
+		git checkout -b "replace_sub1_with_file" "add_sub1" &&
+		git rm sub1 &&
+		echo "content" >sub1 &&
+		git add sub1 &&
+		git commit -m "Replace sub1 with file" &&
+		git checkout -b replace_file_with_sub1 &&
+		git revert HEAD &&
+
+		git checkout -b "invalid_sub1" "add_sub1" &&
+		git update-index --cacheinfo 160000 0123456789012345678901234567890123456789 sub1 &&
+		git commit -m "Invalid sub1 commit" &&
+		git checkout -b valid_sub1 &&
+		git revert HEAD &&
+		git checkout master
+	)
+}
+
+# Helper function to replace gitfile with .git directory
+replace_gitfile_with_git_dir () {
+	(
+		cd "$1" &&
+		git_dir="$(git rev-parse --git-dir)" &&
+		rm -f .git &&
+		cp -a "$git_dir" .git &&
+		GIT_WORK_TREE=. git config --unset core.worktree
+	)
+}
+
+# Test that the .git directory in the submodule is unchanged (except for the
+# core.worktree setting)
+test_git_directory_is_unchanged () {
+	(
+		cd "$1" &&
+		git config core.worktree "../../../$1"
+	) &&
+	git diff -r ".git/modules/$1" "$1/.git" &&
+	(
+		cd "$1" &&
+		GIT_WORK_TREE=. git config --unset core.worktree
+	)
+}
+
+# Helper function to be executed at the start of every test below, it sets up
+# the submodule repo if it doesn't exist and configures the most problematic
+# settings for diff.ignoreSubmodules.
+prolog () {
+	(test -d submodule_update_repo || create_lib_submodule_repo) &&
+	test_config_global diff.ignoreSubmodules all &&
+	test_config diff.ignoreSubmodules all
+}
+
+# Helper function to bring work tree back into the state given by the
+# commit. This includes trying to populate sub1 accordingly if it exists and
+# should be updated to an existing commit.
+reset_work_tree_to () {
+	rm -rf submodule_update &&
+	git clone submodule_update_repo submodule_update &&
+	(
+		cd submodule_update &&
+		rm -rf sub1 &&
+		git checkout -f "$1" &&
+		git status -u -s >actual &&
+		test_must_be_empty actual &&
+		sha1=$(git ls-tree HEAD "sub1" 2>/dev/null | grep 160000 | tr '\t' ' ' | cut -d ' ' -f3) &&
+		if test -n "$sha1" &&
+		   test $(cd "sub1" && git rev-parse --verify "$sha1^{commit}")
+		then
+			git submodule update --init --recursive "sub1"
+		fi
+	)
+}
+
+# Test that the superproject contains the content according to commit "$1"
+# (the work tree must match the index for everything but submodules but the
+# index must exactly match the given commit including any submodule SHA-1s).
+test_superproject_content () {
+	git diff-index --cached "$1" >actual &&
+	test_must_be_empty actual &&
+	git diff-files --ignore-submodules >actual &&
+	test_must_be_empty actual
+}
+
+# Test that the given submodule at path "$1" contains the content according
+# to the submodule commit recorded in the superproject's commit "$2"
+test_submodule_content () {
+	if test $# != 2
+	then
+		echo "test_submodule_content needs two arguments"
+		return 1
+	fi &&
+	submodule="$1" &&
+	commit="$2" &&
+	test -d "$submodule"/ &&
+	if ! test -f "$submodule"/.git && ! test -d "$submodule"/.git
+	then
+		echo "Submodule $submodule is not populated"
+		return 1
+	fi &&
+	sha1=$(git ls-tree "$commit" "$submodule" 2>/dev/null | tr '\t' ' ' | cut -d ' ' -f3) &&
+	if test -z "$sha1"
+	then
+		echo "Couldn't retrieve SHA-1 of $submodule for $commit"
+		return 1
+	fi &&
+	(
+		cd "$submodule" &&
+		git status -u -s >actual &&
+		test_must_be_empty actual &&
+		git diff "$sha1" >actual &&
+		test_must_be_empty actual
+	)
+}
+
+# Test that the following transitions are correctly handled:
+# - Updated submodule
+# - New submodule
+# - Removed submodule
+# - Directory containing tracked files replaced by submodule
+# - Submodule replaced by tracked files in directory
+# - Submodule replaced by tracked file with the same name
+# - tracked file replaced by submodule
+#
+# The default is that submodule contents aren't changed until "git submodule
+# update" is run. And even then that command doesn't delete the work tree of
+# a removed submodule.
+#
+# Removing a submodule containing a .git directory must fail even when forced
+# to protect the history!
+#
+
+# Test that submodule contents are currently not updated when switching
+# between commits that change a submodule.
+test_submodule_switch () {
+	command="$1"
+	######################### Appearing submodule #########################
+	# Switching to a commit letting a submodule appear creates empty dir ...
+	test_expect_success "$command: added submodule creates empty directory" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			$command add_sub1 &&
+			test_superproject_content origin/add_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... and doesn't care if it already exists ...
+	test_expect_success "$command: added submodule leaves existing empty directory alone" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			mkdir sub1 &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			$command add_sub1 &&
+			test_superproject_content origin/add_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... unless there is an untracked file in its place.
+	test_expect_success "$command: added submodule doesn't remove untracked unignored file with same name" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			echo -n >sub1 &&
+			test_must_fail $command add_sub1 &&
+			test_superproject_content origin/no_submodule &&
+			test_must_be_empty sub1
+		)
+	'
+	# Replacing a tracked file with a submodule produces an empty
+	# directory ...
+	test_expect_success "$command: replace tracked file with submodule creates empty directory" '
+		prolog &&
+		reset_work_tree_to replace_sub1_with_file &&
+		(
+			cd submodule_update &&
+			git branch -t replace_file_with_sub1 origin/replace_file_with_sub1 &&
+			$command replace_file_with_sub1 &&
+			test_superproject_content origin/replace_file_with_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/replace_file_with_sub1
+		)
+	'
+	# ... as does removing a directory with tracked files with a
+	# submodule.
+	test_expect_success "$command: replace directory with submodule" '
+		prolog &&
+		reset_work_tree_to replace_sub1_with_directory &&
+		(
+			cd submodule_update &&
+			git branch -t replace_directory_with_sub1 origin/replace_directory_with_sub1 &&
+			$command replace_directory_with_sub1 &&
+			test_superproject_content origin/replace_directory_with_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/replace_directory_with_sub1
+		)
+	'
+
+	######################## Disappearing submodule #######################
+	# Removing a submodule doesn't remove its work tree ...
+	test_expect_success "$command: removed submodule leaves submodule directory and its contents in place" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t remove_sub1 origin/remove_sub1 &&
+			$command remove_sub1 &&
+			test_superproject_content origin/remove_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... especially when it contains a .git directory.
+	test_expect_success "$command: removed submodule leaves submodule containing a .git directory alone" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t remove_sub1 origin/remove_sub1 &&
+			replace_gitfile_with_git_dir sub1 &&
+			$command remove_sub1 &&
+			test_superproject_content origin/remove_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+	# Replacing a submodule with files in a directory must fail as the
+	# submodule work tree isn't removed ...
+	test_expect_success "$command: replace submodule with a directory fails" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_directory origin/replace_sub1_with_directory &&
+			test_must_fail $command replace_sub1_with_directory &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... especially when it contains a .git directory.
+	test_expect_success "$command: replace submodule containing a .git directory with a directory fails" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_directory origin/replace_sub1_with_directory &&
+			replace_gitfile_with_git_dir sub1 &&
+			test_must_fail $command replace_sub1_with_directory &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+	# Replacing it with a file must fail as it could throw away any local
+	# work tree changes ...
+	test_expect_failure "$command: replace submodule directory with a file must fail" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_file origin/replace_sub1_with_file &&
+			test_must_fail $command replace_sub1_with_file &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... or even destroy unpushed parts of submodule history if that
+	# still uses a .git directory.
+	test_expect_failure "$command: replace submodule directory with a file must fail" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_file origin/replace_sub1_with_file &&
+			replace_gitfile_with_git_dir sub1 &&
+			test_must_fail $command replace_sub1_with_file &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+
+	########################## Modified submodule #########################
+	# Updating a submodule sha1 doesn't update the submodule's work tree
+	test_expect_success "$command: modified submodule does not update submodule work tree" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t modify_sub1 origin/modify_sub1 &&
+			$command modify_sub1 &&
+			test_superproject_content origin/modify_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			git submodule update &&
+			test_submodule_content sub1 origin/modify_sub1
+		)
+	'
+
+	# Updating a submodule to an invalid sha1 doesn't update the
+	# submodule's work tree, subsequent update will fail
+	test_expect_success "$command: modified submodule does not update submodule work tree to invalid commit" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t invalid_sub1 origin/invalid_sub1 &&
+			$command invalid_sub1 &&
+			test_superproject_content origin/invalid_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_must_fail git submodule update &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# Updating a submodule from an invalid sha1 doesn't update the
+	# submodule's work tree, subsequent update will succeed
+	test_expect_success "$command: modified submodule does not update submodule work tree from invalid commit" '
+		prolog &&
+		reset_work_tree_to invalid_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t valid_sub1 origin/valid_sub1 &&
+			$command valid_sub1 &&
+			test_superproject_content origin/valid_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/valid_sub1
+		)
+	'
+}
+
+# Test that submodule contents are currently not updated when switching
+# between commits that change a submodule, but throwing away local changes in
+# the superproject is allowed.
+test_submodule_forced_switch () {
+	command="$1"
+	######################### Appearing submodule #########################
+	# Switching to a commit letting a submodule appear creates empty dir ...
+	test_expect_success "$command: added submodule creates empty directory" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			$command add_sub1 &&
+			test_superproject_content origin/add_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... and doesn't care if it already exists ...
+	test_expect_success "$command: added submodule leaves existing empty directory alone" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			mkdir sub1 &&
+			$command add_sub1 &&
+			test_superproject_content origin/add_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... unless there is an untracked file in its place.
+	test_expect_success "$command: added submodule does remove untracked unignored file with same name when forced" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			echo -n >sub1 &&
+			$command add_sub1 &&
+			test_superproject_content origin/add_sub1 &&
+			test_dir_is_empty sub1
+		)
+	'
+	# Replacing a tracked file with a submodule produces an empty
+	# directory ...
+	test_expect_success "$command: replace tracked file with submodule creates empty directory" '
+		prolog &&
+		reset_work_tree_to replace_sub1_with_file &&
+		(
+			cd submodule_update &&
+			git branch -t replace_file_with_sub1 origin/replace_file_with_sub1 &&
+			$command replace_file_with_sub1 &&
+			test_superproject_content origin/replace_file_with_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/replace_file_with_sub1
+		)
+	'
+	# ... as does removing a directory with tracked files with a
+	# submodule.
+	test_expect_success "$command: replace directory with submodule" '
+		prolog &&
+		reset_work_tree_to replace_sub1_with_directory &&
+		(
+			cd submodule_update &&
+			git branch -t replace_directory_with_sub1 origin/replace_directory_with_sub1 &&
+			$command replace_directory_with_sub1 &&
+			test_superproject_content origin/replace_directory_with_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/replace_directory_with_sub1
+		)
+	'
+
+	######################## Disappearing submodule #######################
+	# Removing a submodule doesn't remove its work tree ...
+	test_expect_success "$command: removed submodule leaves submodule directory and its contents in place" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t remove_sub1 origin/remove_sub1 &&
+			$command remove_sub1 &&
+			test_superproject_content origin/remove_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... especially when it contains a .git directory.
+	test_expect_success "$command: removed submodule leaves submodule containing a .git directory alone" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t remove_sub1 origin/remove_sub1 &&
+			replace_gitfile_with_git_dir sub1 &&
+			$command remove_sub1 &&
+			test_superproject_content origin/remove_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+	# Replacing a submodule with files in a directory must fail as the
+	# submodule work tree isn't removed ...
+	test_expect_failure "$command: replace submodule with a directory fails" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_directory origin/replace_sub1_with_directory &&
+			test_must_fail $command replace_sub1_with_directory &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... especially when it contains a .git directory.
+	test_expect_failure "$command: replace submodule containing a .git directory with a directory fails" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_directory origin/replace_sub1_with_directory &&
+			replace_gitfile_with_git_dir sub1 &&
+			test_must_fail $command replace_sub1_with_directory &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+	# Replacing it with a file must fail as it could throw away any local
+	# work tree changes ...
+	test_expect_failure "$command: replace submodule directory with a file must fail" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_file origin/replace_sub1_with_file &&
+			test_must_fail $command replace_sub1_with_file &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... or even destroy unpushed parts of submodule history if that
+	# still uses a .git directory.
+	test_expect_failure "$command: replace submodule directory with a file must fail" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_file origin/replace_sub1_with_file &&
+			replace_gitfile_with_git_dir sub1 &&
+			test_must_fail $command replace_sub1_with_file &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+
+	########################## Modified submodule #########################
+	# Updating a submodule sha1 doesn't update the submodule's work tree
+	test_expect_success "$command: modified submodule does not update submodule work tree" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t modify_sub1 origin/modify_sub1 &&
+			$command modify_sub1 &&
+			test_superproject_content origin/modify_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			git submodule update &&
+			test_submodule_content sub1 origin/modify_sub1
+		)
+	'
+	# Updating a submodule to an invalid sha1 doesn't update the
+	# submodule's work tree, subsequent update will fail
+	test_expect_success "$command: modified submodule does not update submodule work tree to invalid commit" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t invalid_sub1 origin/invalid_sub1 &&
+			$command invalid_sub1 &&
+			test_superproject_content origin/invalid_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_must_fail git submodule update &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# Updating a submodule from an invalid sha1 doesn't update the
+	# submodule's work tree, subsequent update will succeed
+	test_expect_success "$command: modified submodule does not update submodule work tree from invalid commit" '
+		prolog &&
+		reset_work_tree_to invalid_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t valid_sub1 origin/valid_sub1 &&
+			$command valid_sub1 &&
+			test_superproject_content origin/valid_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/valid_sub1
+		)
+	'
+}
-- 
1.9.1.327.g3d8d896

[RFC/PATCH 3/4] checkout: call the new submodule update test framework

[Jens Lehmann]

Test that the checkout command updates the work tree as expected with or
without the '-f' flag.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---

I think this should explain how to use the framework with a single command
and some options.

 t/t2013-checkout-submodule.sh | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/t/t2013-checkout-submodule.sh b/t/t2013-checkout-submodule.sh
index 06b18f8..6847f75 100755
--- a/t/t2013-checkout-submodule.sh
+++ b/t/t2013-checkout-submodule.sh
@@ -3,6 +3,7 @@
 test_description='checkout can handle submodules'

 . ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-submodule-update.sh

 test_expect_success 'setup' '
 	mkdir submodule &&
@@ -62,4 +63,8 @@ test_expect_success '"checkout <submodule>" honors submodule.*.ignore from .git/
 	! test -s actual
 '

+test_submodule_switch "git checkout"
+
+test_submodule_forced_switch "git checkout -f"
+
 test_done
-- 
1.9.1.327.g3d8d896

[RFC/PATCH 4/4] apply: add t4137 for submodule updates

[Jens Lehmann]

Test that the apply command updates the work tree as expected for the
'--index' and the '--3way' options (for submodule changes which don't
result in conflicts).

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---

And this shows how to use the new framework when more than a single command
is needed to switch to a new work tree.

 t/t4137-apply-submodule.sh | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
 create mode 100755 t/t4137-apply-submodule.sh

diff --git a/t/t4137-apply-submodule.sh b/t/t4137-apply-submodule.sh
new file mode 100755
index 0000000..f0a0500
--- /dev/null
+++ b/t/t4137-apply-submodule.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+
+test_description='git am handling submodules'
+
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-submodule-update.sh
+
+apply_index () {
+    git diff --ignore-submodules=dirty "..$1" | git apply --index -
+}
+
+test_submodule_switch "apply_index"
+
+apply_3way () {
+    git diff --ignore-submodules=dirty "..$1" | git apply --3way -
+}
+
+test_submodule_switch "apply_3way"
+
+test_done
-- 
1.9.1.327.g3d8d896

Re: [RFC/PATCH 1/4] test-lib: add test_dir_is_empty()

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> For the upcoming submodule test framework we often need to assert that an
> empty directory exists in the work tree. Add the test_dir_is_empty()
> function which asserts that the given argument is an empty directory.
>
> Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
> ---
>
> I believe this one is pretty straightforward (unless I missed that this
> functionality already exists someplace I forgot to look ;-).

I am not very thrilled to see that it depends on "." and ".." to
always exist, which may be true for all POSIX filesystems, but
still...

Do expected callsites of this helper care if "$1" is a symbolic link
that points at an empty directory?

What do expected callsites really want to ensure?  In other words,
why do they care if the directory is empty?  Is it to make sure,
after some operation, they can "rmdir" the directory?

>  t/test-lib-functions.sh | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
> index 158e10a..93d10cd 100644
> --- a/t/test-lib-functions.sh
> +++ b/t/test-lib-functions.sh
> @@ -489,6 +489,17 @@ test_path_is_dir () {
>  	fi
>  }
>
> +# Check if the directory exists and is empty as expected, barf otherwise.
> +test_dir_is_empty () {
> +	test_path_is_dir "$1" &&
> +	if test $(ls -a1 "$1" | wc -l) != 2
> +	then
> +		echo "Directory '$1' is not empty, it contains:"
> +		ls -la "$1"
> +		return 1
> +	fi
> +}
> +
>  test_path_is_missing () {
>  	if [ -e "$1" ]
>  	then

Re: [RFC/PATCH 1/4] test-lib: add test_dir_is_empty()

[David Kastrup]

Junio C Hamano <gitster@pobox.com> writes:

> Jens Lehmann <Jens.Lehmann@web.de> writes:
>
>> For the upcoming submodule test framework we often need to assert that an
>> empty directory exists in the work tree. Add the test_dir_is_empty()
>> function which asserts that the given argument is an empty directory.
>>
>> Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
>> ---
>>
>> I believe this one is pretty straightforward (unless I missed that this
>> functionality already exists someplace I forgot to look ;-).
>
> I am not very thrilled to see that it depends on "." and ".." to
> always exist, which may be true for all POSIX filesystems, but
> still...

Not even there, though few people will likely use / as their work
tree...

-- 
David Kastrup

Re: [RFC/PATCH 1/4] test-lib: add test_dir_is_empty()

[Jens Lehmann]

Am 25.03.2014 21:49, schrieb Junio C Hamano:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> For the upcoming submodule test framework we often need to assert that an
>> empty directory exists in the work tree. Add the test_dir_is_empty()
>> function which asserts that the given argument is an empty directory.
>>
>> Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
>> ---
>>
>> I believe this one is pretty straightforward (unless I missed that this
>> functionality already exists someplace I forgot to look ;-).
> 
> I am not very thrilled to see that it depends on "." and ".." to
> always exist, which may be true for all POSIX filesystems, but
> still...

Agreed. I didn't find any one-liners to do that ("ls -A" isn't
POSIX), so I decided to wrap that in a function. Testing that
"rmdir" on the directory succeeds (because the directory is
empty) would kinda work, but then we'd have to re-create the
directory afterwards, which really doesn't sound like a good
strategy either as the test would manipulate the to-be-tested
object. I'm not terribly happy with depending on "." and ".."
either, but couldn't come up with something better. At least
the test should fail for any filesystem not having the dot
files ...

> Do expected callsites of this helper care if "$1" is a symbolic link
> that points at an empty directory?

Yep, a symbolic link pointing to an empty directory should make
the test fail.

> What do expected callsites really want to ensure?  In other words,
> why do they care if the directory is empty?  Is it to make sure,
> after some operation, they can "rmdir" the directory?

To assert that a submodule is created but *not* populated. This
is intended to catch any possible fallout from the recursive
checkout later, in case that would kick in when it shouldn't.

>>  t/test-lib-functions.sh | 11 +++++++++++
>>  1 file changed, 11 insertions(+)
>>
>> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
>> index 158e10a..93d10cd 100644
>> --- a/t/test-lib-functions.sh
>> +++ b/t/test-lib-functions.sh
>> @@ -489,6 +489,17 @@ test_path_is_dir () {
>>  	fi
>>  }
>>
>> +# Check if the directory exists and is empty as expected, barf otherwise.
>> +test_dir_is_empty () {
>> +	test_path_is_dir "$1" &&
>> +	if test $(ls -a1 "$1" | wc -l) != 2
>> +	then
>> +		echo "Directory '$1' is not empty, it contains:"
>> +		ls -la "$1"
>> +		return 1
>> +	fi
>> +}
>> +
>>  test_path_is_missing () {
>>  	if [ -e "$1" ]
>>  	then
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [RFC/PATCH 1/4] test-lib: add test_dir_is_empty()

[Michael Haggerty]

On 03/26/2014 09:29 AM, Jens Lehmann wrote:
> Am 25.03.2014 21:49, schrieb Junio C Hamano:
>> Jens Lehmann <Jens.Lehmann@web.de> writes:
>>
>>> For the upcoming submodule test framework we often need to assert that an
>>> empty directory exists in the work tree. Add the test_dir_is_empty()
>>> function which asserts that the given argument is an empty directory.
>>>
>>> Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
>>> ---
>>>
>>> I believe this one is pretty straightforward (unless I missed that this
>>> functionality already exists someplace I forgot to look ;-).
>>
>> I am not very thrilled to see that it depends on "." and ".." to
>> always exist, which may be true for all POSIX filesystems, but
>> still...
> 
> Agreed. I didn't find any one-liners to do that ("ls -A" isn't
> POSIX), so I decided to wrap that in a function. Testing that
> "rmdir" on the directory succeeds (because the directory is
> empty) would kinda work, but then we'd have to re-create the
> directory afterwards, which really doesn't sound like a good
> strategy either as the test would manipulate the to-be-tested
> object. I'm not terribly happy with depending on "." and ".."
> either, but couldn't come up with something better. At least
> the test should fail for any filesystem not having the dot
> files ...
> 
>> Do expected callsites of this helper care if "$1" is a symbolic link
>> that points at an empty directory?
> 
> Yep, a symbolic link pointing to an empty directory should make
> the test fail.
> 
>> What do expected callsites really want to ensure?  In other words,
>> why do they care if the directory is empty?  Is it to make sure,
>> after some operation, they can "rmdir" the directory?
> 
> To assert that a submodule is created but *not* populated. This
> is intended to catch any possible fallout from the recursive
> checkout later, in case that would kick in when it shouldn't.
> 
>>>  t/test-lib-functions.sh | 11 +++++++++++
>>>  1 file changed, 11 insertions(+)
>>>
>>> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
>>> index 158e10a..93d10cd 100644
>>> --- a/t/test-lib-functions.sh
>>> +++ b/t/test-lib-functions.sh
>>> @@ -489,6 +489,17 @@ test_path_is_dir () {
>>>  	fi
>>>  }
>>>
>>> +# Check if the directory exists and is empty as expected, barf otherwise.
>>> +test_dir_is_empty () {
>>> +	test_path_is_dir "$1" &&
>>> +	if test $(ls -a1 "$1" | wc -l) != 2
>>> +	then
>>> +		echo "Directory '$1' is not empty, it contains:"
>>> +		ls -la "$1"
>>> +		return 1
>>> +	fi
>>> +}
>>> +
>>>  test_path_is_missing () {
>>>  	if [ -e "$1" ]
>>>  	then

Why not do something like

    test -z "$(ls -a1 "$1" | egrep -v '^\.\.?$')"

I.e., make the test ignore "." and ".." without depending on their
existence?

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: [RFC/PATCH 1/4] test-lib: add test_dir_is_empty()

[Jens Lehmann]

Am 26.03.2014 11:43, schrieb Michael Haggerty:
> On 03/26/2014 09:29 AM, Jens Lehmann wrote:
>> Am 25.03.2014 21:49, schrieb Junio C Hamano:
>>> Jens Lehmann <Jens.Lehmann@web.de> writes:
>>>>  t/test-lib-functions.sh | 11 +++++++++++
>>>>  1 file changed, 11 insertions(+)
>>>>
>>>> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
>>>> index 158e10a..93d10cd 100644
>>>> --- a/t/test-lib-functions.sh
>>>> +++ b/t/test-lib-functions.sh
>>>> @@ -489,6 +489,17 @@ test_path_is_dir () {
>>>>  	fi
>>>>  }
>>>>
>>>> +# Check if the directory exists and is empty as expected, barf otherwise.
>>>> +test_dir_is_empty () {
>>>> +	test_path_is_dir "$1" &&
>>>> +	if test $(ls -a1 "$1" | wc -l) != 2
>>>> +	then
>>>> +		echo "Directory '$1' is not empty, it contains:"
>>>> +		ls -la "$1"
>>>> +		return 1
>>>> +	fi
>>>> +}
>>>> +
>>>>  test_path_is_missing () {
>>>>  	if [ -e "$1" ]
>>>>  	then
> 
> Why not do something like
> 
>     test -z "$(ls -a1 "$1" | egrep -v '^\.\.?$')"
> 
> I.e., make the test ignore "." and ".." without depending on their
> existence?

Thanks, will do so in the next round.

Re: [RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 2343 bytes --]

On Tue, Mar 25, 2014 at 06:05:05PM +0100, Jens Lehmann wrote:
> *) When a submodule is replaced with a tracked file of the same name
>    the submodule work tree including any local modifications (and
>    even the whole history if it uses a .git directory instead of a
>    gitfile!) is simply removed.
> …
> I think the first bug really needs to be fixed, as that behavior is
> extremely nasty. We should always protect work tree modifications
> (unless forced) and *never* remove a .git directory (even when
> forced).

I think this should be covered by the usual “don't allow checkouts
from dirty workdirs unless the dirty-ing changes are easily applied to
the target tree”.

Are we waiting to land this series (or a successor) before starting on
a fix for this issue?  There have been a number of submodule series in
flight recently, and I'm having trouble keeping track of them all ;).

> *) Forced work tree updates happily manipulate files in the
>    directory of a submodule that has just been removed in the
>    superproject (but is of course still present in the work tree due
>    to the way submodules are currently handled). This becomes
>    dangerous when files in the submodule directory are overwritten
>    by files from the new superproject commit, as any modifications
>    to the submodule files will be lost) and is expected to also
>    destroy history in the - admittedly unlikely case - the new
>    commit adds a file named ".git" to the submodule directory.
> …
> I'm not so sure about the second one. Even though I believe the
> current behavior is not correct (switching commits should never mess
> around in a submodule directory)

This should also be covered by the usual “don't allow checkouts from
dirty workdirs unless the dirty-ing changes are easily applied to the
target tree”.  We don't implement this yet, but I'd like to force
users to move any about-to-be-clobbered state from their submodule
into .git/modules/<name>/ (via commits or stashes) before allowing
them to begin the checkout.  Once we've ensured that the state is
preserved out-of-tree, then clobber away ;).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

[Junio C Hamano]

"W. Trevor King" <wking@tremily.us> writes:

> There have been a number of submodule series in
> flight recently, and I'm having trouble keeping track of them all ;).

Unfortunately I share that same feeling X-<.

Could you guys collectively summarize what issues each of these
in-flight topics try to address and how, how close it is to achieve
concensus, and how it interact with other proposed topics?

Thanks.

Re: [RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

[Jens Lehmann]

Am 17.04.2014 18:41, schrieb W. Trevor King:
> On Tue, Mar 25, 2014 at 06:05:05PM +0100, Jens Lehmann wrote:
>> *) When a submodule is replaced with a tracked file of the same name
>>    the submodule work tree including any local modifications (and
>>    even the whole history if it uses a .git directory instead of a
>>    gitfile!) is simply removed.
>> …
>> I think the first bug really needs to be fixed, as that behavior is
>> extremely nasty. We should always protect work tree modifications
>> (unless forced) and *never* remove a .git directory (even when
>> forced).
> 
> I think this should be covered by the usual “don't allow checkouts
> from dirty workdirs unless the dirty-ing changes are easily applied to
> the target tree”.

Nope, the target tree will be removed completely and everything in
it is silently nuked. It should be allowed with '-f', but only if
the submodule contains a gitfile, and never if it contains a .git
directory (which is just what we do for rm too).

> Are we waiting to land this series (or a successor) before starting on
> a fix for this issue?

I think so, as this bug is there for a long time (so I see no urge
to fix it very soon) and my test harness is intended to document
this current bug (and then soon its fix).

>> *) Forced work tree updates happily manipulate files in the
>>    directory of a submodule that has just been removed in the
>>    superproject (but is of course still present in the work tree due
>>    to the way submodules are currently handled). This becomes
>>    dangerous when files in the submodule directory are overwritten
>>    by files from the new superproject commit, as any modifications
>>    to the submodule files will be lost) and is expected to also
>>    destroy history in the - admittedly unlikely case - the new
>>    commit adds a file named ".git" to the submodule directory.
>> …
>> I'm not so sure about the second one. Even though I believe the
>> current behavior is not correct (switching commits should never mess
>> around in a submodule directory)
> 
> This should also be covered by the usual “don't allow checkouts from
> dirty workdirs unless the dirty-ing changes are easily applied to the
> target tree”.  We don't implement this yet, but I'd like to force
> users to move any about-to-be-clobbered state from their submodule
> into .git/modules/<name>/ (via commits or stashes) before allowing
> them to begin the checkout.  Once we've ensured that the state is
> preserved out-of-tree, then clobber away ;).

I'm intending to fix this in the recursive checkout series, as I'm
a) not sure if any users currently depend on that for a submodule
to directory transition and b) recursive checkout is the place to
consistently care about submodule modifications (the submodule
script doesn't do that and it is impossible to change that without
causing trouble to a lot of users.

Re: [RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

[Jens Lehmann]

Am 17.04.2014 21:23, schrieb Junio C Hamano:
> "W. Trevor King" <wking@tremily.us> writes:
> 
>> There have been a number of submodule series in
>> flight recently, and I'm having trouble keeping track of them all ;).
> 
> Unfortunately I share that same feeling X-<.
> 
> Could you guys collectively summarize what issues each of these
> in-flight topics try to address and how, how close it is to achieve
> concensus, and how it interact with other proposed topics?

I'm aware of these topics:

- My 4 "Never ignore staged but ignored submodules" patches

  I recall that everyone agreed that this is a good change.

- Johan's "Test various 'git submodule update' scenarios"

  Intended to document current behavior of the submodule.<name>.branch
  setting (and others) as a starting point for a discussion of how
  that could (and should) evolve. Needs some cooking.

- My "submodule test harness" RFC series (currently 14 patches)

  Similar to Johan's patch I try to document the current behavior
  of Git, but with the focus on all work tree manipulating commands
  (not only 'submodule update' handling all submodule changes. Will
  send to the list again when I resolved the last outstanding issues,
  current state can be seen in the submodule-test-harness branch of
  my GitHub repo. My next to-be-finished topic.

- Ronald's two "Teach add and commit the --ignore-submodules option"
  options

  Will review v4 soonish, looking good from the first cursory look.

- My "recursive submodule checkout" series

  Needs to be rerolled, I intend to extend my "submodule test harness"
  to cover all relevant scenarios for this series.

- Heiko's "config cache for submodules" patch

  Needed for my recursive checkout series to populate new submodules.

And then a not yet surfaced "do not replace submodules with a file"
fix I intend to send between the "submodule test harness" and the
"recursive submodule checkout" series.

Hope that makes it clearer ;-)

Re: [RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 2150 bytes --]

On Thu, Apr 17, 2014 at 11:08:06PM +0200, Jens Lehmann wrote:
> Am 17.04.2014 18:41, schrieb W. Trevor King:
> > On Tue, Mar 25, 2014 at 06:05:05PM +0100, Jens Lehmann wrote:
> >> *) When a submodule is replaced with a tracked file of the same name
> >>    the submodule work tree including any local modifications (and
> >>    even the whole history if it uses a .git directory instead of a
> >>    gitfile!) is simply removed.
> >> …
> >> I think the first bug really needs to be fixed, as that behavior is
> >> extremely nasty. We should always protect work tree modifications
> >> (unless forced) and *never* remove a .git directory (even when
> >> forced).
> > 
> > I think this should be covered by the usual “don't allow checkouts
> > from dirty workdirs unless the dirty-ing changes are easily applied to
> > the target tree”.
> 
> Nope, the target tree will be removed completely and everything in
> it is silently nuked. It should be allowed with '-f', but only if
> the submodule contains a gitfile, and never if it contains a .git
> directory (which is just what we do for rm too).

I think it's not covered *now* because of a flaw in our “are dirty-ing
changes easily applied to the target tree” detection logic, and the
solution should involve updating that logic to hit on this case.

> b) recursive checkout is the place to consistently care about
> submodule modifications (the submodule script doesn't do that and it
> is impossible to change that without causing trouble to a lot of
> users.

I agree that the submodule script is not the place for this, and the
core checkout code is.  I'd like checkouts to always be recursive, and
see --[no-]recurse-submodules as a finger-breaking stop-gap until we
can complete that transition for checkout, bisect, merge, reset, and
other work-tree altering commands.  I think this is one reason why
some folks prefer the stiffer joints you get from a subtree approach.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

[Jens Lehmann]

Am 17.04.2014 23:55, schrieb W. Trevor King:
> On Thu, Apr 17, 2014 at 11:08:06PM +0200, Jens Lehmann wrote:
>> Am 17.04.2014 18:41, schrieb W. Trevor King:
>>> On Tue, Mar 25, 2014 at 06:05:05PM +0100, Jens Lehmann wrote:
>>>> *) When a submodule is replaced with a tracked file of the same name
>>>>    the submodule work tree including any local modifications (and
>>>>    even the whole history if it uses a .git directory instead of a
>>>>    gitfile!) is simply removed.
>>>> …
>>>> I think the first bug really needs to be fixed, as that behavior is
>>>> extremely nasty. We should always protect work tree modifications
>>>> (unless forced) and *never* remove a .git directory (even when
>>>> forced).
>>>
>>> I think this should be covered by the usual “don't allow checkouts
>>> from dirty workdirs unless the dirty-ing changes are easily applied to
>>> the target tree”.
>>
>> Nope, the target tree will be removed completely and everything in
>> it is silently nuked. It should be allowed with '-f', but only if
>> the submodule contains a gitfile, and never if it contains a .git
>> directory (which is just what we do for rm too).
> 
> I think it's not covered *now* because of a flaw in our “are dirty-ing
> changes easily applied to the target tree” detection logic, and the
> solution should involve updating that logic to hit on this case.

Yup.

>> b) recursive checkout is the place to consistently care about
>> submodule modifications (the submodule script doesn't do that and it
>> is impossible to change that without causing trouble to a lot of
>> users.
> 
> I agree that the submodule script is not the place for this, and the
> core checkout code is.  I'd like checkouts to always be recursive, and
> see --[no-]recurse-submodules as a finger-breaking stop-gap until we
> can complete that transition for checkout, bisect, merge, reset, and
> other work-tree altering commands.  I think this is one reason why
> some folks prefer the stiffer joints you get from a subtree approach.

We are definitely in the same boat here :-)

Re: Re: [RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

[Heiko Voigt]

On Thu, Apr 17, 2014 at 11:30:04PM +0200, Jens Lehmann wrote:
> - Heiko's "config cache for submodules" patch
> 
>   Needed for my recursive checkout series to populate new submodules.

Which will be followed by my recursive fetch series, which is also the
last part of what started with this:

http://thread.gmane.org/gmane.comp.version-control.git/217018

and is described here:

https://github.com/hvoigt/git/wiki/submodule-fetch-config

Cheers Heiko