[PATCH 0/3] subtree: add 'git-subtree-repo' and list command

[PATCH 0/3] subtree: add 'git-subtree-repo' and list command

[Nicola Paolucci]

To my knowledge 'git subtree' currently lacks a way to
track where injected repositories come from originally.
Adding this information allows for useful extensions to 
the command and makes it easier to use subtrees to track
external dependencies.

In this patch series I propose to add a 'git-subtree-repo'
line to the meta-data stored when injecting a tree
in a repository with 'git subtree add'. The result looks 
like this:

    git-subtree-dir: .vim/bundle/fireplace
    git-subtree-split: b999b09cd9d69f359fa5668e81b09dcfde455cca
    git-subtree-repo: https://repo/user/vim-fireplace.git

Thanks a lot to Mathias Nyman who has cleaned up my idea to
add 'git-subtree-repo' and already submitted it for review.
I added a test and a tiny fix to his patch and I resend it 
here (hence the v3 in the first patch).

Using this extra value a simple 'git subtree list' command can 
be implemented which scans the checked out branch for subtrees
injected:

    $ git subtree list
    .vim/bundle/fireplace https://github.com/tpope/vim-fireplace.git b999b0

I also added an optional '--resolve' flag to retrieve symbolic
remote refs associated with the commit ids of the remote repository:

    $ git-subtree.sh list --resolve
    
    vim-airline  https://repo/bling/vim-airline.git 4fa37e5e[...]
    vim-airline  https://repo/bling/vim-airline.git HEAD
    vim-airline  https://repo/bling/vim-airline.git refs/heads/master


Nicola Paolucci (3):
  contrib/subtree: 'add' stores 'git-subtree-repo'
  contrib/subtree: new list command to list subtrees
  contrib/subtree: list --resolve gets symbolic refs

 contrib/subtree/git-subtree.sh     | 140 +++++++++++++++++++++++++++++--------
 contrib/subtree/git-subtree.txt    |  31 ++++++++
 contrib/subtree/t/t7900-subtree.sh |  63 +++++++++++++++++
 3 files changed, 205 insertions(+), 29 deletions(-)

-- 
2.7.1

[PATCH v3 1/3] contrib/subtree: 'add' stores 'git-subtree-repo'

[Nicola Paolucci]

Git-subtree operations 'add' and 'pull', when called with the <repository>
parameter will add this to the commit message:
    git-subtree-repo: <repo_url>

Other operations that don't have the <repository> information, like
'merge' or 'add' without <repository> are unchanged. Users with such a
workflow will be on their own with the --message parameter, if they'd
like to record where the subtree came from.

For example:

$ git subtree add --prefix .vim/bundle/fireplace \
    https://repo/user/vim-fireplace.git master --squash

Will result in a commit like:

    commit ce87dab198fecdff6043d88a26c55d7cd95e8bf9
    Author: Bob Marley <bob@mahrley.com>
    Date:   Tue May 12 13:37:03 2015 +0200

    Squashed '.vim/bundle/fireplace/' content from commit b999b09

    git-subtree-dir: .vim/bundle/fireplace
    git-subtree-split: b999b09cd9d69f359fa5668e81b09dcfde455cca
    git-subtree-repo: https://repo/user/vim-fireplace.git

This allows new ways to interact with injected trees, for example
a new command 'git subtree list' becomes possible:

$ git subtree list
.vim/bundle/fireplace https://repo/user/vim-fireplace.git b999b0

Signed-off-by: Mathias Nyman <mathias.nyman@iki.fi>
Signed-off-by: Nicola Paolucci <npaolucci@atlassian.com>
Thanks-to: Aleksi Aalto <aga@iki.fi>
---
 contrib/subtree/git-subtree.sh     | 73 +++++++++++++++++++++++++-------------
 contrib/subtree/t/t7900-subtree.sh | 19 ++++++++++
 2 files changed, 68 insertions(+), 24 deletions(-)

diff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh
index 7a39b30..278699b 100755
--- a/contrib/subtree/git-subtree.sh
+++ b/contrib/subtree/git-subtree.sh
@@ -335,18 +335,21 @@ add_msg()
 	dir="$1"
 	latest_old="$2"
 	latest_new="$3"
+	repo="$4" # optional
 	if [ -n "$message" ]; then
 		commit_message="$message"
 	else
 		commit_message="Add '$dir/' from commit '$latest_new'"
 	fi
-	cat <<-EOF
-		$commit_message
-		
-		git-subtree-dir: $dir
-		git-subtree-mainline: $latest_old
-		git-subtree-split: $latest_new
-	EOF
+	echo $commit_message
+	echo
+	echo git-subtree-dir: $dir
+	echo git-subtree-mainline: $latest_old
+	echo git-subtree-split: $latest_new
+	if [ -n "$repo" ]; then
+		repo_url=$(get_repository_url "$repo")
+		echo "git-subtree-repo: $repo_url"
+	fi
 }
 
 add_squashed_msg()
@@ -382,8 +385,9 @@ squash_msg()
 	dir="$1"
 	oldsub="$2"
 	newsub="$3"
+	repo="$4" # optional
 	newsub_short=$(git rev-parse --short "$newsub")
-	
+
 	if [ -n "$oldsub" ]; then
 		oldsub_short=$(git rev-parse --short "$oldsub")
 		echo "Squashed '$dir/' changes from $oldsub_short..$newsub_short"
@@ -397,6 +401,10 @@ squash_msg()
 	echo
 	echo "git-subtree-dir: $dir"
 	echo "git-subtree-split: $newsub"
+	if [ -n "$repo" ]; then
+		repo_url=$(get_repository_url "$repo")
+		echo "git-subtree-repo: $repo_url"
+	fi
 }
 
 toptree_for_commit()
@@ -440,12 +448,13 @@ new_squash_commit()
 	old="$1"
 	oldsub="$2"
 	newsub="$3"
+	repo="$4" # optional
 	tree=$(toptree_for_commit $newsub) || exit $?
 	if [ -n "$old" ]; then
-		squash_msg "$dir" "$oldsub" "$newsub" | 
+		squash_msg "$dir" "$oldsub" "$newsub" "$repo" |
 			git commit-tree "$tree" -p "$old" || exit $?
 	else
-		squash_msg "$dir" "" "$newsub" |
+		squash_msg "$dir" "" "$newsub" "$repo" |
 			git commit-tree "$tree" || exit $?
 	fi
 }
@@ -517,6 +526,16 @@ ensure_valid_ref_format()
 	    die "'$1' does not look like a ref"
 }
 
+get_repository_url()
+{
+	repo=$1
+	repo_url=$(git config --get remote.$repo.url)
+	if [ -z "$repo_url" ]; then
+		repo_url=$repo
+	fi
+	echo $repo_url
+}
+
 cmd_add()
 {
 	if [ -e "$dir" ]; then
@@ -548,19 +567,18 @@ cmd_add()
 cmd_add_repository()
 {
 	echo "git fetch" "$@"
-	repository=$1
+	repo=$1
 	refspec=$2
 	git fetch "$@" || exit $?
 	revs=FETCH_HEAD
-	set -- $revs
+	set -- $revs $repo
 	cmd_add_commit "$@"
 }
 
 cmd_add_commit()
 {
-	revs=$(git rev-parse $default --revs-only "$@") || exit $?
-	set -- $revs
-	rev="$1"
+	rev=$(git rev-parse $default --revs-only "$1") || exit $?
+	repo="${@:2}" # optional
 	
 	debug "Adding $dir as '$rev'..."
 	git read-tree --prefix="$dir" $rev || exit $?
@@ -575,12 +593,12 @@ cmd_add_commit()
 	fi
 	
 	if [ -n "$squash" ]; then
-		rev=$(new_squash_commit "" "" "$rev") || exit $?
+		rev=$(new_squash_commit "" "" "$rev" "$repo") || exit $?
 		commit=$(add_squashed_msg "$rev" "$dir" |
 			 git commit-tree $tree $headp -p "$rev") || exit $?
 	else
 		revp=$(peel_committish "$rev") &&
-		commit=$(add_msg "$dir" "$headrev" "$rev" |
+		commit=$(add_msg "$dir" "$headrev" "$rev" "$repo" |
 			 git commit-tree $tree $headp -p "$revp") || exit $?
 	fi
 	git reset "$commit" || exit $?
@@ -609,7 +627,8 @@ cmd_split()
 	else
 		unrevs="$(find_existing_splits "$dir" "$revs")"
 	fi
-	
+
+	rev="$1"
 	# We can't restrict rev-list to only $dir here, because some of our
 	# parents have the $dir contents the root, and those won't match.
 	# (and rev-list --follow doesn't seem to solve this)
@@ -683,15 +702,20 @@ cmd_split()
 
 cmd_merge()
 {
-	revs=$(git rev-parse $default --revs-only "$@") || exit $?
+	revs=$(git rev-parse $default --revs-only "$1") || exit $?
 	ensure_clean
-	
 	set -- $revs
 	if [ $# -ne 1 ]; then
 		die "You must provide exactly one revision.  Got: '$revs'"
 	fi
+	do_merge "$@"
+}
+
+do_merge()
+{
 	rev="$1"
-	
+	repo="$2" # optional
+
 	if [ -n "$squash" ]; then
 		first_split="$(find_latest_squash "$dir")"
 		if [ -z "$first_split" ]; then
@@ -704,7 +728,7 @@ cmd_merge()
 			say "Subtree is already at commit $rev."
 			exit 0
 		fi
-		new=$(new_squash_commit "$old" "$sub" "$rev") || exit $?
+		new=$(new_squash_commit "$old" "$sub" "$rev" "$repo") || exit $?
 		debug "New squash commit: $new"
 		rev="$new"
 	fi
@@ -730,12 +754,13 @@ cmd_pull()
 	if [ $# -ne 2 ]; then
 	    die "You must provide <repository> <ref>"
 	fi
+	repo=$1
 	ensure_clean
 	ensure_valid_ref_format "$2"
 	git fetch "$@" || exit $?
 	revs=FETCH_HEAD
-	set -- $revs
-	cmd_merge "$@"
+	set -- $revs $repo
+	do_merge "$@"
 }
 
 cmd_push()
diff --git a/contrib/subtree/t/t7900-subtree.sh b/contrib/subtree/t/t7900-subtree.sh
index 3bf96a9..ed40e73 100755
--- a/contrib/subtree/t/t7900-subtree.sh
+++ b/contrib/subtree/t/t7900-subtree.sh
@@ -86,6 +86,11 @@ last_commit_message()
 	git log --pretty=format:%s -1
 }
 
+squashed_commit_body()
+{
+	git log --pretty=format:%b --numstat HEAD^2
+}
+
 subtree_test_count=0
 next_test() {
 	subtree_test_count=$(($subtree_test_count+1))
@@ -176,6 +181,20 @@ test_expect_success 'add subproj as subtree into sub dir/ with --squash and --pr
 	)
 '
 
+next_test
+test_expect_success 'add --squash stores git-subtree-repo value' '
+	subtree_test_create_repo "$subtree_test_count" &&
+	subtree_test_create_repo "$subtree_test_count/sub proj" &&
+	test_create_commit "$subtree_test_count" main1 &&
+	test_create_commit "$subtree_test_count/sub proj" sub1 &&
+	(
+		cd "$subtree_test_count" &&
+		git fetch ./"sub proj" master &&
+		git subtree add --prefix="sub dir" "./sub proj" HEAD --squash &&
+		check_equal "$(squashed_commit_body | grep git-subtree-repo)" "git-subtree-repo: ./sub proj"
+	)
+'
+
 #
 # Tests for 'git subtree merge'
 #
-- 
2.7.1

[PATCH 2/3] contrib/subtree: new list command to list subtrees

[Nicola Paolucci]

Example output:

$ git subtree list
.vim/bundle/fireplace https://github.com/tpope/vim-fireplace.git b999b0

Signed-off-by: Nicola Paolucci <npaolucci@atlassian.com>
---
 contrib/subtree/git-subtree.sh     | 54 ++++++++++++++++++++++++++++++++++----
 contrib/subtree/t/t7900-subtree.sh | 18 +++++++++++++
 2 files changed, 67 insertions(+), 5 deletions(-)

diff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh
index 278699b..82f3fce 100755
--- a/contrib/subtree/git-subtree.sh
+++ b/contrib/subtree/git-subtree.sh
@@ -14,6 +14,7 @@ git subtree merge --prefix=<prefix> <commit>
 git subtree pull  --prefix=<prefix> <repository> <ref>
 git subtree push  --prefix=<prefix> <repository> <ref>
 git subtree split --prefix=<prefix> <commit...>
+git subtree list
 --
 h,help        show the help
 q             quiet
@@ -109,19 +110,22 @@ done
 command="$1"
 shift
 case "$command" in
-	add|merge|pull) default= ;;
+	add|merge|pull|list) default= ;;
 	split|push) default="--default HEAD" ;;
 	*) die "Unknown command '$command'" ;;
 esac
 
-if [ -z "$prefix" ]; then
-	die "You must provide the --prefix option."
+if [ "$command" != "list" ]; then
+	if [ -z "$prefix" ]; then
+		die "You must provide the --prefix option."
+	fi
 fi
 
 case "$command" in
-	add) [ -e "$prefix" ] && 
+	add) [ -e "$prefix" ] &&
 		die "prefix '$prefix' already exists." ;;
-	*)   [ -e "$prefix" ] || 
+	list) ;;
+	*)   [ -e "$prefix" ] ||
 		die "'$prefix' does not exist; use 'git subtree add'" ;;
 esac
 
@@ -230,6 +234,41 @@ try_remove_previous()
 	fi
 }
 
+find_subtree_repos()
+{
+	debug "Looking for subtree repos..."
+	sq=
+	main=
+	sub=
+	git log --grep="^git-subtree-dir:" \
+		--pretty=format:'START %H%n%s%n%n%b%nEND%n' HEAD |
+	while read a b c; do
+		debug "$a $b $c"
+		debug "{{$sq/$main/$sub}}"
+		case "$a" in
+			START) sq="$b" ;;
+			git-subtree-dir:) dir="$b $c" ;;
+			git-subtree-mainline:) main="$b" ;;
+			git-subtree-split:) sub="$b" ;;
+			git-subtree-repo:) repo="$b $c" ;;
+			END)
+				if [ -n "$sub" ]; then
+					if [ -n "$main" ]; then
+						# a rejoin commit?
+						# Pretend its sub was a squash.
+						sq="$sub"
+					fi
+					debug "Subtree found: $dir $repo $sub"
+					echo "$dir" "$repo" "$sub"
+				fi
+				sq=
+				main=
+				sub=
+				;;
+		esac
+	done
+}
+
 find_latest_squash()
 {
 	debug "Looking for latest squash ($dir)..."
@@ -536,6 +575,11 @@ get_repository_url()
 	echo $repo_url
 }
 
+cmd_list()
+{
+	find_subtree_repos "$@"
+}
+
 cmd_add()
 {
 	if [ -e "$dir" ]; then
diff --git a/contrib/subtree/t/t7900-subtree.sh b/contrib/subtree/t/t7900-subtree.sh
index ed40e73..ce97446 100755
--- a/contrib/subtree/t/t7900-subtree.sh
+++ b/contrib/subtree/t/t7900-subtree.sh
@@ -196,6 +196,24 @@ test_expect_success 'add --squash stores git-subtree-repo value' '
 '
 
 #
+# Tests for 'git subtree list'
+#
+
+next_test
+test_expect_success 'list outputs list of subtrees' '
+	subtree_test_create_repo "$subtree_test_count" &&
+	subtree_test_create_repo "$subtree_test_count/sub proj" &&
+	test_create_commit "$subtree_test_count" main1 &&
+	test_create_commit "$subtree_test_count/sub proj" sub1 &&
+	(
+		cd "$subtree_test_count" &&
+		git fetch ./"sub proj" master &&
+		git subtree add --prefix="sub dir" "./sub proj" HEAD --squash &&
+		check_equal "$(git subtree list | cut -c -19)" "sub dir ./sub proj "
+	)
+'
+
+#
 # Tests for 'git subtree merge'
 #
 
-- 
2.7.1

[PATCH 3/3] contrib/subtree: list --resolve gets symbolic refs

[Nicola Paolucci]

As the 'list' command finds commit ids for subtrees injected into the
checked out branch the --resolve flag tries to look up the repositories
at 'git-subtree-repo' and retrive the symbolic refs associated with the
commit ids found.

Example:

    $ git-subtree.sh list --resolve

    vim-airline  https://repo/bling/vim-airline.git 4fa37e5e[...]
    vim-airline  https://repo/bling/vim-airline.git HEAD
    vim-airline  https://repo/bling/vim-airline.git refs/heads/master

Signed-off-by: Nicola Paolucci <npaolucci@atlassian.com>
---
 contrib/subtree/git-subtree.sh     | 21 +++++++++++++++++----
 contrib/subtree/git-subtree.txt    | 31 +++++++++++++++++++++++++++++++
 contrib/subtree/t/t7900-subtree.sh | 26 ++++++++++++++++++++++++++
 3 files changed, 74 insertions(+), 4 deletions(-)

diff --git a/contrib/subtree/git-subtree.sh b/contrib/subtree/git-subtree.sh
index 82f3fce..fe62151 100755
--- a/contrib/subtree/git-subtree.sh
+++ b/contrib/subtree/git-subtree.sh
@@ -14,7 +14,7 @@ git subtree merge --prefix=<prefix> <commit>
 git subtree pull  --prefix=<prefix> <repository> <ref>
 git subtree push  --prefix=<prefix> <repository> <ref>
 git subtree split --prefix=<prefix> <commit...>
-git subtree list
+git subtree list  [--resolve]
 --
 h,help        show the help
 q             quiet
@@ -29,6 +29,7 @@ onto=         try connecting new tree to an existing one
 rejoin        merge the new branch back into HEAD
  options for 'add', 'merge', and 'pull'
 squash        merge subtree changes as a single commit
+resolve       resolves ids to refs when possible
 "
 eval "$(echo "$OPTS_SPEC" | git rev-parse --parseopt -- "$@" || echo exit $?)"
 
@@ -48,6 +49,7 @@ annotate=
 squash=
 message=
 prefix=
+resolve=
 
 debug()
 {
@@ -102,6 +104,8 @@ while [ $# -gt 0 ]; do
 		--no-ignore-joins) ignore_joins= ;;
 		--squash) squash=1 ;;
 		--no-squash) squash= ;;
+		--resolve) resolve=1 ;;
+		--no-resolve) resolve= ;;
 		--) break ;;
 		*) die "Unexpected option: $opt" ;;
 	esac
@@ -254,12 +258,21 @@ find_subtree_repos()
 			END)
 				if [ -n "$sub" ]; then
 					if [ -n "$main" ]; then
-						# a rejoin commit?
-						# Pretend its sub was a squash.
 						sq="$sub"
 					fi
 					debug "Subtree found: $dir $repo $sub"
-					echo "$dir" "$repo" "$sub"
+					# Strip potential space at the end in repo
+					repo=$(echo $repo)
+					if [ -n "$resolve" ] && [ -n "$repo" ]; then
+						echo "$dir" "$repo" "$sub"
+						# Retrieve remote refs if repo is available
+						resolved_refs=$(git ls-remote "$repo" | grep "$sub" | cut -c 42- | xargs)
+						for r in $resolved_refs; do
+							echo "$dir" "$repo" "$r"
+						done
+					else
+						echo "$dir" "$repo" "$sub"
+					fi
 				fi
 				sq=
 				main=
diff --git a/contrib/subtree/git-subtree.txt b/contrib/subtree/git-subtree.txt
index 60d76cd..ab36951 100644
--- a/contrib/subtree/git-subtree.txt
+++ b/contrib/subtree/git-subtree.txt
@@ -15,6 +15,7 @@ SYNOPSIS
 'git subtree' push  -P <prefix> <repository> <ref>
 'git subtree' merge -P <prefix> <commit>
 'git subtree' split -P <prefix> [OPTIONS] [<commit>]
+'git subtree' list [--resolve]
 
 
 DESCRIPTION
@@ -106,6 +107,12 @@ split::
 	contents of <prefix> at the root of the project instead
 	of in a subdirectory.  Thus, the newly created history
 	is suitable for export as a separate git repository.
+
+list::
+	List all subtrees injected in checked out branch. Run
+	with optional `--resolve` to retrieve remote symbolic refs
+	associated with the subtree. Address of subtree repo is stored
+	in commits as `git-subtree-repo` at the time of `git subtree add`.
 +
 After splitting successfully, a single commit id is printed to stdout.
 This corresponds to the HEAD of the newly created tree, which you can
@@ -240,6 +247,13 @@ split, because you don't want the subproject's history to be part of
 your project anyway.
 
 
+OPTIONS FOR list
+----------------------------------
+--resolve::
+	Resolves 'git-subtree-split' refs by looking up symbolic refs at
+	'git-subtree-repo'.
+
+
 EXAMPLE 1. Add command
 ----------------------
 Let's assume that you have a local repository that you would like
@@ -341,6 +355,23 @@ Then push the new branch onto the new empty repository:
 	$ git push <new-repo> split:master
 
 
+EXAMPLE 3. List subtrees
+-----------------------------------------
+Suppose you add a subtree with:
+
+	$ git subtree add --prefix dependency https://host/repo.git master --squash
+
+You can list all subtrees in the current branch resolving the refs with:
+
+	$ git subtree list --resolve
+
+Which would output something like:
+
+	depenency  https://host/repo.git 4fa37e5e20b5ae9b[...]
+	depenency  https://host/repo.git HEAD
+	depenency  https://host/repo.git refs/heads/master
+
+
 AUTHOR
 ------
 Written by Avery Pennarun <apenwarr@gmail.com>
diff --git a/contrib/subtree/t/t7900-subtree.sh b/contrib/subtree/t/t7900-subtree.sh
index ce97446..d100001 100755
--- a/contrib/subtree/t/t7900-subtree.sh
+++ b/contrib/subtree/t/t7900-subtree.sh
@@ -81,6 +81,19 @@ test_create_commit() (
 	git commit -m "$commit" || error "Could not commit"
 )
 
+test_create_commit_and_tag() (
+	repo=$1
+	commit=$2
+    tag=$3
+	cd "$repo"
+	mkdir -p $(dirname "$commit") \
+	|| error "Could not create directory for commit"
+	echo "$commit" >"$commit"
+	git add "$commit" || error "Could not add commit"
+	git commit -m "$commit" || error "Could not commit"
+	git tag -m "$commit tag: $tag" $tag || error "Could not tag"
+)
+
 last_commit_message()
 {
 	git log --pretty=format:%s -1
@@ -212,6 +225,19 @@ test_expect_success 'list outputs list of subtrees' '
 		check_equal "$(git subtree list | cut -c -19)" "sub dir ./sub proj "
 	)
 '
+next_test
+test_expect_success 'list --resolve resolves refs' '
+	subtree_test_create_repo "$subtree_test_count" &&
+	subtree_test_create_repo "$subtree_test_count/sub proj" &&
+	test_create_commit_and_tag "$subtree_test_count/sub proj" sub1 test-v0.1 &&
+	test_create_commit "$subtree_test_count" main1 &&
+	(
+		cd "$subtree_test_count" &&
+		git fetch ./"sub proj" master &&
+		git subtree add --prefix="sub dir" "./sub proj" HEAD --squash &&
+		check_equal "$(git subtree -d list --resolve | grep HEAD)" "sub dir ./sub proj HEAD"
+	)
+'
 
 #
 # Tests for 'git subtree merge'
-- 
2.7.1

Re: [PATCH 0/3] subtree: add 'git-subtree-repo' and list command

[David A. Greene]

Nicola Paolucci <npaolucci@atlassian.com> writes:

> To my knowledge 'git subtree' currently lacks a way to
> track where injected repositories come from originally.
> Adding this information allows for useful extensions to 
> the command and makes it easier to use subtrees to track
> external dependencies.

Thanks for working on this.  I just sent a reply to your earlier
iteration.

What is the intent for use of this?  Is it simply to record from where
commits were pulled or do you intend to use this information later on to
have git-subtree guess from where to fetch future commits?

I would be opposed to the latter because I think it potentially limits
the utility of git-subtree and may be misleading.  I frequently pull
commits for a subtree from several different clones of the same
reposiory.  How does git-subtree list handle that situation?

Does git-subtree list really print out repository information for every
commit added by git-subtree?  That's potentially a lot of commits.  It
might be more useful to aggregate repository information and only dump
out unique URLs.  In any case, processing all commits seems like a ton
of work for such a simple operation.  Maybe this information should be
cached in .gitconfig.

I'm actually in the middle of cleaing up metadata but I'm not going to
block these commits due to that.  Just be aware that it may change a
bit.

                  -David