[PATCH/RFC RESEND 0/2] git subtree: an alternative to git submodule

[PATCH/RFC RESEND 0/2] git subtree: an alternative to git submodule

[Avery Pennarun]

Hi all,

I first sent out this patch set a couple of weeks ago
(http://article.gmane.org/gmane.comp.version-control.git/117612) and got a
couple of positive comments, but no negative ones, so I'm guessing people
haven't reviewed it as closely as I would have hoped :)

git subtree has these subcommands:

 - add: connect a given commit to a given subtree, basically using the
   occasionally-documented 'git read-tree --prefix; git commit' trick.

 - merge: a user-friendlier form of 'git merge -s subtree'

 - pull: likewise, but for git pull

 - split: (the magical part!) generate a new commit series from the given
   prefix, so you can submit subtree changes *back* to the upstream project
   you merged in the first place. 

Does anyone have any comments on what it would take to get the git subtree
stuff accepted into git?

Thanks,

Avery


Avery Pennarun (2):
  Add 'git subtree' command for tracking history of subtrees
    separately.
  Automated test script for 'git subtree'.

 .gitignore       |    1 +
 Makefile         |    1 +
 command-list.txt |    1 +
 git-subtree.sh   |  435 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 subtree-test.sh  |  206 ++++++++++++++++++++++++++
 5 files changed, 644 insertions(+), 0 deletions(-)
 create mode 100755 git-subtree.sh
 create mode 100755 subtree-test.sh

[PATCH/RFC RESEND 1/2] Add 'git subtree' command for tracking history of subtrees separately.

[Avery Pennarun]

Many projects are made of a combination of several subprojects/libraries and
some application-specific code.  In some cases, particularly when the
subprojects are all maintained independently, 'git submodule' is the best
way to deal with this situation.  But if you frequently change the
subprojects as part of developing your application, use multiple branches,
and sometimes want to push your subproject changes upstream, the overhead of
manually managing submodules can be excessive.

'git subtree' provides an alternative mechanism, based around the
'git merge -s subtree' merge strategy.  Instead of tracking a submodule
separately, you merge its history into your main project, and occasionally
extract a new "virtual history" from your mainline that can be easily merged
back into the upstream project.  The virtual history can be incrementally
expanded as you make more changes to the superproject.

You would normally then merge the virtual history back into your mainline
(the --rejoin option). This results in extra commits in your application
that appear to change the same files, but these extra commits will tend to
be ignored by git's merge simplification algorithm anyway.

For example, gitweb (commit 1130ef3) was merged into git as of commit
0a8f4f0, after which it was no longer maintained separately.  But imagine it
had been maintained separately, and we wanted to extract git's changes to
gitweb since that time, to share with the upstream.  You could do this:

git subtree split --prefix=gitweb --annotate='(split) ' \
	0a8f4f0^.. --onto=1130ef3 --rejoin

If gitweb had originally been merged using 'git subtree add' (or a previous
split had been done with --rejoin specified), then you could incrementally
produce the list of new changes without needing to remember any commit ids:

git subtree split --prefix=gitweb --annotate='(split) ' --rejoin
---
 .gitignore       |    1 +
 Makefile         |    1 +
 command-list.txt |    1 +
 git-subtree.sh   |  435 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 438 insertions(+), 0 deletions(-)
 create mode 100755 git-subtree.sh

diff --git a/.gitignore b/.gitignore
index 41c0b20..6f0bc34 100644
--- a/.gitignore
+++ b/.gitignore
@@ -127,6 +127,7 @@ git-stash
 git-status
 git-stripspace
 git-submodule
+git-subtree
 git-svn
 git-symbolic-ref
 git-tag
diff --git a/Makefile b/Makefile
index 6e21643..02d72a5 100644
--- a/Makefile
+++ b/Makefile
@@ -305,6 +305,7 @@ SCRIPT_SH += git-request-pull.sh
 SCRIPT_SH += git-sh-setup.sh
 SCRIPT_SH += git-stash.sh
 SCRIPT_SH += git-submodule.sh
+SCRIPT_SH += git-subtree.sh
 SCRIPT_SH += git-web--browse.sh
 
 SCRIPT_PERL += git-add--interactive.perl
diff --git a/command-list.txt b/command-list.txt
index fb03a2e..9be4774 100644
--- a/command-list.txt
+++ b/command-list.txt
@@ -113,6 +113,7 @@ git-stash                               mainporcelain
 git-status                              mainporcelain common
 git-stripspace                          purehelpers
 git-submodule                           mainporcelain
+git-subtree                             mainporcelain
 git-svn                                 foreignscminterface
 git-symbolic-ref                        plumbingmanipulators
 git-tag                                 mainporcelain common
diff --git a/git-subtree.sh b/git-subtree.sh
new file mode 100755
index 0000000..39c377c
--- /dev/null
+++ b/git-subtree.sh
@@ -0,0 +1,435 @@
+#!/bin/bash
+#
+# git-subtree.sh: split/join git repositories in subdirectories of this one
+#
+# Copyright (C) 2009 Avery Pennarun <apenwarr@gmail.com>
+#
+if [ $# -eq 0 ]; then
+    set -- -h
+fi
+OPTS_SPEC="\
+git subtree add --prefix=<prefix> <commit>
+git subtree split [options...] --prefix=<prefix> <commit...>
+git subtree merge --prefix=<prefix> <commit>
+git subtree pull  --prefix=<prefix> <repository> <refspec...>
+--
+h,help        show the help
+q             quiet
+d             show debug messages
+prefix=       the name of the subdir to split out
+ options for 'split'
+annotate=     add a prefix to commit message of new commits
+onto=         try connecting new tree to an existing one
+rejoin        merge the new branch back into HEAD
+ignore-joins  ignore prior --rejoin commits
+"
+eval $(echo "$OPTS_SPEC" | git rev-parse --parseopt -- "$@" || echo exit $?)
+. git-sh-setup
+require_work_tree
+
+quiet=
+debug=
+command=
+onto=
+rejoin=
+ignore_joins=
+annotate=
+
+debug()
+{
+	if [ -n "$debug" ]; then
+		echo "$@" >&2
+	fi
+}
+
+say()
+{
+	if [ -z "$quiet" ]; then
+		echo "$@" >&2
+	fi
+}
+
+assert()
+{
+	if "$@"; then
+		:
+	else
+		die "assertion failed: " "$@"
+	fi
+}
+
+
+#echo "Options: $*"
+
+while [ $# -gt 0 ]; do
+	opt="$1"
+	shift
+	case "$opt" in
+		-q) quiet=1 ;;
+		-d) debug=1 ;;
+		--annotate) annotate="$1"; shift ;;
+		--no-annotate) annotate= ;;
+		--prefix) prefix="$1"; shift ;;
+		--no-prefix) prefix= ;;
+		--onto) onto="$1"; shift ;;
+		--no-onto) onto= ;;
+		--rejoin) rejoin=1 ;;
+		--no-rejoin) rejoin= ;;
+		--ignore-joins) ignore_joins=1 ;;
+		--no-ignore-joins) ignore_joins= ;;
+		--) break ;;
+	esac
+done
+
+command="$1"
+shift
+case "$command" in
+	add|merge|pull) default= ;;
+	split) default="--default HEAD" ;;
+	*) die "Unknown command '$command'" ;;
+esac
+
+if [ -z "$prefix" ]; then
+	die "You must provide the --prefix option."
+fi
+dir="$prefix"
+
+if [ "$command" != "pull" ]; then
+	revs=$(git rev-parse $default --revs-only "$@") || exit $?
+	dirs="$(git rev-parse --no-revs --no-flags "$@")" || exit $?
+	if [ -n "$dirs" ]; then
+		die "Error: Use --prefix instead of bare filenames."
+	fi
+fi
+
+debug "command: {$command}"
+debug "quiet: {$quiet}"
+debug "revs: {$revs}"
+debug "dir: {$dir}"
+debug "opts: {$*}"
+debug
+
+cache_setup()
+{
+	cachedir="$GIT_DIR/subtree-cache/$$"
+	rm -rf "$cachedir" || die "Can't delete old cachedir: $cachedir"
+	mkdir -p "$cachedir" || die "Can't create new cachedir: $cachedir"
+	debug "Using cachedir: $cachedir" >&2
+}
+
+cache_get()
+{
+	for oldrev in $*; do
+		if [ -r "$cachedir/$oldrev" ]; then
+			read newrev <"$cachedir/$oldrev"
+			echo $newrev
+		fi
+	done
+}
+
+cache_set()
+{
+	oldrev="$1"
+	newrev="$2"
+	if [ "$oldrev" != "latest_old" \
+	     -a "$oldrev" != "latest_new" \
+	     -a -e "$cachedir/$oldrev" ]; then
+		die "cache for $oldrev already exists!"
+	fi
+	echo "$newrev" >"$cachedir/$oldrev"
+}
+
+# if a commit doesn't have a parent, this might not work.  But we only want
+# to remove the parent from the rev-list, and since it doesn't exist, it won't
+# be there anyway, so do nothing in that case.
+try_remove_previous()
+{
+	if git rev-parse "$1^" >/dev/null 2>&1; then
+		echo "^$1^"
+	fi
+}
+
+find_existing_splits()
+{
+	debug "Looking for prior splits..."
+	dir="$1"
+	revs="$2"
+	git log --grep="^git-subtree-dir: $dir\$" \
+		--pretty=format:'%s%n%n%b%nEND' $revs |
+	while read a b junk; do
+		case "$a" in
+			git-subtree-mainline:) main="$b" ;;
+			git-subtree-split:) sub="$b" ;;
+			*)
+				if [ -n "$main" -a -n "$sub" ]; then
+					debug "  Prior: $main -> $sub"
+					cache_set $main $sub
+					try_remove_previous "$main"
+					try_remove_previous "$sub"
+					main=
+					sub=
+				fi
+				;;
+		esac
+	done
+}
+
+copy_commit()
+{
+	# We're doing to set some environment vars here, so
+	# do it in a subshell to get rid of them safely later
+	debug copy_commit "{$1}" "{$2}" "{$3}"
+	git log -1 --pretty=format:'%an%n%ae%n%ad%n%cn%n%ce%n%cd%n%s%n%n%b' "$1" |
+	(
+		read GIT_AUTHOR_NAME
+		read GIT_AUTHOR_EMAIL
+		read GIT_AUTHOR_DATE
+		read GIT_COMMITTER_NAME
+		read GIT_COMMITTER_EMAIL
+		read GIT_COMMITTER_DATE
+		export  GIT_AUTHOR_NAME \
+			GIT_AUTHOR_EMAIL \
+			GIT_AUTHOR_DATE \
+			GIT_COMMITTER_NAME \
+			GIT_COMMITTER_EMAIL \
+			GIT_COMMITTER_DATE
+		(echo -n "$annotate"; cat ) |
+		git commit-tree "$2" $3  # reads the rest of stdin
+	) || die "Can't copy commit $1"
+}
+
+add_msg()
+{
+	dir="$1"
+	latest_old="$2"
+	latest_new="$3"
+	cat <<-EOF
+		Add '$dir/' from commit '$latest_new'
+		
+		git-subtree-dir: $dir
+		git-subtree-mainline: $latest_old
+		git-subtree-split: $latest_new
+	EOF
+}
+
+merge_msg()
+{
+	dir="$1"
+	latest_old="$2"
+	latest_new="$3"
+	cat <<-EOF
+		Split '$dir/' into commit '$latest_new'
+		
+		git-subtree-dir: $dir
+		git-subtree-mainline: $latest_old
+		git-subtree-split: $latest_new
+	EOF
+}
+
+toptree_for_commit()
+{
+	commit="$1"
+	git log -1 --pretty=format:'%T' "$commit" -- || exit $?
+}
+
+subtree_for_commit()
+{
+	commit="$1"
+	dir="$2"
+	git ls-tree "$commit" -- "$dir" |
+	while read mode type tree name; do
+		assert [ "$name" = "$dir" ]
+		echo $tree
+		break
+	done
+}
+
+tree_changed()
+{
+	tree=$1
+	shift
+	if [ $# -ne 1 ]; then
+		return 0   # weird parents, consider it changed
+	else
+		ptree=$(toptree_for_commit $1)
+		if [ "$ptree" != "$tree" ]; then
+			return 0   # changed
+		else
+			return 1   # not changed
+		fi
+	fi
+}
+
+copy_or_skip()
+{
+	rev="$1"
+	tree="$2"
+	newparents="$3"
+	assert [ -n "$tree" ]
+
+	identical=
+	nonidentical=
+	p=
+	gotparents=
+	for parent in $newparents; do
+		ptree=$(toptree_for_commit $parent) || exit $?
+		[ -z "$ptree" ] && continue
+		if [ "$ptree" = "$tree" ]; then
+			# an identical parent could be used in place of this rev.
+			identical="$parent"
+		else
+			nonidentical="$parent"
+		fi
+		
+		# sometimes both old parents map to the same newparent;
+		# eliminate duplicates
+		is_new=1
+		for gp in $gotparents; do
+			if [ "$gp" = "$parent" ]; then
+				is_new=
+				break
+			fi
+		done
+		if [ -n "$is_new" ]; then
+			gotparents="$gotparents $parent"
+			p="$p -p $parent"
+		fi
+	done
+	
+	if [ -n "$identical" ]; then
+		echo $identical
+	else
+		copy_commit $rev $tree "$p" || exit $?
+	fi
+}
+
+ensure_clean()
+{
+	if ! git diff-index HEAD --exit-code --quiet; then
+		die "Working tree has modifications.  Cannot add."
+	fi
+	if ! git diff-index --cached HEAD --exit-code --quiet; then
+		die "Index has modifications.  Cannot add."
+	fi
+}
+
+cmd_add()
+{
+	if [ -e "$dir" ]; then
+		die "'$dir' already exists.  Cannot add."
+	fi
+	ensure_clean
+	
+	set -- $revs
+	if [ $# -ne 1 ]; then
+		die "You must provide exactly one revision.  Got: '$revs'"
+	fi
+	rev="$1"
+	
+	debug "Adding $dir as '$rev'..."
+	git read-tree --prefix="$dir" $rev || exit $?
+	git checkout "$dir" || exit $?
+	tree=$(git write-tree) || exit $?
+	
+	headrev=$(git rev-parse HEAD) || exit $?
+	if [ -n "$headrev" -a "$headrev" != "$rev" ]; then
+		headp="-p $headrev"
+	else
+		headp=
+	fi
+	commit=$(add_msg "$dir" "$headrev" "$rev" |
+		 git commit-tree $tree $headp -p "$rev") || exit $?
+	git reset "$commit" || exit $?
+}
+
+cmd_split()
+{
+	debug "Splitting $dir..."
+	cache_setup || exit $?
+	
+	if [ -n "$onto" ]; then
+		debug "Reading history for --onto=$onto..."
+		git rev-list $onto |
+		while read rev; do
+			# the 'onto' history is already just the subdir, so
+			# any parent we find there can be used verbatim
+			debug "  cache: $rev"
+			cache_set $rev $rev
+		done
+	fi
+	
+	if [ -n "$ignore_joins" ]; then
+		unrevs=
+	else
+		unrevs="$(find_existing_splits "$dir" "$revs")"
+	fi
+	
+	# We can't restrict rev-list to only $dir here, because some of our
+	# parents have the $dir contents the root, and those won't match.
+	# (and rev-list --follow doesn't seem to solve this)
+	grl='git rev-list --reverse --parents $revs $unrevs'
+	revmax=$(eval "$grl" | wc -l)
+	revcount=0
+	createcount=0
+	eval "$grl" |
+	while read rev parents; do
+		revcount=$(($revcount + 1))
+		say -n "$revcount/$revmax ($createcount)
"
+		debug "Processing commit: $rev"
+		exists=$(cache_get $rev)
+		if [ -n "$exists" ]; then
+			debug "  prior: $exists"
+			continue
+		fi
+		createcount=$(($createcount + 1))
+		debug "  parents: $parents"
+		newparents=$(cache_get $parents)
+		debug "  newparents: $newparents"
+		
+		tree=$(subtree_for_commit $rev "$dir")
+		debug "  tree is: $tree"
+		[ -z $tree ] && continue
+
+		newrev=$(copy_or_skip "$rev" "$tree" "$newparents") || exit $?
+		debug "  newrev is: $newrev"
+		cache_set $rev $newrev
+		cache_set latest_new $newrev
+		cache_set latest_old $rev
+	done || exit $?
+	latest_new=$(cache_get latest_new)
+	if [ -z "$latest_new" ]; then
+		die "No new revisions were found"
+	fi
+	
+	if [ -n "$rejoin" ]; then
+		debug "Merging split branch into HEAD..."
+		latest_old=$(cache_get latest_old)
+		git merge -s ours \
+			-m "$(merge_msg $dir $latest_old $latest_new)" \
+			$latest_new >&2
+	fi
+	echo $latest_new
+	exit 0
+}
+
+cmd_merge()
+{
+	ensure_clean
+	
+	set -- $revs
+	if [ $# -ne 1 ]; then
+		die "You must provide exactly one revision.  Got: '$revs'"
+	fi
+	rev="$1"
+	
+	git merge -s subtree $rev
+}
+
+cmd_pull()
+{
+	ensure_clean
+	set -x
+	git pull -s subtree "$@"
+}
+
+"cmd_$command" "$@"
-- 
1.6.3.2.g3d624

[PATCH/RFC RESEND 2/2] Automated test script for 'git subtree'.

[Avery Pennarun]

TEMPORARY: this script hasn't yet been integrated into the main git unit
tests; it runs standalone for the moment.
---
 subtree-test.sh |  206 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 206 insertions(+), 0 deletions(-)
 create mode 100755 subtree-test.sh

diff --git a/subtree-test.sh b/subtree-test.sh
new file mode 100755
index 0000000..38dff7a
--- /dev/null
+++ b/subtree-test.sh
@@ -0,0 +1,206 @@
+#!/bin/bash
+. shellopts.sh
+set -e
+
+create()
+{
+	echo "$1" >"$1"
+	git add "$1"
+}
+
+check()
+{
+	echo
+	echo "check:" "$@"
+	if "$@"; then
+		echo ok
+		return 0
+	else
+		echo FAILED
+		exit 1
+	fi
+}
+
+check_equal()
+{
+	echo
+	echo "check a:" "{$1}"
+	echo "      b:" "{$2}"
+	if [ "$1" = "$2" ]; then
+		return 0
+	else
+		echo FAILED
+		exit 1
+	fi
+}
+
+fixnl()
+{	
+	t=""
+	while read x; do
+		t="$t$x "
+	done
+	echo $t
+}
+
+multiline()
+{
+	while read x; do
+		set -- $x
+		for d in "$@"; do
+			echo "$d"
+		done
+	done
+}
+
+rm -rf mainline subproj
+mkdir mainline subproj
+
+cd subproj
+git init
+
+create sub1
+git commit -m 'sub1'
+git branch sub1
+git branch -m master subproj
+check true
+
+create sub2
+git commit -m 'sub2'
+git branch sub2
+
+create sub3
+git commit -m 'sub3'
+git branch sub3
+
+cd ../mainline
+git init
+create main4
+git commit -m 'main4'
+git branch -m master mainline
+
+git fetch ../subproj sub1
+git branch sub1 FETCH_HEAD
+git subtree add --prefix=subdir FETCH_HEAD
+
+# this shouldn't actually do anything, since FETCH_HEAD is already a parent
+git merge -m 'merge -s -ours' -s ours FETCH_HEAD
+
+create subdir/main-sub5
+git commit -m 'main-sub5'
+
+create main6
+git commit -m 'main6 boring'
+
+create subdir/main-sub7
+git commit -m 'main-sub7'
+
+git fetch ../subproj sub2
+git branch sub2 FETCH_HEAD
+git subtree merge --prefix=subdir FETCH_HEAD
+git branch pre-split
+
+spl1=$(git subtree split --annotate='*' \
+		--prefix subdir --onto FETCH_HEAD --rejoin)
+echo "spl1={$spl1}"
+git branch spl1 "$spl1"
+
+create subdir/main-sub8
+git commit -m 'main-sub8'
+
+cd ../subproj
+git fetch ../mainline spl1
+git branch spl1 FETCH_HEAD
+git merge FETCH_HEAD
+
+create sub9
+git commit -m 'sub9'
+
+cd ../mainline
+split2=$(git subtree split --annotate='*' --prefix subdir --rejoin)
+git branch split2 "$split2"
+
+create subdir/main-sub10
+git commit -m 'main-sub10'
+
+spl3=$(git subtree split --annotate='*' --prefix subdir --rejoin)
+git branch spl3 "$spl3"
+
+cd ../subproj
+git fetch ../mainline spl3
+git branch spl3 FETCH_HEAD
+git merge FETCH_HEAD
+git branch subproj-merge-spl3
+
+chkm="main4 main6"
+chkms="main-sub10 main-sub5 main-sub7 main-sub8"
+chkms_sub=$(echo $chkms | multiline | sed 's,^,subdir/,' | fixnl)
+chks="sub1 sub2 sub3 sub9"
+chks_sub=$(echo $chks | multiline | sed 's,^,subdir/,' | fixnl)
+
+# make sure exactly the right set of files ends up in the subproj
+subfiles=$(git ls-files | fixnl)
+check_equal "$subfiles" "$chkms $chks"
+
+# make sure the subproj history *only* contains commits that affect the subdir.
+allchanges=$(git log --name-only --pretty=format:'' | sort | fixnl)
+check_equal "$allchanges" "$chkms $chks"
+
+cd ../mainline
+git fetch ../subproj subproj-merge-spl3
+git branch subproj-merge-spl3 FETCH_HEAD
+git subtree pull --prefix=subdir ../subproj subproj-merge-spl3
+
+# make sure exactly the right set of files ends up in the mainline
+mainfiles=$(git ls-files | fixnl)
+check_equal "$mainfiles" "$chkm $chkms_sub $chks_sub"
+
+# make sure each filename changed exactly once in the entire history.
+# 'main-sub??' and '/subdir/main-sub??' both change, because those are the
+# changes that were split into their own history.  And 'subdir/sub??' never
+# change, since they were *only* changed in the subtree branch.
+allchanges=$(git log --name-only --pretty=format:'' | sort | fixnl)
+check_equal "$allchanges" "$chkm $chkms $chks $chkms_sub"
+
+# make sure the --rejoin commits never make it into subproj
+check_equal "$(git log --pretty=format:'%s' HEAD^2 | grep -i split)" ""
+
+# make sure no 'git subtree' tagged commits make it into subproj. (They're
+# meaningless to subproj since one side of the merge refers to the mainline)
+check_equal "$(git log --pretty=format:'%s%n%b' HEAD^2 | grep 'git-subtree.*:')" ""
+
+# make sure no patch changes more than one file.  The original set of commits
+# changed only one file each.  A multi-file change would imply that we pruned
+# commits too aggressively.
+joincommits()
+{
+	commit=
+	all=
+	while read x y; do
+		echo "{$x}" >&2
+		if [ -z "$x" ]; then
+			continue
+		elif [ "$x" = "commit:" ]; then
+			if [ -n "$commit" ]; then
+				echo "$commit $all"
+				all=
+			fi
+			commit="$y"
+		else
+			all="$all $y"
+		fi
+	done
+	echo "$commit $all"
+}
+x=
+git log --pretty=format:'commit: %H' | joincommits |
+(	while read commit a b; do
+		echo "Verifying commit $commit"
+		check_equal "$b" ""
+		x=1
+	done
+	check_equal "$x" 1
+) || exit 1
+
+echo
+echo 'ok'
-- 
1.6.3.2.g3d624

Re: git subtree: an alternative to git submodule

[Avery Pennarun]

On Fri, May 8, 2009 at 6:39 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> I first sent out this patch set a couple of weeks ago
> (http://article.gmane.org/gmane.comp.version-control.git/117612) and got a
> couple of positive comments, but no negative ones, so I'm guessing people
> haven't reviewed it as closely as I would have hoped :)
>
> git subtree has these subcommands:
>
>  - add: connect a given commit to a given subtree, basically using the
>   occasionally-documented 'git read-tree --prefix; git commit' trick.
>
>  - merge: a user-friendlier form of 'git merge -s subtree'
>
>  - pull: likewise, but for git pull
>
>  - split: (the magical part!) generate a new commit series from the given
>   prefix, so you can submit subtree changes *back* to the upstream project
>   you merged in the first place.
>
> Does anyone have any comments on what it would take to get the git subtree
> stuff accepted into git?

Hi all,

Just checking in again.  Since I originally announced git-subtree, it
(or rather the concept) has received a bit of positive feedback out in
the wild.  My original blog post about it:
http://alumnit.ca/~apenwarr/log/?m=200904#30

The heroku mailing list:
http://groups.google.com/group/heroku/browse_thread/thread/5e6807fcd2572f64

Ycombinator news: http://news.ycombinator.com/item?id=604405
And again: http://news.ycombinator.com/item?id=604889

Meanwhile, I've been keeping it in a separate git repo here:
http://github.com/apenwarr/git-subtree/commits/master (that's a web
link, not a git link).

The common thread in all these discussions is along the lines of,
"This sounds cool!  I didn't try it though.  Maybe I should!"
Commenters then seem to disappear into a black hole.  The black hole
of my code!  Bwahahaha!  Sigh.

I would love to hear some feedback about this, particularly for the
question of what needs to be done to get it adopted into git's
mainline, or if it's in fact so evil that this is unlikely to ever
happen.  Obviously I would need to write a man page, but I've been
hesitant to do that in case people have suggestions that need the
whole UI to change.  Perhaps that's a chicken-and-egg problem, though,
and I should just get on with writing it.

Flame away!

Thanks,

Avery

P.S. Dscho is usually really good at flaming me for being stupid, but
he has been strangely silent on this topic so far.  Don't miss this
exciting opportunity!

Re: git subtree: an alternative to git submodule

[Junio C Hamano]

Avery Pennarun <apenwarr@gmail.com> writes:

> ...  Obviously I would need to write a man page, but I've been
> hesitant to do that in case people have suggestions that need the
> whole UI to change.  Perhaps that's a chicken-and-egg problem, though,...

If you fear that you might get into a situation that the UI _must_ change
because it does not fit people's needs or workflows, that is a sign that
the UI and the workflow it was designed to support may not have been well
thought out yet.  At least, you do not even _know_ if it is well thought
out or not.  It is understandable that people would say "sounds cool,
could potentially be good, but I'll wait and see if it is real" and leave.

With a clear description of "Here is the workflow _I_ assume you use.  If
your usage fits this pattern, this tool may be for you, and here is how it
works and how to use it," it becomes easier for people to say "My usage is
similar but slightly different in this way.  How (perhaps slightly
differently from your description) would I use it?" to help improve the
tool.

Investing the time and effort to get the ball rolling, whether it gets
included in an official tree or not in the immediate future, is a way to
show that the original author _cares_ about the feature.  That would
further help pursuade potential users to look at it.

It is an easy mistake to make to consider inclusion to my tree your goal.
It can be one of the means to give exposure to wider audience, but it does
not have to be your only avenue to do so.

I could throw it in contrib/ but that would help merely by giving easier
access to people who decide that they want to invest their own time to see
if it fits their needs and if they can improve it to match their needs.
You need to do your part of convincing them that it is worth looking at it
first; that is not something "throwing it in contrib/" would help at all.

With proliferation of free hosting services, however, I think contrib/
area for such purposes outlived its usefulness.  People can now fork and
gather interested and enthused users very easily and can make *me* beg to
merge from them to include their new, popular, and already polished
features.

Re: git subtree: an alternative to git submodule

[Avery Pennarun]

On Fri, May 15, 2009 at 2:11 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Avery Pennarun <apenwarr@gmail.com> writes:
>> ...  Obviously I would need to write a man page, but I've been
>> hesitant to do that in case people have suggestions that need the
>> whole UI to change.  Perhaps that's a chicken-and-egg problem, though,...
>
> If you fear that you might get into a situation that the UI _must_ change
> because it does not fit people's needs or workflows, that is a sign that
> the UI and the workflow it was designed to support may not have been well
> thought out yet.  At least, you do not even _know_ if it is well thought
> out or not.  It is understandable that people would say "sounds cool,
> could potentially be good, but I'll wait and see if it is real" and leave.

Well, I'm already using it myself in my own projects and I like it.
So I'm pretty confident that it is *a* useful workflow.  Whether it's
useful for others is a good question, and the only way to know the
answer is to put it out there.

But I'm at a bit of a loss as to why so many people (er, as compared
to none) seem to have gotten excited about the tool, but then it
fizzled.  This implies to me that something is missing.  Perhaps it's
just the documentation; I'll work on that next, then.

> It is an easy mistake to make to consider inclusion to my tree your goal.
> It can be one of the means to give exposure to wider audience, but it does
> not have to be your only avenue to do so.

Thanks for pointing that out.  In fact my primary goal wasn't really
to get it included in the tree (otherwise I *would* have written the
documentation and even included a signed-off-by line :)) but to get
comments on the feature.  In fact, it would be detrimental to have it
included in your tree and then find out afterwards that it ought to be
ripped out and replaced.

> With proliferation of free hosting services, however, I think contrib/
> area for such purposes outlived its usefulness.  People can now fork and
> gather interested and enthused users very easily and can make *me* beg to
> merge from them to include their new, popular, and already polished
> features.

I suppose you could merge it in using git-subtree and then you
wouldn't even have to beg :)

Okay, documentation next.  At least I have somewhere to go from here.

Have fun,

Avery

Re: git subtree: an alternative to git submodule

[Ping Yin]

On Sat, May 16, 2009 at 2:31 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Fri, May 15, 2009 at 2:11 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Avery Pennarun <apenwarr@gmail.com> writes:
>>> ...  Obviously I would need to write a man page, but I've been
>>> hesitant to do that in case people have suggestions that need the
>>> whole UI to change.  Perhaps that's a chicken-and-egg problem, though,...
>>
>> If you fear that you might get into a situation that the UI _must_ change
>> because it does not fit people's needs or workflows, that is a sign that
>> the UI and the workflow it was designed to support may not have been well
>> thought out yet.  At least, you do not even _know_ if it is well thought
>> out or not.  It is understandable that people would say "sounds cool,
>> could potentially be good, but I'll wait and see if it is real" and leave.
>
> Well, I'm already using it myself in my own projects and I like it.
> So I'm pretty confident that it is *a* useful workflow.  Whether it's
> useful for others is a good question, and the only way to know the
> answer is to put it out there.
>
> But I'm at a bit of a loss as to why so many people (er, as compared
> to none) seem to have gotten excited about the tool, but then it
> fizzled.  This implies to me that something is missing.  Perhaps it's
> just the documentation; I'll work on that next, then.
>

It's really a cool feature, but i havn't tried it. Why?

It will spends me some time saving and applying the patches and then
testing it (i don't have the appropriate environment setuped). But I
am busy and there is no urgent need to use this feature ( it is only a
rare case for me).  So i will wait until i need the feature or there
is an easy to fetch the code ( pu of official reposotory or other
repository with these patches applied).

I don't whether this is a common reason, but at least it is the reason for me.

Re: git subtree: an alternative to git submodule

[Ping Yin]

Ping Yin



On Mon, May 18, 2009 at 11:55 PM, Ping Yin <pkufranky@gmail.com> wrote:
> On Sat, May 16, 2009 at 2:31 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
>> On Fri, May 15, 2009 at 2:11 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>> Avery Pennarun <apenwarr@gmail.com> writes:
>>>> ...  Obviously I would need to write a man page, but I've been
>>>> hesitant to do that in case people have suggestions that need the
>>>> whole UI to change.  Perhaps that's a chicken-and-egg problem, though,...
>>>
>>> If you fear that you might get into a situation that the UI _must_ change
>>> because it does not fit people's needs or workflows, that is a sign that
>>> the UI and the workflow it was designed to support may not have been well
>>> thought out yet.  At least, you do not even _know_ if it is well thought
>>> out or not.  It is understandable that people would say "sounds cool,
>>> could potentially be good, but I'll wait and see if it is real" and leave.
>>
>> Well, I'm already using it myself in my own projects and I like it.
>> So I'm pretty confident that it is *a* useful workflow.  Whether it's
>> useful for others is a good question, and the only way to know the
>> answer is to put it out there.
>>
>> But I'm at a bit of a loss as to why so many people (er, as compared
>> to none) seem to have gotten excited about the tool, but then it
>> fizzled.  This implies to me that something is missing.  Perhaps it's
>> just the documentation; I'll work on that next, then.
>>
>
> It's really a cool feature, but i havn't tried it. Why?
>
> It will spends me some time saving and applying the patches and then
> testing it (i don't have the appropriate environment setuped). But I
> am busy and there is no urgent need to use this feature ( it is only a
> rare case for me).  So i will wait until i need the feature or there
> is an easy to fetch the code ( pu of official reposotory or other
> repository with these patches applied).
>
> I don't whether this is a common reason, but at least it is the reason for me.
>

s/dont't/don't know/

Re: git subtree: an alternative to git submodule

[Avery Pennarun]

On Mon, May 18, 2009 at 11:55 AM, Ping Yin <pkufranky@gmail.com> wrote:
> It's really a cool feature, but i havn't tried it. Why?
>
> It will spends me some time saving and applying the patches and then
> testing it (i don't have the appropriate environment setuped). But I
> am busy and there is no urgent need to use this feature ( it is only a
> rare case for me).  So i will wait until i need the feature or there
> is an easy to fetch the code ( pu of official reposotory or other
> repository with these patches applied).

Excellent, thanks for the feedback.  In fact you can git clone the
code from here:

  git clone git://github.com/apenwarr/git-subtree.git

(It's not a copy of the git repo; it's a tiny standalone repo.)

The important file is 'git-subtree'.  Copy this anywhere on your PATH,
and magically the 'git subtree' command will work.

I admit that your next roadblock will probably be lack of
documentation, though, as Junio points out.

Have fun,

Avery

Re: git subtree: an alternative to git submodule

[Ping Yin]

On Tue, May 19, 2009 at 12:38 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Mon, May 18, 2009 at 11:55 AM, Ping Yin <pkufranky@gmail.com> wrote:
>> It's really a cool feature, but i havn't tried it. Why?
>>
>> It will spends me some time saving and applying the patches and then
>> testing it (i don't have the appropriate environment setuped). But I
>> am busy and there is no urgent need to use this feature ( it is only a
>> rare case for me).  So i will wait until i need the feature or there
>> is an easy to fetch the code ( pu of official reposotory or other
>> repository with these patches applied).
>
> Excellent, thanks for the feedback.  In fact you can git clone the
> code from here:
>
>  git clone git://github.com/apenwarr/git-subtree.git
>
> (It's not a copy of the git repo; it's a tiny standalone repo.)
>
> The important file is 'git-subtree'.  Copy this anywhere on your PATH,
> and magically the 'git subtree' command will work.
>
> I admit that your next roadblock will probably be lack of
> documentation, though, as Junio points out.
>

Some problems after trying.

* "split" generates a commit hash which i take some time to figure out
the meaning. May it should accept one more argument (repository name)
to generate the repository directly?

* "pull" creates the merged files to the wrong directory. Following is
the output

git subtree -d pull --prefix=foo git@example.com:foo.git master
command: {pull}
quiet: {}
revs: {}
dir: {foo}
opts: {git@example.com:foo.git master}

+ git pull -s subtree git@example.com:foo.git master
From example.com:foo
 * branch            master     -> FETCH_HEAD
Merge made by subtree.
 scripts/data.example/creatives/1.swf     |  Bin 0 -> 174109 bytes
 scripts/data.example/creatives/2.swf      |  Bin 0 -> 103622 bytes
 scripts/data.example/creatives/3.swf |  Bin 0 -> 35347 bytes
 scripts/data.example/creatives/4.swf  |  Bin 0 -> 16300 bytes
 5 files changed, 0 insertions(+), 0 deletions(-)
...


* "merge" doesn't respect the prefix option. With the following
commands, foo.txt is not merged to subdir foo/

touch foo.txt && git add foo.txt && git commit -m "add foo.txt"
git branch foo  && git reset --hard HEAD^
git subtree merge --prefix=foo foo

Re: git subtree: an alternative to git submodule

[Jakub Narebski]

Avery Pennarun <apenwarr@gmail.com> writes:
> On Fri, May 8, 2009 at 6:39 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> >
> > I first sent out this patch set a couple of weeks ago
> > (http://article.gmane.org/gmane.comp.version-control.git/117612) and got a
> > couple of positive comments, but no negative ones, so I'm guessing people
> > haven't reviewed it as closely as I would have hoped :)
[...]

> Just checking in again.  Since I originally announced git-subtree, it
> (or rather the concept) has received a bit of positive feedback out in
> the wild.  My original blog post about it:
> http://alumnit.ca/~apenwarr/log/?m=200904#30

[...]
> Meanwhile, I've been keeping it in a separate git repo here:
> http://github.com/apenwarr/git-subtree/commits/master (that's a web
> link, not a git link).

I have added information about git-subtree to git wiki on
http://git.or.cz/gitwiki/InterfacesFrontendsAndTools (which page I
imagined to be clearinghouse / list of relevant git related tools).
Please correct any mistakes, and perhaps expand information there.

You might want to add SubtreeMerge page to git wiki, and link it
from subproject pages (SubmoduleSupport, GitSubmoduleTutorial)
and perhaps also from GitDocumentation. But it is not necessary.

-- 
Jakub Narebski
Poland
ShadeHawk on #git