[PATCH v2 00/19] Multiparent diff tree-walker + combine-diff speedup

[PATCH v2 00/19] Multiparent diff tree-walker + combine-diff speedup

[Kirill Smelkov]

Hello up there.

Here go combine-diff speedup patches in form of first reworking diff
tree-walker to work in general case - when a commit have several parents, not
only one - we are traversing all 1+nparent trees in parallel.

Then we are taking advantage of the new diff tree-walker for speeding up
combine-diff, which for linux.git results in ~14 times speedup.

This is the second posting for the whole series - sent here patches should go
instead of already-in-pu ks/diff-tree-more and ks/tree-diff-nway into
ks/tree-diff-nway - patches are related and seeing them all at once is more
logical to me.

I've tried to do my homework based on review feedback and the changes compared
to v1 are:

- fixed last-minute thinko/bug last time introduced on my side (sorry) with
  opt->pathchange manipulation in __diff_tree_sha1() - we were forgetting to
  restore opt->pathchange, which led to incorrect log -c (merges _and_ plain
  diff-tree) output;

  This time, I've verified several times, log output stays really the same.

- direct use of alloca() changed to portability wrappers xalloca/xalloca_free
  which gracefully degrade to xmalloc/free on systems, where alloca is not
  available (see new patch 17).

- "i = 0; do { ... } while (++i < nparent)" is back to usual looping
  "for (i = 0; i < nparent; ++)", as I've re-measured timings and the
  difference is negligible.

  ( Initially, when I was fighting for every cycle it made sense, but real
    no-slowdown turned out to be related to avoiding mallocs, load trees in correct
    order and reducing register pressure. )

- S_IFXMIN_NEQ definition moved out to cache.h, to have all modes registry in one place;


- diff_tree() becomes static (new patch 13), as nobody is using it outside
  tree-diff.c (and is later renamed to __diff_tree_sha1);

- p0 -> first_parent; corrected comments about how emit_diff_first_parent_only
  behaves;


not changed:

- low-level helpers are still named with "__" prefix as, imho, that is the best
  convention to name such helpers, without sacrificing signal/noise ratio. All
  of them are now static though.


Signoffs were left intact, if a patch was already applied to pu with one, and
had not changed.

Please apply and thanks,
Kirill

P.S. Sorry for the delay - I was very busy.


Kirill Smelkov (19):
  combine-diff: move show_log_first logic/action out of paths scanning
  combine-diff: move changed-paths scanning logic into its own function
  tree-diff: no need to manually verify that there is no mode change for a path
  tree-diff: no need to pass match to skip_uninteresting()
  tree-diff: show_tree() is not needed
  tree-diff: consolidate code for emitting diffs and recursion in one place
  tree-diff: don't assume compare_tree_entry() returns -1,0,1
  tree-diff: move all action-taking code out of compare_tree_entry()
  tree-diff: rename compare_tree_entry -> tree_entry_pathcmp
  tree-diff: show_path prototype is not needed anymore
  tree-diff: simplify tree_entry_pathcmp
  tree-diff: remove special-case diff-emitting code for empty-tree cases
  tree-diff: diff_tree() should now be static
  tree-diff: rework diff_tree interface to be sha1 based
  tree-diff: no need to call "full" diff_tree_sha1 from show_path()
  tree-diff: reuse base str(buf) memory on sub-tree recursion
  Portable alloca for Git
  tree-diff: rework diff_tree() to generate diffs for multiparent cases as well
  combine-diff: speed it up, by using multiparent diff tree-walker directly

 Makefile          |   6 +
 cache.h           |  15 ++
 combine-diff.c    | 170 +++++++++++---
 config.mak.uname  |  10 +-
 configure.ac      |   8 +
 diff.c            |   2 +
 diff.h            |  12 +-
 git-compat-util.h |   8 +
 tree-diff.c       | 666 +++++++++++++++++++++++++++++++++++++++++++-----------
 9 files changed, 724 insertions(+), 173 deletions(-)

-- 
1.9.rc1.181.g641f458

[PATCH 01/19] combine-diff: move show_log_first logic/action out of paths scanning

[Kirill Smelkov]

Judging from sample outputs and tests nothing changes in diff -c output,
and this change will help later patches, when we'll be refactoring paths
scanning into its own function with several variants - the
show_log_first logic / code will stay common to all of them.

NOTE: only now we have to take care to explicitly not show anything if
    parents array is empty, as in fact there are some clients in Git code,
    which calls diff_tree_combined() in such a way.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 combine-diff.c | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/combine-diff.c b/combine-diff.c
index 24ca7e2..68d2e53 100644
--- a/combine-diff.c
+++ b/combine-diff.c
@@ -1311,6 +1311,20 @@ void diff_tree_combined(const unsigned char *sha1,
 	struct combine_diff_path *p, *paths = NULL;
 	int i, num_paths, needsep, show_log_first, num_parent = parents->nr;
 
+	/* nothing to do, if no parents */
+	if (!num_parent)
+		return;
+
+	show_log_first = !!rev->loginfo && !rev->no_commit_id;
+	needsep = 0;
+	if (show_log_first) {
+		show_log(rev);
+
+		if (rev->verbose_header && opt->output_format)
+			printf("%s%c", diff_line_prefix(opt),
+			       opt->line_termination);
+	}
+
 	diffopts = *opt;
 	copy_pathspec(&diffopts.pathspec, &opt->pathspec);
 	diffopts.output_format = DIFF_FORMAT_NO_OUTPUT;
@@ -1319,8 +1333,6 @@ void diff_tree_combined(const unsigned char *sha1,
 	/* tell diff_tree to emit paths in sorted (=tree) order */
 	diffopts.orderfile = NULL;
 
-	show_log_first = !!rev->loginfo && !rev->no_commit_id;
-	needsep = 0;
 	/* find set of paths that everybody touches */
 	for (i = 0; i < num_parent; i++) {
 		/* show stat against the first parent even
@@ -1336,14 +1348,6 @@ void diff_tree_combined(const unsigned char *sha1,
 		diffcore_std(&diffopts);
 		paths = intersect_paths(paths, i, num_parent);
 
-		if (show_log_first && i == 0) {
-			show_log(rev);
-
-			if (rev->verbose_header && opt->output_format)
-				printf("%s%c", diff_line_prefix(opt),
-				       opt->line_termination);
-		}
-
 		/* if showing diff, show it in requested order */
 		if (diffopts.output_format != DIFF_FORMAT_NO_OUTPUT &&
 		    opt->orderfile) {
-- 
1.9.rc1.181.g641f458

[PATCH 02/19] combine-diff: move changed-paths scanning logic into its own function

[Kirill Smelkov]

Move code for finding paths for which diff(commit,parent_i) is not-empty
for all parents to separate function - at present we have generic (and
slow) code for this job, which translates 1 n-parent problem to n
1-parent problems and then intersect results, and will be adding another
limited, but faster, paths scanning implementation in the next patch.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 combine-diff.c | 80 ++++++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 53 insertions(+), 27 deletions(-)

diff --git a/combine-diff.c b/combine-diff.c
index 68d2e53..1732dfd 100644
--- a/combine-diff.c
+++ b/combine-diff.c
@@ -1301,6 +1301,51 @@ static const char *path_path(void *obj)
 	return path->path;
 }
 
+
+/* find set of paths that every parent touches */
+static struct combine_diff_path *find_paths(const unsigned char *sha1,
+	const struct sha1_array *parents, struct diff_options *opt)
+{
+	struct combine_diff_path *paths = NULL;
+	int i, num_parent = parents->nr;
+
+	int output_format = opt->output_format;
+	const char *orderfile = opt->orderfile;
+
+	opt->output_format = DIFF_FORMAT_NO_OUTPUT;
+	/* tell diff_tree to emit paths in sorted (=tree) order */
+	opt->orderfile = NULL;
+
+	for (i = 0; i < num_parent; i++) {
+		/*
+		 * show stat against the first parent even when doing
+		 * combined diff.
+		 */
+		int stat_opt = (output_format &
+				(DIFF_FORMAT_NUMSTAT|DIFF_FORMAT_DIFFSTAT));
+		if (i == 0 && stat_opt)
+			opt->output_format = stat_opt;
+		else
+			opt->output_format = DIFF_FORMAT_NO_OUTPUT;
+		diff_tree_sha1(parents->sha1[i], sha1, "", opt);
+		diffcore_std(opt);
+		paths = intersect_paths(paths, i, num_parent);
+
+		/* if showing diff, show it in requested order */
+		if (opt->output_format != DIFF_FORMAT_NO_OUTPUT &&
+		    orderfile) {
+			diffcore_order(orderfile);
+		}
+
+		diff_flush(opt);
+	}
+
+	opt->output_format = output_format;
+	opt->orderfile = orderfile;
+	return paths;
+}
+
+
 void diff_tree_combined(const unsigned char *sha1,
 			const struct sha1_array *parents,
 			int dense,
@@ -1308,7 +1353,7 @@ void diff_tree_combined(const unsigned char *sha1,
 {
 	struct diff_options *opt = &rev->diffopt;
 	struct diff_options diffopts;
-	struct combine_diff_path *p, *paths = NULL;
+	struct combine_diff_path *p, *paths;
 	int i, num_paths, needsep, show_log_first, num_parent = parents->nr;
 
 	/* nothing to do, if no parents */
@@ -1327,35 +1372,16 @@ void diff_tree_combined(const unsigned char *sha1,
 
 	diffopts = *opt;
 	copy_pathspec(&diffopts.pathspec, &opt->pathspec);
-	diffopts.output_format = DIFF_FORMAT_NO_OUTPUT;
 	DIFF_OPT_SET(&diffopts, RECURSIVE);
 	DIFF_OPT_CLR(&diffopts, ALLOW_EXTERNAL);
-	/* tell diff_tree to emit paths in sorted (=tree) order */
-	diffopts.orderfile = NULL;
 
-	/* find set of paths that everybody touches */
-	for (i = 0; i < num_parent; i++) {
-		/* show stat against the first parent even
-		 * when doing combined diff.
-		 */
-		int stat_opt = (opt->output_format &
-				(DIFF_FORMAT_NUMSTAT|DIFF_FORMAT_DIFFSTAT));
-		if (i == 0 && stat_opt)
-			diffopts.output_format = stat_opt;
-		else
-			diffopts.output_format = DIFF_FORMAT_NO_OUTPUT;
-		diff_tree_sha1(parents->sha1[i], sha1, "", &diffopts);
-		diffcore_std(&diffopts);
-		paths = intersect_paths(paths, i, num_parent);
-
-		/* if showing diff, show it in requested order */
-		if (diffopts.output_format != DIFF_FORMAT_NO_OUTPUT &&
-		    opt->orderfile) {
-			diffcore_order(opt->orderfile);
-		}
-
-		diff_flush(&diffopts);
-	}
+	/* find set of paths that everybody touches
+	 *
+	 * NOTE find_paths() also handles --stat, as it computes
+	 * diff(sha1,parent_i) for all i to do the job, specifically
+	 * for parent0.
+	 */
+	paths = find_paths(sha1, parents, &diffopts);
 
 	/* find out number of surviving paths */
 	for (num_paths = 0, p = paths; p; p = p->next)
-- 
1.9.rc1.181.g641f458

[PATCH 03/19] tree-diff: no need to manually verify that there is no mode change for a path

[Kirill Smelkov]

Because if there is, such two tree entries would never be compared as
equal - the code in base_name_compare() explicitly compares modes, if
there is a change for dir bit, even for equal paths, entries would
compare as different.

The code I'm removing here is from 2005 April 262e82b4 (Fix diff-tree
recursion), which pre-dates base_name_compare() introduction in 958ba6c9
(Introduce "base_name_compare()" helper function) by a month.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 tree-diff.c | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 11c3550..5810b00 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -23,6 +23,11 @@ static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2,
 
 	pathlen1 = tree_entry_len(&t1->entry);
 	pathlen2 = tree_entry_len(&t2->entry);
+
+	/*
+	 * NOTE files and directories *always* compare differently,
+	 * even when having the same name.
+	 */
 	cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
 	if (cmp < 0) {
 		show_entry(opt, "-", t1, base);
@@ -35,16 +40,6 @@ static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2,
 	if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER) && !hashcmp(sha1, sha2) && mode1 == mode2)
 		return 0;
 
-	/*
-	 * If the filemode has changed to/from a directory from/to a regular
-	 * file, we need to consider it a remove and an add.
-	 */
-	if (S_ISDIR(mode1) != S_ISDIR(mode2)) {
-		show_entry(opt, "-", t1, base);
-		show_entry(opt, "+", t2, base);
-		return 0;
-	}
-
 	strbuf_add(base, path1, pathlen1);
 	if (DIFF_OPT_TST(opt, RECURSIVE) && S_ISDIR(mode1)) {
 		if (DIFF_OPT_TST(opt, TREE_IN_RECURSIVE)) {
-- 
1.9.rc1.181.g641f458

[PATCH 04/19] tree-diff: no need to pass match to skip_uninteresting()

[Kirill Smelkov]

It is neither used there as input, nor the output written through it, is
used outside.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 tree-diff.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 5810b00..a8c2aec 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -109,13 +109,14 @@ static void show_entry(struct diff_options *opt, const char *prefix,
 }
 
 static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
-			       struct diff_options *opt,
-			       enum interesting *match)
+			       struct diff_options *opt)
 {
+	enum interesting match;
+
 	while (t->size) {
-		*match = tree_entry_interesting(&t->entry, base, 0, &opt->pathspec);
-		if (*match) {
-			if (*match == all_entries_not_interesting)
+		match = tree_entry_interesting(&t->entry, base, 0, &opt->pathspec);
+		if (match) {
+			if (match == all_entries_not_interesting)
 				t->size = 0;
 			break;
 		}
@@ -128,8 +129,6 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 {
 	struct strbuf base;
 	int baselen = strlen(base_str);
-	enum interesting t1_match = entry_not_interesting;
-	enum interesting t2_match = entry_not_interesting;
 
 	/* Enable recursion indefinitely */
 	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
@@ -141,8 +140,8 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 		if (diff_can_quit_early(opt))
 			break;
 		if (opt->pathspec.nr) {
-			skip_uninteresting(t1, &base, opt, &t1_match);
-			skip_uninteresting(t2, &base, opt, &t2_match);
+			skip_uninteresting(t1, &base, opt);
+			skip_uninteresting(t2, &base, opt);
 		}
 		if (!t1->size) {
 			if (!t2->size)
-- 
1.9.rc1.181.g641f458

[PATCH 05/19] tree-diff: show_tree() is not needed

[Kirill Smelkov]

We don't need special code for showing added/removed subtree, because we
can do the same via diff_tree_sha1, just passing NULL for absent tree.

And compared to show_tree(), which was calling show_entry() for every
tree entry, that would lead to the same show_entry() callings:

    show_tree(t):
        for e in t.entries:
            show_entry(e)

    diff_tree_sha1(NULL, new):  /* the same applies to (old, NULL) */
        diff_tree(t1=NULL, t2)
            ...
            if (!t1->size)
                show_entry(t2)
            ...

and possible overhead is negligible, since after the patch, timing for

    `git log --raw --no-abbrev --no-renames`

for navy.git and `linux.git v3.10..v3.11` is practically the same.

So let's say goodbye to show_tree() - it removes some code, but also,
and what is important, consolidates more code for showing/recursing into
trees into one place.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

( re-posting without change )

 tree-diff.c | 35 +++--------------------------------
 1 file changed, 3 insertions(+), 32 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index a8c2aec..2ad7788 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -55,25 +55,7 @@ static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2,
 	return 0;
 }
 
-/* A whole sub-tree went away or appeared */
-static void show_tree(struct diff_options *opt, const char *prefix,
-		      struct tree_desc *desc, struct strbuf *base)
-{
-	enum interesting match = entry_not_interesting;
-	for (; desc->size; update_tree_entry(desc)) {
-		if (match != all_entries_interesting) {
-			match = tree_entry_interesting(&desc->entry, base, 0,
-						       &opt->pathspec);
-			if (match == all_entries_not_interesting)
-				break;
-			if (match == entry_not_interesting)
-				continue;
-		}
-		show_entry(opt, prefix, desc, base);
-	}
-}
-
-/* A file entry went away or appeared */
+/* An entry went away or appeared */
 static void show_entry(struct diff_options *opt, const char *prefix,
 		       struct tree_desc *desc, struct strbuf *base)
 {
@@ -85,23 +67,12 @@ static void show_entry(struct diff_options *opt, const char *prefix,
 
 	strbuf_add(base, path, pathlen);
 	if (DIFF_OPT_TST(opt, RECURSIVE) && S_ISDIR(mode)) {
-		enum object_type type;
-		struct tree_desc inner;
-		void *tree;
-		unsigned long size;
-
-		tree = read_sha1_file(sha1, &type, &size);
-		if (!tree || type != OBJ_TREE)
-			die("corrupt tree sha %s", sha1_to_hex(sha1));
-
 		if (DIFF_OPT_TST(opt, TREE_IN_RECURSIVE))
 			opt->add_remove(opt, *prefix, mode, sha1, 1, base->buf, 0);
 
 		strbuf_addch(base, '/');
-
-		init_tree_desc(&inner, tree, size);
-		show_tree(opt, prefix, &inner, base);
-		free(tree);
+		diff_tree_sha1(*prefix == '-' ? sha1 : NULL,
+			       *prefix == '+' ? sha1 : NULL, base->buf, opt);
 	} else
 		opt->add_remove(opt, prefix[0], mode, sha1, 1, base->buf, 0);
 
-- 
1.9.rc1.181.g641f458

[PATCH 06/19] tree-diff: consolidate code for emitting diffs and recursion in one place

[Kirill Smelkov]

Currently both compare_tree_entry() and show_path() invoke opt diff
callbacks (opt->add_remove() and opt->change()), and also they both have
code which decides whether to recurse into sub-tree, and whether to emit
a tree as separate entry if DIFF_OPT_TREE_IN_RECURSIVE is set.

I.e. we have code duplication and logic scattered on two places.

Let's consolidate it - all diff emmiting code and recurion logic moves
to show_entry, which is now named as show_path, because it shows diff
for a path, based on up to two tree entries, with actual diff emitting
code being kept in new helper emit_diff() for clarity.

What we have as the result, is that compare_tree_entry is now free from
code with logic for diff generation, and also performance is not
affected as timings for

    `git log --raw --no-abbrev --no-renames`

for navy.git and `linux.git v3.10..v3.11`, just like in previous patch,
stay the same.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 tree-diff.c | 115 ++++++++++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 84 insertions(+), 31 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 2ad7788..a5b9ff9 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -6,8 +6,8 @@
 #include "diffcore.h"
 #include "tree.h"
 
-static void show_entry(struct diff_options *opt, const char *prefix,
-		       struct tree_desc *desc, struct strbuf *base);
+static void show_path(struct strbuf *base, struct diff_options *opt,
+		      struct tree_desc *t1, struct tree_desc *t2);
 
 static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2,
 			      struct strbuf *base, struct diff_options *opt)
@@ -16,7 +16,6 @@ static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2,
 	const char *path1, *path2;
 	const unsigned char *sha1, *sha2;
 	int cmp, pathlen1, pathlen2;
-	int old_baselen = base->len;
 
 	sha1 = tree_entry_extract(t1, &path1, &mode1);
 	sha2 = tree_entry_extract(t2, &path2, &mode2);
@@ -30,51 +29,105 @@ static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2,
 	 */
 	cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
 	if (cmp < 0) {
-		show_entry(opt, "-", t1, base);
+		show_path(base, opt, t1, /*t2=*/NULL);
 		return -1;
 	}
 	if (cmp > 0) {
-		show_entry(opt, "+", t2, base);
+		show_path(base, opt, /*t1=*/NULL, t2);
 		return 1;
 	}
 	if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER) && !hashcmp(sha1, sha2) && mode1 == mode2)
 		return 0;
 
-	strbuf_add(base, path1, pathlen1);
-	if (DIFF_OPT_TST(opt, RECURSIVE) && S_ISDIR(mode1)) {
-		if (DIFF_OPT_TST(opt, TREE_IN_RECURSIVE)) {
-			opt->change(opt, mode1, mode2,
-				    sha1, sha2, 1, 1, base->buf, 0, 0);
-		}
-		strbuf_addch(base, '/');
-		diff_tree_sha1(sha1, sha2, base->buf, opt);
-	} else {
-		opt->change(opt, mode1, mode2, sha1, sha2, 1, 1, base->buf, 0, 0);
-	}
-	strbuf_setlen(base, old_baselen);
+	show_path(base, opt, t1, t2);
 	return 0;
 }
 
-/* An entry went away or appeared */
-static void show_entry(struct diff_options *opt, const char *prefix,
-		       struct tree_desc *desc, struct strbuf *base)
+
+/* convert path, t1/t2 -> opt->diff_*() callbacks */
+static void emit_diff(struct diff_options *opt, struct strbuf *path,
+		      struct tree_desc *t1, struct tree_desc *t2)
+{
+	unsigned int mode1 = t1 ? t1->entry.mode : 0;
+	unsigned int mode2 = t2 ? t2->entry.mode : 0;
+
+	if (mode1 && mode2) {
+		opt->change(opt, mode1, mode2, t1->entry.sha1, t2->entry.sha1,
+			1, 1, path->buf, 0, 0);
+	}
+	else {
+		const unsigned char *sha1;
+		unsigned int mode;
+		int addremove;
+
+		if (mode2) {
+			addremove = '+';
+			sha1 = t2->entry.sha1;
+			mode = mode2;
+		}
+		else {
+			addremove = '-';
+			sha1 = t1->entry.sha1;
+			mode = mode1;
+		}
+
+		opt->add_remove(opt, addremove, mode, sha1, 1, path->buf, 0);
+	}
+}
+
+
+/* new path should be added to diff
+ *
+ * 3 cases on how/when it should be called and behaves:
+ *
+ *	!t1,  t2	-> path added, parent lacks it
+ *	 t1, !t2	-> path removed from parent
+ *	 t1,  t2	-> path modified
+ */
+static void show_path(struct strbuf *base, struct diff_options *opt,
+		      struct tree_desc *t1, struct tree_desc *t2)
 {
 	unsigned mode;
 	const char *path;
-	const unsigned char *sha1 = tree_entry_extract(desc, &path, &mode);
-	int pathlen = tree_entry_len(&desc->entry);
+	int pathlen;
 	int old_baselen = base->len;
+	int isdir, recurse = 0, emitthis = 1;
+
+	/* at least something has to be valid */
+	assert(t1 || t2);
+
+	if (t2) {
+		/* path present in resulting tree */
+		tree_entry_extract(t2, &path, &mode);
+		pathlen = tree_entry_len(&t2->entry);
+		isdir = S_ISDIR(mode);
+	}
+	else {
+		/* a path was removed - take path from parent. Also take
+		 * mode from parent, to decide on recursion.
+		 */
+		tree_entry_extract(t1, &path, &mode);
+		pathlen = tree_entry_len(&t1->entry);
+
+		isdir = S_ISDIR(mode);
+		mode = 0;
+	}
+
+	if (DIFF_OPT_TST(opt, RECURSIVE) && isdir) {
+		recurse = 1;
+		emitthis = DIFF_OPT_TST(opt, TREE_IN_RECURSIVE);
+	}
 
 	strbuf_add(base, path, pathlen);
-	if (DIFF_OPT_TST(opt, RECURSIVE) && S_ISDIR(mode)) {
-		if (DIFF_OPT_TST(opt, TREE_IN_RECURSIVE))
-			opt->add_remove(opt, *prefix, mode, sha1, 1, base->buf, 0);
 
+	if (emitthis)
+		emit_diff(opt, base, t1, t2);
+
+	if (recurse) {
 		strbuf_addch(base, '/');
-		diff_tree_sha1(*prefix == '-' ? sha1 : NULL,
-			       *prefix == '+' ? sha1 : NULL, base->buf, opt);
-	} else
-		opt->add_remove(opt, prefix[0], mode, sha1, 1, base->buf, 0);
+		diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
+			       t2 ? t2->entry.sha1 : NULL, base->buf, opt);
+	}
 
 	strbuf_setlen(base, old_baselen);
 }
@@ -117,12 +170,12 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 		if (!t1->size) {
 			if (!t2->size)
 				break;
-			show_entry(opt, "+", t2, &base);
+			show_path(&base, opt, /*t1=*/NULL, t2);
 			update_tree_entry(t2);
 			continue;
 		}
 		if (!t2->size) {
-			show_entry(opt, "-", t1, &base);
+			show_path(&base, opt, t1, /*t2=*/NULL);
 			update_tree_entry(t1);
 			continue;
 		}
-- 
1.9.rc1.181.g641f458

[PATCH 07/19] tree-diff: don't assume compare_tree_entry() returns -1,0,1

[Kirill Smelkov]

It does, but we'll be reworking it in the next patch after it won't, and
besides it is better to stick to standard
strcmp/memcmp/base_name_compare/etc... convention, where comparison
function returns <0, =0, >0

Regarding performance, comparing for <0, =0, >0 should be a little bit
faster, than switch, because it is just 1 test-without-immediate
instruction and then up to 3 conditional branches, and in switch you
have up to 3 tests with immediate and up to 3 conditional branches.

No worry, that update_tree_entry(t2) is duplicated for =0 and >0 - it
will be good after we'll be adding support for multiparent walker and
will stay that way.

=0 case goes first, because it happens more often in real diffs - i.e.
paths are the same.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 tree-diff.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index a5b9ff9..5f7dbbf 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -179,18 +179,24 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 			update_tree_entry(t1);
 			continue;
 		}
-		switch (compare_tree_entry(t1, t2, &base, opt)) {
-		case -1:
+
+		cmp = compare_tree_entry(t1, t2, &base, opt);
+
+		/* t1 = t2 */
+		if (cmp == 0) {
 			update_tree_entry(t1);
-			continue;
-		case 0:
+			update_tree_entry(t2);
+		}
+
+		/* t1 < t2 */
+		else if (cmp < 0) {
 			update_tree_entry(t1);
-			/* Fallthrough */
-		case 1:
+		}
+
+		/* t1 > t2 */
+		else {
 			update_tree_entry(t2);
-			continue;
 		}
-		die("git diff-tree: internal error");
 	}
 
 	strbuf_release(&base);
-- 
1.9.rc1.181.g641f458

[PATCH 08/19] tree-diff: move all action-taking code out of compare_tree_entry()

[Kirill Smelkov]

- let it do only comparison.

This way the code is cleaner and more structured - cmp function only
compares, and the driver takes action based on comparison result.

There should be no change in performance, as effectively, we just move
if series from on place into another, and merge it to was-already-there
same switch/if, so the result is maybe a little bit faster.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 tree-diff.c | 28 ++++++++++++----------------
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 5f7dbbf..6207372 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -9,8 +9,7 @@
 static void show_path(struct strbuf *base, struct diff_options *opt,
 		      struct tree_desc *t1, struct tree_desc *t2);
 
-static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2,
-			      struct strbuf *base, struct diff_options *opt)
+static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2)
 {
 	unsigned mode1, mode2;
 	const char *path1, *path2;
@@ -28,19 +27,7 @@ static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2,
 	 * even when having the same name.
 	 */
 	cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
-	if (cmp < 0) {
-		show_path(base, opt, t1, /*t2=*/NULL);
-		return -1;
-	}
-	if (cmp > 0) {
-		show_path(base, opt, /*t1=*/NULL, t2);
-		return 1;
-	}
-	if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER) && !hashcmp(sha1, sha2) && mode1 == mode2)
-		return 0;
-
-	show_path(base, opt, t1, t2);
-	return 0;
+	return cmp;
 }
 
 
@@ -161,6 +148,8 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 	strbuf_add(&base, base_str, baselen);
 
 	for (;;) {
+		int cmp;
+
 		if (diff_can_quit_early(opt))
 			break;
 		if (opt->pathspec.nr) {
@@ -180,21 +169,28 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 			continue;
 		}
 
-		cmp = compare_tree_entry(t1, t2, &base, opt);
+		cmp = compare_tree_entry(t1, t2);
 
 		/* t1 = t2 */
 		if (cmp == 0) {
+			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
+			    hashcmp(t1->entry.sha1, t2->entry.sha1) ||
+			    (t1->entry.mode != t2->entry.mode))
+				show_path(&base, opt, t1, t2);
+
 			update_tree_entry(t1);
 			update_tree_entry(t2);
 		}
 
 		/* t1 < t2 */
 		else if (cmp < 0) {
+			show_path(&base, opt, t1, /*t2=*/NULL);
 			update_tree_entry(t1);
 		}
 
 		/* t1 > t2 */
 		else {
+			show_path(&base, opt, /*t1=*/NULL, t2);
 			update_tree_entry(t2);
 		}
 	}
-- 
1.9.rc1.181.g641f458

[PATCH 09/19] tree-diff: rename compare_tree_entry -> tree_entry_pathcmp

[Kirill Smelkov]

Since previous commit, this function does not compare entry hashes, and
mode are compared fully outside of it. So what it does is compare entry
names and DIR bit in modes. Reflect this in its name.

Add documentation stating the semantics, and move the note about
files/dirs comparison to it.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 tree-diff.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 6207372..3345534 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -9,7 +9,14 @@
 static void show_path(struct strbuf *base, struct diff_options *opt,
 		      struct tree_desc *t1, struct tree_desc *t2);
 
-static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2)
+/*
+ * Compare two tree entries, taking into account only path/S_ISDIR(mode),
+ * but not their sha1's.
+ *
+ * NOTE files and directories *always* compare differently, even when having
+ *      the same name - thanks to base_name_compare().
+ */
+static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 {
 	unsigned mode1, mode2;
 	const char *path1, *path2;
@@ -22,10 +29,6 @@ static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2)
 	pathlen1 = tree_entry_len(&t1->entry);
 	pathlen2 = tree_entry_len(&t2->entry);
 
-	/*
-	 * NOTE files and directories *always* compare differently,
-	 * even when having the same name.
-	 */
 	cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
 	return cmp;
 }
@@ -169,7 +172,7 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 			continue;
 		}
 
-		cmp = compare_tree_entry(t1, t2);
+		cmp = tree_entry_pathcmp(t1, t2);
 
 		/* t1 = t2 */
 		if (cmp == 0) {
-- 
1.9.rc1.181.g641f458

[PATCH 10/19] tree-diff: show_path prototype is not needed anymore

[Kirill Smelkov]

We moved all action-taking code below show_path() in recent HEAD~~
(tree-diff: move all action-taking code out of compare_tree_entry).

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 tree-diff.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 3345534..20a4fda 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -6,9 +6,6 @@
 #include "diffcore.h"
 #include "tree.h"
 
-static void show_path(struct strbuf *base, struct diff_options *opt,
-		      struct tree_desc *t1, struct tree_desc *t2);
-
 /*
  * Compare two tree entries, taking into account only path/S_ISDIR(mode),
  * but not their sha1's.
-- 
1.9.rc1.181.g641f458

[PATCH 11/19] tree-diff: simplify tree_entry_pathcmp

[Kirill Smelkov]

Since an earlier "Finally switch over tree descriptors to contain a
pre-parsed entry", we can safely access all tree_desc->entry fields
directly instead of first "extracting" them through
tree_entry_extract.

Use it. The code generated stays the same - only it now visually looks
cleaner.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 tree-diff.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 20a4fda..cf96ad7 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -15,18 +15,13 @@
  */
 static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 {
-	unsigned mode1, mode2;
-	const char *path1, *path2;
-	const unsigned char *sha1, *sha2;
-	int cmp, pathlen1, pathlen2;
+	struct name_entry *e1, *e2;
+	int cmp;
 
-	sha1 = tree_entry_extract(t1, &path1, &mode1);
-	sha2 = tree_entry_extract(t2, &path2, &mode2);
-
-	pathlen1 = tree_entry_len(&t1->entry);
-	pathlen2 = tree_entry_len(&t2->entry);
-
-	cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
+	e1 = &t1->entry;
+	e2 = &t2->entry;
+	cmp = base_name_compare(e1->path, tree_entry_len(e1), e1->mode,
+				e2->path, tree_entry_len(e2), e2->mode);
 	return cmp;
 }
 
-- 
1.9.rc1.181.g641f458

[PATCH 12/19] tree-diff: remove special-case diff-emitting code for empty-tree cases

[Kirill Smelkov]

via teaching tree_entry_pathcmp() how to compare empty tree descriptors:

While walking trees, we iterate their entries from lowest to highest in
sort order, so empty tree means all entries were already went over.

If we artificially assign +infinity value to such tree "entry", it will
go after all usual entries, and through the usual driver loop we will be
taking the same actions, which were hand-coded for special cases, i.e.

    t1 empty, t2 non-empty
        pathcmp(+∞, t2) -> +1
        show_path(/*t1=*/NULL, t2);     /* = t1 > t2 case in main loop */

    t1 non-empty, t2-empty
        pathcmp(t1, +∞) -> -1
        show_path(t1, /*t2=*/NULL);     /* = t1 < t2 case in main loop */

Right now we never go to when compared tree descriptors are infinity, as
this condition is checked in the loop beginning as finishing criteria,
but will do in the future, when there will be several parents iterated
simultaneously, and some pair of them would run to the end.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 tree-diff.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index cf96ad7..2fd6d0e 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -12,12 +12,19 @@
  *
  * NOTE files and directories *always* compare differently, even when having
  *      the same name - thanks to base_name_compare().
+ *
+ * NOTE empty (=invalid) descriptor(s) take part in comparison as +infty.
  */
 static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 {
 	struct name_entry *e1, *e2;
 	int cmp;
 
+	if (!t1->size)
+		return t2->size ? +1 /* +∞ > c */  : 0 /* +∞ = +∞ */;
+	else if (!t2->size)
+		return -1;	/* c < +∞ */
+
 	e1 = &t1->entry;
 	e2 = &t2->entry;
 	cmp = base_name_compare(e1->path, tree_entry_len(e1), e1->mode,
@@ -151,18 +158,8 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 			skip_uninteresting(t1, &base, opt);
 			skip_uninteresting(t2, &base, opt);
 		}
-		if (!t1->size) {
-			if (!t2->size)
-				break;
-			show_path(&base, opt, /*t1=*/NULL, t2);
-			update_tree_entry(t2);
-			continue;
-		}
-		if (!t2->size) {
-			show_path(&base, opt, t1, /*t2=*/NULL);
-			update_tree_entry(t1);
-			continue;
-		}
+		if (!t1->size && !t2->size)
+			break;
 
 		cmp = tree_entry_pathcmp(t1, t2);
 
-- 
1.9.rc1.181.g641f458

[PATCH 13/19] tree-diff: diff_tree() should now be static

[Kirill Smelkov]

We reworked all its users to use the functionality through
diff_tree_sha1 variant in recent patches (see "tree-diff: allow
diff_tree_sha1 to accept NULL sha1" and what comes next).

diff_tree() is now not used outside tree-diff.c - make it static.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

 ( new patch )

 diff.h      | 2 --
 tree-diff.c | 4 ++--
 2 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/diff.h b/diff.h
index e79f3b3..5d7b9f7 100644
--- a/diff.h
+++ b/diff.h
@@ -189,8 +189,6 @@ const char *diff_line_prefix(struct diff_options *);
 
 extern const char mime_boundary_leader[];
 
-extern int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
-		     const char *base, struct diff_options *opt);
 extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 			  const char *base, struct diff_options *opt);
 extern int diff_root_tree_sha1(const unsigned char *new, const char *base,
diff --git a/tree-diff.c b/tree-diff.c
index 2fd6d0e..b99622c 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -137,8 +137,8 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
 	}
 }
 
-int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
-	      const char *base_str, struct diff_options *opt)
+static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
+		     const char *base_str, struct diff_options *opt)
 {
 	struct strbuf base;
 	int baselen = strlen(base_str);
-- 
1.9.rc1.181.g641f458

[PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Kirill Smelkov]

In the next commit this will allow to reduce intermediate calls, when
recursing into subtrees - at that stage we know only subtree sha1, and
it is natural for tree walker to start from that phase. For now we do

    diff_tree
        show_path
            diff_tree_sha1
                diff_tree
                    ...

and the change will allow to reduce it to

    diff_tree
        show_path
            diff_tree

Also, it will allow to omit allocating strbuf for each subtree, and just
reuse the common strbuf via playing with its len.

The above-mentioned improvements go in the next 2 patches.

The downside is that try_to_follow_renames(), if active, we cause
re-reading of 2 initial trees, which was negligible based on my timings,
and which is outweighed cogently by the upsides.

NOTE To keep with the current interface and semantics, I needed to
rename the function from diff_tree() to diff_tree_sha1(). As
diff_tree_sha1() was already used, and the function we are talking here
is its more low-level helper, let's use Linux convention for prefixing
such helpers with double underscore. So the final renaming is

    diff_tree() -> __diff_tree_sha1()

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

Changes since v1:

 - don't need to touch diff.h, as diff_tree() became static.

 tree-diff.c | 60 ++++++++++++++++++++++++++++--------------------------------
 1 file changed, 28 insertions(+), 32 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index b99622c..f90acf5 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -137,12 +137,17 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
 	}
 }
 
-static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
-		     const char *base_str, struct diff_options *opt)
+static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
+			    const char *base_str, struct diff_options *opt)
 {
+	struct tree_desc t1, t2;
+	void *t1tree, *t2tree;
 	struct strbuf base;
 	int baselen = strlen(base_str);
 
+	t1tree = fill_tree_descriptor(&t1, old);
+	t2tree = fill_tree_descriptor(&t2, new);
+
 	/* Enable recursion indefinitely */
 	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
@@ -155,39 +160,41 @@ static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 		if (diff_can_quit_early(opt))
 			break;
 		if (opt->pathspec.nr) {
-			skip_uninteresting(t1, &base, opt);
-			skip_uninteresting(t2, &base, opt);
+			skip_uninteresting(&t1, &base, opt);
+			skip_uninteresting(&t2, &base, opt);
 		}
-		if (!t1->size && !t2->size)
+		if (!t1.size && !t2.size)
 			break;
 
-		cmp = tree_entry_pathcmp(t1, t2);
+		cmp = tree_entry_pathcmp(&t1, &t2);
 
 		/* t1 = t2 */
 		if (cmp == 0) {
 			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
-			    hashcmp(t1->entry.sha1, t2->entry.sha1) ||
-			    (t1->entry.mode != t2->entry.mode))
-				show_path(&base, opt, t1, t2);
+			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
+			    (t1.entry.mode != t2.entry.mode))
+				show_path(&base, opt, &t1, &t2);
 
-			update_tree_entry(t1);
-			update_tree_entry(t2);
+			update_tree_entry(&t1);
+			update_tree_entry(&t2);
 		}
 
 		/* t1 < t2 */
 		else if (cmp < 0) {
-			show_path(&base, opt, t1, /*t2=*/NULL);
-			update_tree_entry(t1);
+			show_path(&base, opt, &t1, /*t2=*/NULL);
+			update_tree_entry(&t1);
 		}
 
 		/* t1 > t2 */
 		else {
-			show_path(&base, opt, /*t1=*/NULL, t2);
-			update_tree_entry(t2);
+			show_path(&base, opt, /*t1=*/NULL, &t2);
+			update_tree_entry(&t2);
 		}
 	}
 
 	strbuf_release(&base);
+	free(t2tree);
+	free(t1tree);
 	return 0;
 }
 
@@ -202,7 +209,7 @@ static inline int diff_might_be_rename(void)
 		!DIFF_FILE_VALID(diff_queued_diff.queue[0]->one);
 }
 
-static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, const char *base, struct diff_options *opt)
+static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
 {
 	struct diff_options diff_opts;
 	struct diff_queue_struct *q = &diff_queued_diff;
@@ -240,7 +247,7 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
 	diff_opts.break_opt = opt->break_opt;
 	diff_opts.rename_score = opt->rename_score;
 	diff_setup_done(&diff_opts);
-	diff_tree(t1, t2, base, &diff_opts);
+	__diff_tree_sha1(old, new, base, &diff_opts);
 	diffcore_std(&diff_opts);
 	free_pathspec(&diff_opts.pathspec);
 
@@ -301,23 +308,12 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
 
 int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
 {
-	void *tree1, *tree2;
-	struct tree_desc t1, t2;
-	unsigned long size1, size2;
 	int retval;
 
-	tree1 = fill_tree_descriptor(&t1, old);
-	tree2 = fill_tree_descriptor(&t2, new);
-	size1 = t1.size;
-	size2 = t2.size;
-	retval = diff_tree(&t1, &t2, base, opt);
-	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename()) {
-		init_tree_desc(&t1, tree1, size1);
-		init_tree_desc(&t2, tree2, size2);
-		try_to_follow_renames(&t1, &t2, base, opt);
-	}
-	free(tree1);
-	free(tree2);
+	retval = __diff_tree_sha1(old, new, base, opt);
+	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
+		try_to_follow_renames(old, new, base, opt);
+
 	return retval;
 }
 
-- 
1.9.rc1.181.g641f458

[PATCH 15/19] tree-diff: no need to call "full" diff_tree_sha1 from show_path()

[Kirill Smelkov]

As described in previous commit, when recursing into sub-trees, we can
use lower-level tree walker, since its interface is now sha1 based.

The change is ok, because diff_tree_sha1() only invokes
__diff_tree_sha1(), and also, if base is empty, try_to_follow_renames().
But base is not empty here, as we have added a path and '/' before
recursing.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 tree-diff.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index f90acf5..aea0297 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -114,8 +114,8 @@ static void show_path(struct strbuf *base, struct diff_options *opt,
 
 	if (recurse) {
 		strbuf_addch(base, '/');
-		diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
-			       t2 ? t2->entry.sha1 : NULL, base->buf, opt);
+		__diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
+				 t2 ? t2->entry.sha1 : NULL, base->buf, opt);
 	}
 
 	strbuf_setlen(base, old_baselen);
-- 
1.9.rc1.181.g641f458

[PATCH v2 16/19] tree-diff: reuse base str(buf) memory on sub-tree recursion

[Kirill Smelkov]

instead of allocating it all the time for every subtree in
__diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all
callee just use it in stacking style, without memory allocations.

This should be faster, and for me this change gives the following
slight speedups for

    git log --raw --no-abbrev --no-renames --format='%H'

                navy.git    linux.git v3.10..v3.11

    before      0.618s      1.903s
    after       0.611s      1.889s
    speedup     1.1%        0.7%

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

Changes since v1:

 - don't need to touch diff.h, as the function we are changing became static.

 tree-diff.c | 36 ++++++++++++++++++------------------
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index aea0297..c76821d 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -115,7 +115,7 @@ static void show_path(struct strbuf *base, struct diff_options *opt,
 	if (recurse) {
 		strbuf_addch(base, '/');
 		__diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
-				 t2 ? t2->entry.sha1 : NULL, base->buf, opt);
+				 t2 ? t2->entry.sha1 : NULL, base, opt);
 	}
 
 	strbuf_setlen(base, old_baselen);
@@ -138,12 +138,10 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
 }
 
 static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
-			    const char *base_str, struct diff_options *opt)
+			    struct strbuf *base, struct diff_options *opt)
 {
 	struct tree_desc t1, t2;
 	void *t1tree, *t2tree;
-	struct strbuf base;
-	int baselen = strlen(base_str);
 
 	t1tree = fill_tree_descriptor(&t1, old);
 	t2tree = fill_tree_descriptor(&t2, new);
@@ -151,17 +149,14 @@ static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 	/* Enable recursion indefinitely */
 	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
-	strbuf_init(&base, PATH_MAX);
-	strbuf_add(&base, base_str, baselen);
-
 	for (;;) {
 		int cmp;
 
 		if (diff_can_quit_early(opt))
 			break;
 		if (opt->pathspec.nr) {
-			skip_uninteresting(&t1, &base, opt);
-			skip_uninteresting(&t2, &base, opt);
+			skip_uninteresting(&t1, base, opt);
+			skip_uninteresting(&t2, base, opt);
 		}
 		if (!t1.size && !t2.size)
 			break;
@@ -173,7 +168,7 @@ static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
 			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
 			    (t1.entry.mode != t2.entry.mode))
-				show_path(&base, opt, &t1, &t2);
+				show_path(base, opt, &t1, &t2);
 
 			update_tree_entry(&t1);
 			update_tree_entry(&t2);
@@ -181,18 +176,17 @@ static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 
 		/* t1 < t2 */
 		else if (cmp < 0) {
-			show_path(&base, opt, &t1, /*t2=*/NULL);
+			show_path(base, opt, &t1, /*t2=*/NULL);
 			update_tree_entry(&t1);
 		}
 
 		/* t1 > t2 */
 		else {
-			show_path(&base, opt, /*t1=*/NULL, &t2);
+			show_path(base, opt, /*t1=*/NULL, &t2);
 			update_tree_entry(&t2);
 		}
 	}
 
-	strbuf_release(&base);
 	free(t2tree);
 	free(t1tree);
 	return 0;
@@ -209,7 +203,7 @@ static inline int diff_might_be_rename(void)
 		!DIFF_FILE_VALID(diff_queued_diff.queue[0]->one);
 }
 
-static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
+static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, struct strbuf *base, struct diff_options *opt)
 {
 	struct diff_options diff_opts;
 	struct diff_queue_struct *q = &diff_queued_diff;
@@ -306,13 +300,19 @@ static void try_to_follow_renames(const unsigned char *old, const unsigned char
 	q->nr = 1;
 }
 
-int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
+int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base_str, struct diff_options *opt)
 {
+	struct strbuf base;
 	int retval;
 
-	retval = __diff_tree_sha1(old, new, base, opt);
-	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
-		try_to_follow_renames(old, new, base, opt);
+	strbuf_init(&base, PATH_MAX);
+	strbuf_addstr(&base, base_str);
+
+	retval = __diff_tree_sha1(old, new, &base, opt);
+	if (!*base_str && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
+		try_to_follow_renames(old, new, &base, opt);
+
+	strbuf_release(&base);
 
 	return retval;
 }
-- 
1.9.rc1.181.g641f458

[PATCH 17/19] Portable alloca for Git

[Kirill Smelkov]

In the next patch we'll have to use alloca() for performance reasons,
but since alloca is non-standardized and is not portable, let's have a
trick with compatibility wrappers:

1. at configure time, determine, do we have working alloca() through
   alloca.h, and define

    #define HAVE_ALLOCA_H

   if yes.

2. in code

    #ifdef HAVE_ALLOCA_H
    # include <alloca.h>
    # define xalloca(size)      (alloca(size))
    # define xalloca_free(p)    do {} while(0)
    #else
    # define xalloca(size)      (xmalloc(size))
    # define xalloca_free(p)    (free(p))
    #endif

   and use it like

   func() {
       p = xalloca(size);
       ...

       xalloca_free(p);
   }

This way, for systems, where alloca is available, we'll have optimal
on-stack allocations with fast executions. On the other hand, on
systems, where alloca is not available, this gracefully fallbacks to
xmalloc/free.

Both autoconf and config.mak.uname configurations were updated. For
autoconf, we are not bothering considering cases, when no alloca.h is
available, but alloca() works some other way - its simply alloca.h is
available and works or not, everything else is deep legacy.

For config.mak.uname, I've tried to make my almost-sure guess for where
alloca() is available, but since I only have access to Linux it is the
only change I can be sure about myself, with relevant to other changed
systems people Cc'ed.

NOTE

SunOS and Windows had explicit -DHAVE_ALLOCA_H in their configurations.
I've changed that to now-common HAVE_ALLOCA_H=YesPlease which should be
correct.

Cc: Brandon Casey <drafnel@gmail.com>
Cc: Marius Storm-Olsen <mstormo@gmail.com>
Cc: Johannes Sixt <j6t@kdbg.org>
Cc: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Cc: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Cc: Gerrit Pape <pape@smarden.org>
Cc: Petr Salinger <Petr.Salinger@seznam.cz>
Cc: Jonathan Nieder <jrnieder@gmail.com>
Cc: Thomas Schwinge <tschwinge@gnu.org>
Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

( new patch )

 Makefile          |  6 ++++++
 config.mak.uname  | 10 ++++++++--
 configure.ac      |  8 ++++++++
 git-compat-util.h |  8 ++++++++
 4 files changed, 30 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index dddaf4f..0334806 100644
--- a/Makefile
+++ b/Makefile
@@ -30,6 +30,8 @@ all::
 # Define LIBPCREDIR=/foo/bar if your libpcre header and library files are in
 # /foo/bar/include and /foo/bar/lib directories.
 #
+# Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.
+#
 # Define NO_CURL if you do not have libcurl installed.  git-http-fetch and
 # git-http-push are not built, and you cannot use http:// and https://
 # transports (neither smart nor dumb).
@@ -1099,6 +1101,10 @@ ifdef USE_LIBPCRE
 	EXTLIBS += -lpcre
 endif
 
+ifdef HAVE_ALLOCA_H
+	BASIC_CFLAGS += -DHAVE_ALLOCA_H
+endif
+
 ifdef NO_CURL
 	BASIC_CFLAGS += -DNO_CURL
 	REMOTE_CURL_PRIMARY =
diff --git a/config.mak.uname b/config.mak.uname
index 7d31fad..71602ee 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -28,6 +28,7 @@ ifeq ($(uname_S),OSF1)
 	NO_NSEC = YesPlease
 endif
 ifeq ($(uname_S),Linux)
+	HAVE_ALLOCA_H = YesPlease
 	NO_STRLCPY = YesPlease
 	NO_MKSTEMPS = YesPlease
 	HAVE_PATHS_H = YesPlease
@@ -35,6 +36,7 @@ ifeq ($(uname_S),Linux)
 	HAVE_DEV_TTY = YesPlease
 endif
 ifeq ($(uname_S),GNU/kFreeBSD)
+	HAVE_ALLOCA_H = YesPlease
 	NO_STRLCPY = YesPlease
 	NO_MKSTEMPS = YesPlease
 	HAVE_PATHS_H = YesPlease
@@ -103,6 +105,7 @@ ifeq ($(uname_S),SunOS)
 	NEEDS_NSL = YesPlease
 	SHELL_PATH = /bin/bash
 	SANE_TOOL_PATH = /usr/xpg6/bin:/usr/xpg4/bin
+	HAVE_ALLOCA_H = YesPlease
 	NO_STRCASESTR = YesPlease
 	NO_MEMMEM = YesPlease
 	NO_MKDTEMP = YesPlease
@@ -146,7 +149,7 @@ ifeq ($(uname_S),SunOS)
 	endif
 	INSTALL = /usr/ucb/install
 	TAR = gtar
-	BASIC_CFLAGS += -D__EXTENSIONS__ -D__sun__ -DHAVE_ALLOCA_H
+	BASIC_CFLAGS += -D__EXTENSIONS__ -D__sun__
 endif
 ifeq ($(uname_O),Cygwin)
 	ifeq ($(shell expr "$(uname_R)" : '1\.[1-6]\.'),4)
@@ -166,6 +169,7 @@ ifeq ($(uname_O),Cygwin)
 	else
 		NO_REGEX = UnfortunatelyYes
 	endif
+	HAVE_ALLOCA_H = YesPlease
 	NEEDS_LIBICONV = YesPlease
 	NO_FAST_WORKING_DIRECTORY = UnfortunatelyYes
 	NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease
@@ -239,6 +243,7 @@ ifeq ($(uname_S),AIX)
 endif
 ifeq ($(uname_S),GNU)
 	# GNU/Hurd
+	HAVE_ALLOCA_H = YesPlease
 	NO_STRLCPY = YesPlease
 	NO_MKSTEMPS = YesPlease
 	HAVE_PATHS_H = YesPlease
@@ -316,6 +321,7 @@ endif
 ifeq ($(uname_S),Windows)
 	GIT_VERSION := $(GIT_VERSION).MSVC
 	pathsep = ;
+	HAVE_ALLOCA_H = YesPlease
 	NO_PREAD = YesPlease
 	NEEDS_CRYPTO_WITH_SSL = YesPlease
 	NO_LIBGEN_H = YesPlease
@@ -363,7 +369,7 @@ ifeq ($(uname_S),Windows)
 	COMPAT_OBJS = compat/msvc.o compat/winansi.o \
 		compat/win32/pthread.o compat/win32/syslog.o \
 		compat/win32/dirent.o
-	COMPAT_CFLAGS = -D__USE_MINGW_ACCESS -DNOGDI -DHAVE_STRING_H -DHAVE_ALLOCA_H -Icompat -Icompat/regex -Icompat/win32 -DSTRIP_EXTENSION=\".exe\"
+	COMPAT_CFLAGS = -D__USE_MINGW_ACCESS -DNOGDI -DHAVE_STRING_H -Icompat -Icompat/regex -Icompat/win32 -DSTRIP_EXTENSION=\".exe\"
 	BASIC_LDFLAGS = -IGNORE:4217 -IGNORE:4049 -NOLOGO -SUBSYSTEM:CONSOLE -NODEFAULTLIB:MSVCRT.lib
 	EXTLIBS = user32.lib advapi32.lib shell32.lib wininet.lib ws2_32.lib
 	PTHREAD_LIBS =
diff --git a/configure.ac b/configure.ac
index 2f43393..0eae704 100644
--- a/configure.ac
+++ b/configure.ac
@@ -272,6 +272,14 @@ AS_HELP_STRING([],           [ARG can be also prefix for libpcre library and hea
 	GIT_CONF_SUBST([LIBPCREDIR])
     fi)
 #
+# Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.
+AC_FUNC_ALLOCA
+case $ac_cv_working_alloca_h in
+    yes)    HAVE_ALLOCA_H=YesPlease;;
+    *)      HAVE_ALLOCA_H='';;
+esac
+GIT_CONF_SUBST([HAVE_ALLOCA_H])
+#
 # Define NO_CURL if you do not have curl installed.  git-http-pull and
 # git-http-push are not built, and you cannot use http:// and https://
 # transports.
diff --git a/git-compat-util.h b/git-compat-util.h
index cbd86c3..63b2b3b 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -526,6 +526,14 @@ extern void release_pack_memory(size_t);
 typedef void (*try_to_free_t)(size_t);
 extern try_to_free_t set_try_to_free_routine(try_to_free_t);
 
+#ifdef HAVE_ALLOCA_H
+# include <alloca.h>
+# define xalloca(size)      (alloca(size))
+# define xalloca_free(p)    do {} while (0)
+#else
+# define xalloca(size)      (xmalloc(size))
+# define xalloca_free(p)    (free(p))
+#endif
 extern char *xstrdup(const char *str);
 extern void *xmalloc(size_t size);
 extern void *xmallocz(size_t size);
-- 
1.9.rc1.181.g641f458

[PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Kirill Smelkov]

Previously diff_tree(), which is now named __diff_tree_sha1(), was
generating diff_filepair(s) for two trees t1 and t2, and that was
usually used for a commit as t1=HEAD~, and t2=HEAD - i.e. to see changes
a commit introduces.

In Git, however, we have fundamentally built flexibility in that a
commit can have many parents - 1 for a plain commit, 2 for a simple merge,
but also more than 2 for merging several heads at once.

For merges there is a so called combine-diff, which shows diff, a merge
introduces by itself, omitting changes done by any parent. That works
through first finding paths, that are different to all parents, and then
showing generalized diff, with separate columns for +/- for each parent.
The code lives in combine-diff.c .

There is an impedance mismatch, however, in that a commit could
generally have any number of parents, and that while diffing trees, we
divide cases for 2-tree diffs and more-than-2-tree diffs. I mean there
is no special casing for multiple parents commits in e.g.
revision-walker .

That impedance mismatch *hurts* *performance* *badly* for generating
combined diffs - in "combine-diff: optimize combine_diff_path
sets intersection" I've already removed some slowness from it, but from
the timings provided there, it could be seen, that combined diffs still
cost more than an order of magnitude more cpu time, compared to diff for
usual commits, and that would only be an optimistic estimate, if we take
into account that for e.g. linux.git there is only one merge for several
dozens of plain commits.

That slowness comes from the fact that currently, while generating
combined diff, a lot of time is spent computing diff(commit,commit^2)
just to only then intersect that huge diff to almost small set of files
from diff(commit,commit^1).

That's because at present, to compute combine-diff, for first finding
paths, that "every parent touches", we use the following combine-diff
property/definition:

D(A,P1...Pn) = D(A,P1) ^ ... ^ D(A,Pn)      (w.r.t. paths)

where

D(A,P1...Pn) is combined diff between commit A, and parents Pi

and

D(A,Pi) is usual two-tree diff Pi..A

So if any of that D(A,Pi) is huge, tracting 1 n-parent combine-diff as n
1-parent diffs and intersecting results will be slow.

And usually, for linux.git and other topic-based workflows, that
D(A,P2) is huge, because, if merge-base of A and P2, is several dozens
of merges (from A, via first parent) below, that D(A,P2) will be diffing
sum of merges from several subsystems to 1 subsystem.

The solution is to avoid computing n 1-parent diffs, and to find
changed-to-all-parents paths via scanning A's and all Pi's trees
simultaneously, at each step comparing their entries, and based on that
comparison, populate paths result, and deduce we could *skip*
*recursing* into subdirectories, if at least for 1 parent, sha1 of that
dir tree is the same as in A. That would save us from doing significant
amount of needless work.

Such approach is very similar to what diff_tree() does, only there we
deal with scanning only 2 trees simultaneously, and for n+1 tree, the
logic is a bit more complex:

D(A,X1...Xn) calculation scheme
-------------------------------

D(A,X1...Xn) = D(A,X1) ^ ... ^ D(A,Xn)       (regarding resulting paths set)

     D(A,Xj)         - diff between A..Xj
     D(A,X1...Xn)    - combined diff from A to parents X1,...,Xn

We start from all trees, which are sorted, and compare their entries in
lock-step:

      A     X1       Xn
      -     -        -
     |a|   |x1|     |xn|
     |-|   |--| ... |--|      i = argmin(x1...xn)
     | |   |  |     |  |
     |-|   |--|     |--|
     |.|   |. |     |. |
      .     .        .
      .     .        .

at any time there could be 3 cases:

     1)  a < xi;
     2)  a > xi;
     3)  a = xi.

Schematic deduction of what every case means, and what to do, follows:

1)  a < xi  ->  ∀j a ∉ Xj  ->  "+a" ∈ D(A,Xj)  ->  D += "+a";  a↓

2)  a > xi

    2.1) ∃j: xj > xi  ->  "-xi" ∉ D(A,Xj)  ->  D += ø;  ∀ xk=xi  xk↓
    2.2) ∀j  xj = xi  ->  xj ∉ A  ->  "-xj" ∈ D(A,Xj)  ->  D += "-xi";  ∀j xj↓

3)  a = xi

    3.1) ∃j: xj > xi  ->  "+a" ∈ D(A,Xj)  ->  only xk=xi remains to investigate
    3.2) xj = xi  ->  investigate δ(a,xj)
     |
     |
     v

    3.1+3.2) looking at δ(a,xk) ∀k: xk=xi - if all != ø  ->

                      ⎧δ(a,xk)  - if xk=xi
             ->  D += ⎨
                      ⎩"+a"     - if xk>xi

    in any case a↓  ∀ xk=xi  xk↓

~

For comparison, here is how diff_tree() works:

D(A,B) calculation scheme
-------------------------

    A     B
    -     -
   |a|   |b|    a < b   ->  a ∉ B   ->   D(A,B) +=  +a    a↓
   |-|   |-|    a > b   ->  b ∉ A   ->   D(A,B) +=  -b    b↓
   | |   | |    a = b   ->  investigate δ(a,b)            a↓ b↓
   |-|   |-|
   |.|   |.|
    .     .
    .     .

~~~~~~~~

This patch generalizes diff tree-walker to work with arbitrary number of
parents as described above - i.e. now there is a resulting tree t, and
some parents trees tp[i] i=[0..nparent). The generalization builds on
the fact that usual diff

D(A,B)

is by definition the same as combined diff

D(A,[B]),

so if we could rework the code for common case and make it be not slower
for nparent=1 case, usual diff(t1,t2) generation will not be slower, and
multiparent diff tree-walker would greatly benefit generating
combine-diff.

What we do is as follows:

1) diff tree-walker __diff_tree_sha1() is internally reworked to be
   a paths generator (new name diff_tree_paths()), with each generated path
   being `struct combine_diff_path` with info for path, new sha1,mode and for
   every parent which sha1,mode it was in it.

2) From that info, we can still generate usual diff queue with
   struct diff_filepairs, via "exporting" generated
   combine_diff_path, if we know we run for nparent=1 case.
   (see emit_diff() which is now named emit_diff_first_parent_only())

3) In order for diff_can_quit_early(), which checks

       DIFF_OPT_TST(opt, HAS_CHANGES))

   to work, that exporting have to be happening not in bulk, but
   incrementally, one diff path at a time.

   For such consumers, there is a new callback in diff_options
   introduced:

       ->pathchange(opt, struct combine_diff_path *)

   which, if set to !NULL, is called for every generated path.

   (see new compat __diff_tree_sha1() wrapper around new paths
    generator for setup)

4) The paths generation itself, is reworked from previous
   __diff_tree_sha1() code according to "D(A,X1...Xn) calculation
   scheme" provided above:

   On the start we allocate [nparent] arrays in place what was
   earlier just for one parent tree.

   then we just generalize loops, and comparison according to the
   algorithm.

Some notes(*):

1) alloca(), for small arrays, is used for "runs not slower for
   nparent=1 case than before" goal - if we change it to xmalloc()/free()
   the timings get ~1% worse. For alloca() we use just-introduced
   xalloca/xalloca_free compatibility wrappers, so it should not be a
   portability problem.

2) For every parent tree, we need to keep a tag, whether entry from that
   parent equals to entry from minimal parent. For performance reasons I'm
   keeping that tag in entry's mode field in unused bit - see S_IFXMIN_NEQ.
   Not doing so, we'd need to alloca another [nparent] array, which hurts
   performance.

3) For emitted paths, memory could be reused, if we know the path was
   processed via callback and will not be needed later. We use efficient
   hand-made realloc-style __path_appendnew(), that saves us from ~1-1.5%
   of potential additional slowdown.

4) goto(s) are used in several places, as the code executes a little bit
   faster with lowered register pressure.

Also

- we should now check for FIND_COPIES_HARDER not only when two entries
  names are the same, and their hashes are equal, but also for a case,
  when a path was removed from some of all parents having it.

  The reason is, if we don't, that path won't be emitted at all (see
  "a > xi" case), and we'll just skip it, and FIND_COPIES_HARDER wants
  all paths - with diff or without - to be emitted, to be later analyzed
  for being copies sources.

  The new check is only necessary for nparent >1, as for nparent=1 case
  xmin_eqtotal always =1 =nparent, and a path is always added to diff as
  removal.

~~~~~~~~

Timings for

    # without -c, i.e. testing only nparent=1 case
    `git log --raw --no-abbrev --no-renames`

before and after the patch are as follows:

                navy.git        linux.git v3.10..v3.11

    before      0.611s          1.889s
    after       0.619s          1.907s
    slowdown    1.3%            0.9%

This timings show we did no harm to usual diff(tree1,tree2) generation.
From the table we can see that we actually did ~1% slowdown, but I think
I've "earned" that 1% in the previous patch ("tree-diff: reuse base
str(buf) memory on sub-tree recursion", HEAD~~) so for nparent=1 case,
net timings stays approximately the same.

The output also stayed the same.

(*) If we revert 1)-4) to more usual techniques, for nparent=1 case,
    we'll get ~2-2.5% of additional slowdown, which I've tried to avoid, as
   "do no harm for nparent=1 case" rule.

For linux.git, combined diff will run an order of magnitude faster and
appropriate timings will be provided in the next commit, as we'll be
taking advantage of the new diff tree-walker for combined-diff
generation there.

P.S. and combined diff is not some exotic/for-play-only stuff - for
example for a program I write to represent Git archives as readonly
filesystem, there is initial scan with

    `git log --reverse --raw --no-abbrev --no-renames -c`

to extract log of what was created/changed when, as a result building a
map

    {}  sha1    ->  in which commit (and date) a content was added

that `-c` means also show combined diff for merges, and without them, if
a merge is non-trivial (merges changes from two parents with both having
separate changes to a file), or an evil one, the map will not be full,
i.e. some valid sha1 would be absent from it.

That case was my initial motivation for combined diffs speedup.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

Changes since v1:

- fixed last-minute thinko/bug last time introduced on my side (sorry) with
  opt->pathchange manipulation in __diff_tree_sha1() - we were forgetting to
  restore opt->pathchange, which led to incorrect log -c (merges _and_ plain
  diff-tree) output;

  This time, I've verified several times, log output stays really the same.

- direct use of alloca() changed to portability wrappers xalloca/xalloca_free
  which gracefully degrade to xmalloc/free on systems, where alloca is not
  available (see new patch 17).

- "i = 0; do { ... } while (++i < nparent)" is back to usual looping
  "for (i = 0; i < nparent; ++)", as I've re-measured timings and the
  difference is negligible.

  ( Initially, when I was fighting for every cycle it made sense, but real
    no-slowdown turned out to be related to avoiding mallocs, load trees in correct
    order and reducing register pressure. )

- S_IFXMIN_NEQ definition moved out to cache.h, to have all modes registry in one place;


- p0 -> first_parent; corrected comments about how emit_diff_first_parent_only
  behaves;


not changed:

- low-level helpers are still named with "__" prefix as, imho, that is the best
  convention to name such helpers, without sacrificing signal/noise ratio. All
  of them are now static though.

 cache.h     |  15 ++
 diff.c      |   1 +
 diff.h      |  10 ++
 tree-diff.c | 508 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------
 4 files changed, 471 insertions(+), 63 deletions(-)

diff --git a/cache.h b/cache.h
index dc040fb..e7f5a0c 100644
--- a/cache.h
+++ b/cache.h
@@ -75,6 +75,21 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);
 #define S_ISGITLINK(m)	(((m) & S_IFMT) == S_IFGITLINK)
 
 /*
+ * Some mode bits are also used internally for computations.
+ *
+ * They *must* not overlap with any valid modes, and they *must* not be emitted
+ * to outside world - i.e. appear on disk or network. In other words, it's just
+ * temporary fields, which we internally use, but they have to stay in-house.
+ *
+ * ( such approach is valid, as standard S_IF* fits into 16 bits, and in Git
+ *   codebase mode is `unsigned int` which is assumed to be at least 32 bits )
+ */
+
+/* used internally in tree-diff */
+#define S_DIFFTREE_IFXMIN_NEQ	0x80000000
+
+
+/*
  * Intensive research over the course of many years has shown that
  * port 9418 is totally unused by anything else. Or
  *
diff --git a/diff.c b/diff.c
index 8e4a6a9..cda4aa8 100644
--- a/diff.c
+++ b/diff.c
@@ -3216,6 +3216,7 @@ void diff_setup(struct diff_options *options)
 	options->context = diff_context_default;
 	DIFF_OPT_SET(options, RENAME_EMPTY);
 
+	/* pathchange left =NULL by default */
 	options->change = diff_change;
 	options->add_remove = diff_addremove;
 	options->use_color = diff_use_color_default;
diff --git a/diff.h b/diff.h
index 5d7b9f7..732dca7 100644
--- a/diff.h
+++ b/diff.h
@@ -15,6 +15,10 @@ struct diff_filespec;
 struct userdiff_driver;
 struct sha1_array;
 struct commit;
+struct combine_diff_path;
+
+typedef int (*pathchange_fn_t)(struct diff_options *options,
+		 struct combine_diff_path *path);
 
 typedef void (*change_fn_t)(struct diff_options *options,
 		 unsigned old_mode, unsigned new_mode,
@@ -157,6 +161,7 @@ struct diff_options {
 	int close_file;
 
 	struct pathspec pathspec;
+	pathchange_fn_t pathchange;
 	change_fn_t change;
 	add_remove_fn_t add_remove;
 	diff_format_fn_t format_callback;
@@ -189,6 +194,11 @@ const char *diff_line_prefix(struct diff_options *);
 
 extern const char mime_boundary_leader[];
 
+extern
+struct combine_diff_path *diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parent_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt);
 extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 			  const char *base, struct diff_options *opt);
 extern int diff_root_tree_sha1(const unsigned char *new, const char *base,
diff --git a/tree-diff.c b/tree-diff.c
index c76821d..b682d77 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -7,6 +7,22 @@
 #include "tree.h"
 
 /*
+ * internal mode marker, saying a tree entry != entry of tp[imin]
+ * (see __diff_tree_paths for what it means there)
+ *
+ * we will update/use/emit entry for diff only with it unset.
+ */
+#define S_IFXMIN_NEQ	S_DIFFTREE_IFXMIN_NEQ
+
+
+static struct combine_diff_path *__diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parents_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt);
+static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
+			    struct strbuf *base, struct diff_options *opt);
+
+/*
  * Compare two tree entries, taking into account only path/S_ISDIR(mode),
  * but not their sha1's.
  *
@@ -33,72 +49,153 @@ static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 }
 
 
-/* convert path, t1/t2 -> opt->diff_*() callbacks */
-static void emit_diff(struct diff_options *opt, struct strbuf *path,
-		      struct tree_desc *t1, struct tree_desc *t2)
+/*
+ * convert path -> opt->diff_*() callbacks
+ *
+ * emits diff to first parent only, and tells diff tree-walker that we are done
+ * with p and it can be freed.
+ */
+static int emit_diff_first_parent_only(struct diff_options *opt, struct combine_diff_path *p)
 {
-	unsigned int mode1 = t1 ? t1->entry.mode : 0;
-	unsigned int mode2 = t2 ? t2->entry.mode : 0;
-
-	if (mode1 && mode2) {
-		opt->change(opt, mode1, mode2, t1->entry.sha1, t2->entry.sha1,
-			1, 1, path->buf, 0, 0);
+	struct combine_diff_parent *p0 = &p->parent[0];
+	if (p->mode && p0->mode) {
+		opt->change(opt, p0->mode, p->mode, p0->sha1, p->sha1,
+			1, 1, p->path, 0, 0);
 	}
 	else {
 		const unsigned char *sha1;
 		unsigned int mode;
 		int addremove;
 
-		if (mode2) {
+		if (p->mode) {
 			addremove = '+';
-			sha1 = t2->entry.sha1;
-			mode = mode2;
+			sha1 = p->sha1;
+			mode = p->mode;
 		}
 		else {
 			addremove = '-';
-			sha1 = t1->entry.sha1;
-			mode = mode1;
+			sha1 = p0->sha1;
+			mode = p0->mode;
 		}
 
-		opt->add_remove(opt, addremove, mode, sha1, 1, path->buf, 0);
+		opt->add_remove(opt, addremove, mode, sha1, 1, p->path, 0);
 	}
+
+	return 0;	/* we are done with p */
 }
 
 
-/* new path should be added to diff
+/*
+ * Make a new combine_diff_path from path/mode/sha1
+ * and append it to paths list tail.
+ *
+ * Memory for created elements could be reused:
+ *
+ *	- if last->next == NULL, the memory is allocated;
+ *
+ *	- if last->next != NULL, it is assumed that p=last->next was returned
+ *	  earlier by this function, and p->next was *not* modified.
+ *	  The memory is then reused from p.
+ *
+ * so for clients,
+ *
+ * - if you do need to keep the element
+ *
+ *	p = __path_appendnew(p, ...);
+ *	process(p);
+ *	p->next = NULL;
+ *
+ * - if you don't need to keep the element after processing
+ *
+ *	pprev = p;
+ *	p = __path_appendnew(p, ...);
+ *	process(p);
+ *	p = pprev;
+ *	; don't forget to free tail->next in the end
+ *
+ * p->parent[] remains uninitialized.
+ */
+static struct combine_diff_path *__path_appendnew(struct combine_diff_path *last,
+	int nparent, const struct strbuf *base, const char *path, int pathlen,
+	unsigned mode, const unsigned char *sha1)
+{
+	struct combine_diff_path *p;
+	int len = base->len + pathlen;
+	int alloclen = combine_diff_path_size(nparent, len);
+
+	/* if last->next is !NULL - it is a pre-allocated memory, we can reuse */
+	p = last->next;
+	if (p && (alloclen > (intptr_t)p->next)) {
+		free(p);
+		p = NULL;
+	}
+
+	if (!p) {
+		p = xmalloc(alloclen);
+
+		/*
+		 * until we go to it next round, .next holds how many bytes we
+		 * allocated (for faster realloc - we don't need copying old data).
+		 */
+		p->next = (struct combine_diff_path *)(intptr_t)alloclen;
+	}
+
+	last->next = p;
+
+	p->path = (char *)&(p->parent[nparent]);
+	memcpy(p->path, base->buf, base->len);
+	memcpy(p->path + base->len, path, pathlen);
+	p->path[len] = 0;
+	p->mode = mode;
+	hashcpy(p->sha1, sha1 ? sha1 : null_sha1);
+
+	return p;
+}
+
+/*
+ * new path should be added to combine diff
  *
  * 3 cases on how/when it should be called and behaves:
  *
- *	!t1,  t2	-> path added, parent lacks it
- *	 t1, !t2	-> path removed from parent
- *	 t1,  t2	-> path modified
+ *	 t, !tp		-> path added, all parents lack it
+ *	!t,  tp		-> path removed from all parents
+ *	 t,  tp		-> path modified/added
+ *			   (M for tp[i]=tp[imin], A otherwise)
  */
-static void show_path(struct strbuf *base, struct diff_options *opt,
-		      struct tree_desc *t1, struct tree_desc *t2)
+static struct combine_diff_path *emit_path(struct combine_diff_path *p,
+	struct strbuf *base, struct diff_options *opt, int nparent,
+	struct tree_desc *t, struct tree_desc *tp,
+	int imin)
 {
 	unsigned mode;
 	const char *path;
+	const unsigned char *sha1;
 	int pathlen;
 	int old_baselen = base->len;
-	int isdir, recurse = 0, emitthis = 1;
+	int i, isdir, recurse = 0, emitthis = 1;
 
 	/* at least something has to be valid */
-	assert(t1 || t2);
+	assert(t || tp);
 
-	if (t2) {
+	if (t) {
 		/* path present in resulting tree */
-		tree_entry_extract(t2, &path, &mode);
-		pathlen = tree_entry_len(&t2->entry);
+		sha1 = tree_entry_extract(t, &path, &mode);
+		pathlen = tree_entry_len(&t->entry);
 		isdir = S_ISDIR(mode);
 	}
 	else {
-		/* a path was removed - take path from parent. Also take
-		 * mode from parent, to decide on recursion.
+		/*
+		 * a path was removed - take path from imin parent. Also take
+		 * mode from that parent, to decide on recursion(1).
+		 *
+		 * 1) all modes for tp[k]=tp[imin] should be the same wrt
+		 *    S_ISDIR, thanks to base_name_compare().
 		 */
-		tree_entry_extract(t1, &path, &mode);
-		pathlen = tree_entry_len(&t1->entry);
+		tree_entry_extract(&tp[imin], &path, &mode);
+		pathlen = tree_entry_len(&tp[imin].entry);
 
 		isdir = S_ISDIR(mode);
+		sha1 = NULL;
 		mode = 0;
 	}
 
@@ -107,18 +204,81 @@ static void show_path(struct strbuf *base, struct diff_options *opt,
 		emitthis = DIFF_OPT_TST(opt, TREE_IN_RECURSIVE);
 	}
 
-	strbuf_add(base, path, pathlen);
+	if (emitthis) {
+		int keep;
+		struct combine_diff_path *pprev = p;
+		p = __path_appendnew(p, nparent, base, path, pathlen, mode, sha1);
 
-	if (emitthis)
-		emit_diff(opt, base, t1, t2);
+		for (i = 0; i < nparent; ++i) {
+			/*
+			 * tp[i] is valid, if present and if tp[i]==tp[imin] -
+			 * otherwise, we should ignore it.
+			 */
+			int tpi_valid = tp && !(tp[i].entry.mode & S_IFXMIN_NEQ);
+
+			const unsigned char *sha1_i;
+			unsigned mode_i;
+
+			p->parent[i].status =
+				!t ? DIFF_STATUS_DELETED :
+					tpi_valid ?
+						DIFF_STATUS_MODIFIED :
+						DIFF_STATUS_ADDED;
+
+			if (tpi_valid) {
+				sha1_i = tp[i].entry.sha1;
+				mode_i = tp[i].entry.mode;
+			}
+			else {
+				sha1_i = NULL;
+				mode_i = 0;
+			}
+
+			p->parent[i].mode = mode_i;
+			hashcpy(p->parent[i].sha1, sha1_i ? sha1_i : null_sha1);
+		}
+
+		keep = 1;
+		if (opt->pathchange)
+			keep = opt->pathchange(opt, p);
+
+		/*
+		 * If a path was filtered or consumed - we don't need to add it
+		 * to the list and can reuse its memory, leaving it as
+		 * pre-allocated element on the tail.
+		 *
+		 * On the other hand, if path needs to be kept, we need to
+		 * correct its .next to NULL, as it was pre-initialized to how
+		 * much memory was allocated.
+		 *
+		 * see __path_appendnew() for details.
+		 */
+		if (!keep)
+			p = pprev;
+		else
+			p->next = NULL;
+	}
 
 	if (recurse) {
+		const unsigned char **parents_sha1;
+
+		parents_sha1 = xalloca(nparent * sizeof(parents_sha1[0]));
+		for (i = 0; i < nparent; ++i) {
+			/* same rule as in emitthis */
+			int tpi_valid = tp && !(tp[i].entry.mode & S_IFXMIN_NEQ);
+
+			parents_sha1[i] = tpi_valid ? tp[i].entry.sha1
+						    : NULL;
+		}
+
+		strbuf_add(base, path, pathlen);
 		strbuf_addch(base, '/');
-		__diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
-				 t2 ? t2->entry.sha1 : NULL, base, opt);
+		p = __diff_tree_paths(p, sha1, parents_sha1, nparent, base, opt);
+		xalloca_free(parents_sha1);
 	}
 
 	strbuf_setlen(base, old_baselen);
+	return p;
 }
 
 static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
@@ -137,59 +297,260 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
 	}
 }
 
-static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
-			    struct strbuf *base, struct diff_options *opt)
+
+/*
+ * generate paths for combined diff D(sha1,parents_sha1[])
+ *
+ * Resulting paths are appended to combine_diff_path linked list, and also, are
+ * emitted on the go via opt->pathchange() callback, so it is possible to
+ * process the result as batch or incrementally.
+ *
+ * The paths are generated scanning new tree and all parents trees
+ * simultaneously, similarly to what diff_tree() was doing for 2 trees.
+ * The theory behind such scan is as follows:
+ *
+ *
+ * D(A,X1...Xn) calculation scheme
+ * -------------------------------
+ *
+ * D(A,X1...Xn) = D(A,X1) ^ ... ^ D(A,Xn)	(regarding resulting paths set)
+ *
+ *	D(A,Xj)		- diff between A..Xj
+ *	D(A,X1...Xn)	- combined diff from A to parents X1,...,Xn
+ *
+ *
+ * We start from all trees, which are sorted, and compare their entries in
+ * lock-step:
+ *
+ *	 A     X1       Xn
+ *	 -     -        -
+ *	|a|   |x1|     |xn|
+ *	|-|   |--| ... |--|      i = argmin(x1...xn)
+ *	| |   |  |     |  |
+ *	|-|   |--|     |--|
+ *	|.|   |. |     |. |
+ *	 .     .        .
+ *	 .     .        .
+ *
+ * at any time there could be 3 cases:
+ *
+ *	1)  a < xi;
+ *	2)  a > xi;
+ *	3)  a = xi.
+ *
+ * Schematic deduction of what every case means, and what to do, follows:
+ *
+ * 1)  a < xi  ->  ∀j a ∉ Xj  ->  "+a" ∈ D(A,Xj)  ->  D += "+a";  a↓
+ *
+ * 2)  a > xi
+ *
+ *     2.1) ∃j: xj > xi  ->  "-xi" ∉ D(A,Xj)  ->  D += ø;  ∀ xk=xi  xk↓
+ *     2.2) ∀j  xj = xi  ->  xj ∉ A  ->  "-xj" ∈ D(A,Xj)  ->  D += "-xi";  ∀j xj↓
+ *
+ * 3)  a = xi
+ *
+ *     3.1) ∃j: xj > xi  ->  "+a" ∈ D(A,Xj)  ->  only xk=xi remains to investigate
+ *     3.2) xj = xi  ->  investigate δ(a,xj)
+ *      |
+ *      |
+ *      v
+ *
+ *     3.1+3.2) looking at δ(a,xk) ∀k: xk=xi - if all != ø  ->
+ *
+ *                       ⎧δ(a,xk)  - if xk=xi
+ *              ->  D += ⎨
+ *                       ⎩"+a"     - if xk>xi
+ *
+ *
+ *     in any case a↓  ∀ xk=xi  xk↓
+ *
+ *
+ * ~~~~~~~~
+ *
+ * NOTE
+ *
+ *	Usual diff D(A,B) is by definition the same as combined diff D(A,[B]),
+ *	so this diff paths generator can, and is used, for plain diffs
+ *	generation too.
+ *
+ *	Please keep attention to the common D(A,[B]) case when working on the
+ *	code, in order not to slow it down.
+ *
+ * NOTE
+ *	nparent must be > 0.
+ */
+
+
+/* ∀ xk=xi  xk↓ */
+static inline void update_tp_entries(struct tree_desc *tp, int nparent)
 {
-	struct tree_desc t1, t2;
-	void *t1tree, *t2tree;
+	int i;
+	for (i = 0; i < nparent; ++i)
+		if (!(tp[i].entry.mode & S_IFXMIN_NEQ))
+			update_tree_entry(&tp[i]);
+}
+
+static struct combine_diff_path *__diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parents_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt)
+{
+	struct tree_desc t, *tp;
+	void *ttree, **tptree;
+	int i;
+
+	tp     = xalloca(nparent * sizeof(tp[0]));
+	tptree = xalloca(nparent * sizeof(tptree[0]));
 
-	t1tree = fill_tree_descriptor(&t1, old);
-	t2tree = fill_tree_descriptor(&t2, new);
+	/*
+	 * load parents first, as they are probably already cached.
+	 *
+	 * ( log_tree_diff() parses commit->parent before calling here via
+	 *   diff_tree_sha1(parent, commit) )
+	 */
+	for (i = 0; i < nparent; ++i)
+		tptree[i] = fill_tree_descriptor(&tp[i], parents_sha1[i]);
+	ttree = fill_tree_descriptor(&t, sha1);
 
 	/* Enable recursion indefinitely */
 	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
 	for (;;) {
-		int cmp;
+		int imin, cmp;
 
 		if (diff_can_quit_early(opt))
 			break;
+
 		if (opt->pathspec.nr) {
-			skip_uninteresting(&t1, base, opt);
-			skip_uninteresting(&t2, base, opt);
+			skip_uninteresting(&t, base, opt);
+			for (i = 0; i < nparent; i++)
+				skip_uninteresting(&tp[i], base, opt);
 		}
-		if (!t1.size && !t2.size)
-			break;
 
-		cmp = tree_entry_pathcmp(&t1, &t2);
+		/* comparing is finished when all trees are done */
+		if (!t.size) {
+			int done = 1;
+			for (i = 0; i < nparent; ++i)
+				if (tp[i].size) {
+					done = 0;
+					break;
+				}
+			if (done)
+				break;
+		}
+
+		/*
+		 * lookup imin = argmin(x1...xn),
+		 * mark entries whether they =tp[imin] along the way
+		 */
+		imin = 0;
+		tp[0].entry.mode &= ~S_IFXMIN_NEQ;
+
+		for (i = 1; i < nparent; ++i) {
+			cmp = tree_entry_pathcmp(&tp[i], &tp[imin]);
+			if (cmp < 0) {
+				imin = i;
+				tp[i].entry.mode &= ~S_IFXMIN_NEQ;
+			}
+			else if (cmp == 0) {
+				tp[i].entry.mode &= ~S_IFXMIN_NEQ;
+			}
+			else {
+				tp[i].entry.mode |= S_IFXMIN_NEQ;
+			}
+		}
+
+		/* fixup markings for entries before imin */
+		for (i = 0; i < imin; ++i)
+			tp[i].entry.mode |= S_IFXMIN_NEQ;	/* x[i] > x[imin] */
+
+
+
+		/* compare a vs x[imin] */
+		cmp = tree_entry_pathcmp(&t, &tp[imin]);
 
-		/* t1 = t2 */
+		/* a = xi */
 		if (cmp == 0) {
-			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
-			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
-			    (t1.entry.mode != t2.entry.mode))
-				show_path(base, opt, &t1, &t2);
+			/* are either xk > xi or diff(a,xk) != ø ? */
+			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
+				for (i = 0; i < nparent; ++i) {
+					/* x[i] > x[imin] */
+					if (tp[i].entry.mode & S_IFXMIN_NEQ)
+						continue;
 
-			update_tree_entry(&t1);
-			update_tree_entry(&t2);
+					/* diff(a,xk) != ø */
+					if (hashcmp(t.entry.sha1, tp[i].entry.sha1) ||
+					    (t.entry.mode != tp[i].entry.mode))
+						continue;
+
+					goto skip_emit_t_tp;
+				}
+			}
+
+			/* D += {δ(a,xk) if xk=xi;  "+a" if xk > xi} */
+			p = emit_path(p, base, opt, nparent,
+					&t, tp, imin);
+
+		skip_emit_t_tp:
+			/* a↓,  ∀ xk=ximin  xk↓ */
+			update_tree_entry(&t);
+			update_tp_entries(tp, nparent);
 		}
 
-		/* t1 < t2 */
+		/* a < xi */
 		else if (cmp < 0) {
-			show_path(base, opt, &t1, /*t2=*/NULL);
-			update_tree_entry(&t1);
+			/* D += "+a" */
+			p = emit_path(p, base, opt, nparent,
+					&t, /*tp=*/NULL, -1);
+
+			/* a↓ */
+			update_tree_entry(&t);
 		}
 
-		/* t1 > t2 */
+		/* a > xi */
 		else {
-			show_path(base, opt, /*t1=*/NULL, &t2);
-			update_tree_entry(&t2);
+			/* ∀j xj=ximin -> D += "-xi" */
+			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
+				for (i = 0; i < nparent; ++i)
+					if (tp[i].entry.mode & S_IFXMIN_NEQ)
+						goto skip_emit_tp;
+			}
+
+			p = emit_path(p, base, opt, nparent,
+					/*t=*/NULL, tp, imin);
+
+		skip_emit_tp:
+			/* ∀ xk=ximin  xk↓ */
+			update_tp_entries(tp, nparent);
 		}
 	}
 
-	free(t2tree);
-	free(t1tree);
-	return 0;
+	free(ttree);
+	for (i = nparent-1; i >= 0; i--)
+		free(tptree[i]);
+	xalloca_free(tptree);
+	xalloca_free(tp);
+
+	return p;
+}
+
+struct combine_diff_path *diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parents_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt)
+{
+	p = __diff_tree_paths(p, sha1, parents_sha1, nparent, base, opt);
+
+	/*
+	 * free pre-allocated last element, if any
+	 * (see __path_appendnew() for details about why)
+	 */
+	if (p->next) {
+		free(p->next);
+		p->next = NULL;
+	}
+
+	return p;
 }
 
 /*
@@ -300,6 +661,27 @@ static void try_to_follow_renames(const unsigned char *old, const unsigned char
 	q->nr = 1;
 }
 
+static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
+			    struct strbuf *base, struct diff_options *opt)
+{
+	struct combine_diff_path phead, *p;
+	const unsigned char *parents_sha1[1] = {old};
+	pathchange_fn_t pathchange_old = opt->pathchange;
+
+	phead.next = NULL;
+	opt->pathchange = emit_diff_first_parent_only;
+	diff_tree_paths(&phead, new, parents_sha1, 1, base, opt);
+
+	for (p = phead.next; p;) {
+		struct combine_diff_path *pprev = p;
+		p = p->next;
+		free(pprev);
+	}
+
+	opt->pathchange = pathchange_old;
+	return 0;
+}
+
 int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base_str, struct diff_options *opt)
 {
 	struct strbuf base;
-- 
1.9.rc1.181.g641f458

[PATCH 19/19] combine-diff: speed it up, by using multiparent diff tree-walker directly

[Kirill Smelkov]

As was recently shown in "combine-diff: optimize
combine_diff_path sets intersection", combine-diff runs very slowly. In
that commit we optimized paths sets intersection, but that accounted
only for ~ 25% of the slowness, and as my tracing showed, for linux.git
v3.10..v3.11, for merges a lot of time is spent computing
diff(commit,commit^2) just to only then intersect that huge diff to
almost small set of files from diff(commit,commit^1).

In previous commit, we described the problem in more details, and
reworked the diff tree-walker to be general one - i.e. to work in
multiple parent case too. Now is the time to take advantage of it for
finding paths for combine diff.

The implementation is straightforward - if we know, we can get generated
diff paths directly, and at present that means no diff filtering or
rename/copy detection was requested(*), we can call multiparent tree-walker
directly and get ready paths.

(*) because e.g. at present, all diffcore transformations work on
    diff_filepair queues, but in the future, that limitation can be
    lifted, if filters would operate directly on combine_diff_paths.

Timings for `git log --raw --no-abbrev --no-renames` without `-c` ("git log")
and with `-c` ("git log -c") and with `-c --merges` ("git log -c --merges")
before and after the patch are as follows:

                linux.git v3.10..v3.11

            log     log -c     log -c --merges

    before  1.9s    16.4s      15.2s
    after   1.9s     2.4s       1.1s

The result stayed the same.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

( re-posting without change )

 combine-diff.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 diff.c         |  1 +
 2 files changed, 84 insertions(+), 5 deletions(-)

diff --git a/combine-diff.c b/combine-diff.c
index 1732dfd..12764fb 100644
--- a/combine-diff.c
+++ b/combine-diff.c
@@ -1303,7 +1303,7 @@ static const char *path_path(void *obj)
 
 
 /* find set of paths that every parent touches */
-static struct combine_diff_path *find_paths(const unsigned char *sha1,
+static struct combine_diff_path *find_paths_generic(const unsigned char *sha1,
 	const struct sha1_array *parents, struct diff_options *opt)
 {
 	struct combine_diff_path *paths = NULL;
@@ -1316,6 +1316,7 @@ static struct combine_diff_path *find_paths(const unsigned char *sha1,
 	/* tell diff_tree to emit paths in sorted (=tree) order */
 	opt->orderfile = NULL;
 
+	/* D(A,P1...Pn) = D(A,P1) ^ ... ^ D(A,Pn)  (wrt paths) */
 	for (i = 0; i < num_parent; i++) {
 		/*
 		 * show stat against the first parent even when doing
@@ -1346,6 +1347,35 @@ static struct combine_diff_path *find_paths(const unsigned char *sha1,
 }
 
 
+/*
+ * find set of paths that everybody touches, assuming diff is run without
+ * rename/copy detection, etc, comparing all trees simultaneously (= faster).
+ */
+static struct combine_diff_path *find_paths_multitree(
+	const unsigned char *sha1, const struct sha1_array *parents,
+	struct diff_options *opt)
+{
+	int i, nparent = parents->nr;
+	const unsigned char **parents_sha1;
+	struct combine_diff_path paths_head;
+	struct strbuf base;
+
+	parents_sha1 = xmalloc(nparent * sizeof(parents_sha1[0]));
+	for (i = 0; i < nparent; i++)
+		parents_sha1[i] = parents->sha1[i];
+
+	/* fake list head, so worker can assume it is non-NULL */
+	paths_head.next = NULL;
+
+	strbuf_init(&base, PATH_MAX);
+	diff_tree_paths(&paths_head, sha1, parents_sha1, nparent, &base, opt);
+
+	strbuf_release(&base);
+	free(parents_sha1);
+	return paths_head.next;
+}
+
+
 void diff_tree_combined(const unsigned char *sha1,
 			const struct sha1_array *parents,
 			int dense,
@@ -1355,6 +1385,7 @@ void diff_tree_combined(const unsigned char *sha1,
 	struct diff_options diffopts;
 	struct combine_diff_path *p, *paths;
 	int i, num_paths, needsep, show_log_first, num_parent = parents->nr;
+	int need_generic_pathscan;
 
 	/* nothing to do, if no parents */
 	if (!num_parent)
@@ -1377,11 +1408,58 @@ void diff_tree_combined(const unsigned char *sha1,
 
 	/* find set of paths that everybody touches
 	 *
-	 * NOTE find_paths() also handles --stat, as it computes
-	 * diff(sha1,parent_i) for all i to do the job, specifically
-	 * for parent0.
+	 * NOTE
+	 *
+	 * Diffcore transformations are bound to diff_filespec and logic
+	 * comparing two entries - i.e. they do not apply directly to combine
+	 * diff.
+	 *
+	 * If some of such transformations is requested - we launch generic
+	 * path scanning, which works significantly slower compared to
+	 * simultaneous all-trees-in-one-go scan in find_paths_multitree().
+	 *
+	 * TODO some of the filters could be ported to work on
+	 * combine_diff_paths - i.e. all functionality that skips paths, so in
+	 * theory, we could end up having only multitree path scanning.
+	 *
+	 * NOTE please keep this semantically in sync with diffcore_std()
 	 */
-	paths = find_paths(sha1, parents, &diffopts);
+	need_generic_pathscan = opt->skip_stat_unmatch	||
+			DIFF_OPT_TST(opt, FOLLOW_RENAMES)	||
+			opt->break_opt != -1	||
+			opt->detect_rename	||
+			opt->pickaxe		||
+			opt->filter;
+
+
+	if (need_generic_pathscan) {
+		/*
+		 * NOTE generic case also handles --stat, as it computes
+		 * diff(sha1,parent_i) for all i to do the job, specifically
+		 * for parent0.
+		 */
+		paths = find_paths_generic(sha1, parents, &diffopts);
+	}
+	else {
+		int stat_opt;
+		paths = find_paths_multitree(sha1, parents, &diffopts);
+
+		/*
+		 * show stat against the first parent even
+		 * when doing combined diff.
+		 */
+		stat_opt = (opt->output_format &
+				(DIFF_FORMAT_NUMSTAT|DIFF_FORMAT_DIFFSTAT));
+		if (stat_opt) {
+			diffopts.output_format = stat_opt;
+
+			diff_tree_sha1(parents->sha1[0], sha1, "", &diffopts);
+			diffcore_std(&diffopts);
+			if (opt->orderfile)
+				diffcore_order(opt->orderfile);
+			diff_flush(&diffopts);
+		}
+	}
 
 	/* find out number of surviving paths */
 	for (num_paths = 0, p = paths; p; p = p->next)
diff --git a/diff.c b/diff.c
index cda4aa8..f2fff46 100644
--- a/diff.c
+++ b/diff.c
@@ -4764,6 +4764,7 @@ void diffcore_fix_diff_index(struct diff_options *options)
 
 void diffcore_std(struct diff_options *options)
 {
+	/* NOTE please keep the following in sync with diff_tree_combined() */
 	if (options->skip_stat_unmatch)
 		diffcore_skip_stat_unmatch(options);
 	if (!options->found_follow) {
-- 
1.9.rc1.181.g641f458

Re: [PATCH v2 00/19] Multiparent diff tree-walker + combine-diff speedup

[Duy Nguyen]

On Mon, Feb 24, 2014 at 11:21 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
> Hello up there.
>
> Here go combine-diff speedup patches in form of first reworking diff
> tree-walker to work in general case - when a commit have several parents, not
> only one - we are traversing all 1+nparent trees in parallel.
>
> Then we are taking advantage of the new diff tree-walker for speeding up
> combine-diff, which for linux.git results in ~14 times speedup.

I think there is another use case for this n-tree walker (but I'm not
entirely sure yet as I haven't really read the series). In git-log
(either with pathspec or --patch) we basically do this

diff HEAD^ HEAD
diff HEAD^^ HEAD^
diff HEAD^^^ HEAD^^
diff HEAD^^^^ HEAD^^^
...

so except HEAD (and the last commit), all commits' tree will be
read/diff'd twice. With n-tree walker I think we may be able to diff
them in batch to reduce extra processing: commit lists are split into
16-commit blocks where 16 trees are fed to the new tree walker at the
same time. I hope it would make git-log a bit faster (especially for
-S). Maybe not much.
-- 
Duy

Re: [PATCH v2 00/19] Multiparent diff tree-walker + combine-diff speedup

[Kirill Smelkov]

On Tue, Feb 25, 2014 at 06:43:24AM +0700, Duy Nguyen wrote:
> On Mon, Feb 24, 2014 at 11:21 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
> > Hello up there.
> >
> > Here go combine-diff speedup patches in form of first reworking diff
> > tree-walker to work in general case - when a commit have several parents, not
> > only one - we are traversing all 1+nparent trees in parallel.
> >
> > Then we are taking advantage of the new diff tree-walker for speeding up
> > combine-diff, which for linux.git results in ~14 times speedup.
> 
> I think there is another use case for this n-tree walker (but I'm not
> entirely sure yet as I haven't really read the series). In git-log
> (either with pathspec or --patch) we basically do this
> 
> diff HEAD^ HEAD
> diff HEAD^^ HEAD^
> diff HEAD^^^ HEAD^^
> diff HEAD^^^^ HEAD^^^
> ...
> 
> so except HEAD (and the last commit), all commits' tree will be
> read/diff'd twice. With n-tree walker I think we may be able to diff
> them in batch to reduce extra processing: commit lists are split into
> 16-commit blocks where 16 trees are fed to the new tree walker at the
> same time. I hope it would make git-log a bit faster (especially for
> -S). Maybe not much.

Thanks for commenting.

Unfortunately, as it is now, no, and I doubt savings will be
significant. The real speedup comes from the fact that for combined
diff, we can omit recursing into subdirectories, if we know some diff
D(commit,parent_i) is empty. Let me quote myself from

http://article.gmane.org/gmane.comp.version-control.git/242217

On Sun, Feb 16, 2014 at 12:08:29PM +0400, Kirill Smelkov wrote:
> On Fri, Feb 14, 2014 at 09:37:00AM -0800, Junio C Hamano wrote:
> > I wonder if this machinery can be reused for "log -m" as well (or
> > perhaps you do that already?).  After all, by performing a single
> > parallel scan, you are gathering all the necessary information to
> > let you pretend that you did N pairwise diff-tree.
> 
> Unfortunately, as it is now, no, and let me explain why:
> 
> The reason that is not true, is that we omit recursing into directories,
> if we know D(A,some-parent) for that path is empty. That means we don't
> calculate D(A,any-other-parents) for that path and subpaths.
> 
> More structured description is that combined diff and "log -m", which
> could be though as all diffs D(A,Pi) are different things:
> 
>     - the combined diff is D(A,B) generalization based on "^" (sets
>       intersection) operator, and
> 
>     - log -m, aka "all diffs" is D(A,B) generalization based on "v"
>       (sets union) operator.
> 
> Intersection means, we can omit calculating parts from other sets, if we
> know some set does not have an element (remember "don't recurse into
> subdirectories"?), and unioning does not have this property.
> 
> It does so happen, that "^" case (combine-diff) is more interesting,
> because in the end it allows to see new information - the diff a merge
> itself introduces. "log -m" does not have this property and is no more
> interesting to what plain diff(HEAD,HEAD^n) can provide - in other words
> it's just a convenience.
> 
> Now, the diff tree-walker could be generalized once more, to allow
> clients specify, which diffs combination operator to use - intersection
> or unioning, but I doubt that for unioning case that would add
> significant speedup - we can't reduce any diff generation based on
> another diff and the only saving is that we traverse resulting commit
> tree once, but for some cases that could be maybe slower, say if result
> and some parents don't have a path and some parent does, we'll be
> recursing into that path and do more work compared to plain D(A,Pi) for
> Pi that lacks the path.
> 
> In short: it could be generalized more, if needed, but I propose we
> first establish the ground with generalizing to just combine-diff.

besides

    D(HEAD~,  HEAD)
    D(HEAD~2, HEAD~)
    ...
    D(HEAD~{n}, HEAD~{n-1})

is different even from "log -m" case as now there is no single commit
with several parents.

On a related note, while developing this n-tree walker, I've learned
that it is important to load trees in correct order. Quoting patch 18:

-       t1tree = fill_tree_descriptor(&t1, old);
-       t2tree = fill_tree_descriptor(&t2, new);
+       /*
+        * load parents first, as they are probably already cached.
+        *
+        * ( log_tree_diff() parses commit->parent before calling here via
+        *   diff_tree_sha1(parent, commit) )
+        */
+       for (i = 0; i < nparent; ++i)
+               tptree[i] = fill_tree_descriptor(&tp[i], parents_sha1[i]);
+       ttree = fill_tree_descriptor(&t, sha1);

so it loads parent's tree first. If we change this to be the other way,
i.e. load commit's tree first, and then parent's tree, there will be up
to 4% slowdown for whole plain `git log` (without -c).

So maybe what could be done to speedup plain log is for diff tree-walker
to populate some form of recently-loaded trees while walking, and drop
trees from will not-be used anymore commits - e.g. after doing
HEAD~..HEAD for next diff for HEAD~~..HEAD~ HEAD~ trees will be there
and HEAD trees should be dropped (many of them coincides, so reference
counting could help).

This way, we'll leave the walker logic intact, and only there will be
more handy trees-pool supported by fill_tree_descriptor, which imho is a
better design. And this way we could indeed add some not-big, but
noticeable speedup.

Though it is another topic, and at present my time is limited to only go
with combined diff speedup.

Thanks,
Kirill

Re: [PATCH 17/19] Portable alloca for Git

[Thomas Schwinge]

[-- Attachment #1: Type: text/plain, Size: 1166 bytes --]

Hi!

On Mon, 24 Feb 2014 20:21:49 +0400, Kirill Smelkov <kirr@mns.spb.ru> wrote:
> Both autoconf and config.mak.uname configurations were updated. For
> autoconf, we are not bothering considering cases, when no alloca.h is
> available, but alloca() works some other way - its simply alloca.h is
> available and works or not, everything else is deep legacy.

Sounds good for GNU Hurd, or any system using glibc (but have not
explicitly tested your patch).

> For config.mak.uname, I've tried to make my almost-sure guess for where
> alloca() is available, but since I only have access to Linux it is the
> only change I can be sure about myself, with relevant to other changed
> systems people Cc'ed.

> diff --git a/config.mak.uname b/config.mak.uname
> index 7d31fad..71602ee 100644
> --- a/config.mak.uname
> +++ b/config.mak.uname
> @@ -239,6 +243,7 @@ ifeq ($(uname_S),AIX)
>  endif
>  ifeq ($(uname_S),GNU)
>  	# GNU/Hurd
> +	HAVE_ALLOCA_H = YesPlease
>  	NO_STRLCPY = YesPlease
>  	NO_MKSTEMPS = YesPlease
>  	HAVE_PATHS_H = YesPlease

Acked-by: Thomas Schwinge <thomas@codesourcery.com> (GNU Hurd changes)


Grüße,
 Thomas

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: [PATCH 17/19] Portable alloca for Git

[Erik Faye-Lund]

On Mon, Feb 24, 2014 at 5:21 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
> diff --git a/Makefile b/Makefile
> index dddaf4f..0334806 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -316,6 +321,7 @@ endif
>  ifeq ($(uname_S),Windows)
>         GIT_VERSION := $(GIT_VERSION).MSVC
>         pathsep = ;
> +       HAVE_ALLOCA_H = YesPlease
>         NO_PREAD = YesPlease
>         NEEDS_CRYPTO_WITH_SSL = YesPlease
>         NO_LIBGEN_H = YesPlease

In MSVC, alloca is defined in in malloc.h, not alloca.h:

http://msdn.microsoft.com/en-us/library/wb1s57t5.aspx

In fact, it has no alloca.h at all. But we don't have malloca.h in
mingw either, so creating a compat/win32/alloca.h that includes
malloc.h is probably sufficient.

Re: [PATCH 17/19] Portable alloca for Git

[Erik Faye-Lund]

On Fri, Feb 28, 2014 at 2:44 PM, Erik Faye-Lund <kusmabite@gmail.com> wrote:
> On Mon, Feb 24, 2014 at 5:21 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
>> diff --git a/Makefile b/Makefile
>> index dddaf4f..0334806 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -316,6 +321,7 @@ endif
>>  ifeq ($(uname_S),Windows)
>>         GIT_VERSION := $(GIT_VERSION).MSVC
>>         pathsep = ;
>> +       HAVE_ALLOCA_H = YesPlease
>>         NO_PREAD = YesPlease
>>         NEEDS_CRYPTO_WITH_SSL = YesPlease
>>         NO_LIBGEN_H = YesPlease
>
> In MSVC, alloca is defined in in malloc.h, not alloca.h:
>
> http://msdn.microsoft.com/en-us/library/wb1s57t5.aspx
>
> In fact, it has no alloca.h at all. But we don't have malloca.h in
> mingw either, so creating a compat/win32/alloca.h that includes
> malloc.h is probably sufficient.

"But we don't have alloca.h in mingw either", sorry.

Re: [PATCH 17/19] Portable alloca for Git

[Kirill Smelkov]

On Fri, Feb 28, 2014 at 02:50:04PM +0100, Erik Faye-Lund wrote:
> On Fri, Feb 28, 2014 at 2:44 PM, Erik Faye-Lund <kusmabite@gmail.com> wrote:
> > On Mon, Feb 24, 2014 at 5:21 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
> >> diff --git a/Makefile b/Makefile
> >> index dddaf4f..0334806 100644
> >> --- a/Makefile
> >> +++ b/Makefile
> >> @@ -316,6 +321,7 @@ endif
> >>  ifeq ($(uname_S),Windows)
> >>         GIT_VERSION := $(GIT_VERSION).MSVC
> >>         pathsep = ;
> >> +       HAVE_ALLOCA_H = YesPlease
> >>         NO_PREAD = YesPlease
> >>         NEEDS_CRYPTO_WITH_SSL = YesPlease
> >>         NO_LIBGEN_H = YesPlease
> >
> > In MSVC, alloca is defined in in malloc.h, not alloca.h:
> >
> > http://msdn.microsoft.com/en-us/library/wb1s57t5.aspx
> >
> > In fact, it has no alloca.h at all. But we don't have malloca.h in
> > mingw either, so creating a compat/win32/alloca.h that includes
> > malloc.h is probably sufficient.
> 
> "But we don't have alloca.h in mingw either", sorry.

Don't we have that for MSVC already in

    compat/vcbuild/include/alloca.h

and

    ifeq ($(uname_S),Windows)
        ...
        BASIC_CFLAGS = ... -Icompat/vcbuild/include ...


in config.mak.uname ?

And as I've not touched MINGW part in config.mak.uname the patch stays
valid as it is :) and we can incrementally update what platforms have
working alloca with follow-up patches.

In fact that would be maybe preferred, for maintainers to enable alloca
with knowledge and testing, as one person can't have them all at hand.

Thanks,
Kirill

Re: [PATCH 17/19] Portable alloca for Git

[Erik Faye-Lund]

On Fri, Feb 28, 2014 at 6:00 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
> On Fri, Feb 28, 2014 at 02:50:04PM +0100, Erik Faye-Lund wrote:
>> On Fri, Feb 28, 2014 at 2:44 PM, Erik Faye-Lund <kusmabite@gmail.com> wrote:
>> > On Mon, Feb 24, 2014 at 5:21 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
>> >> diff --git a/Makefile b/Makefile
>> >> index dddaf4f..0334806 100644
>> >> --- a/Makefile
>> >> +++ b/Makefile
>> >> @@ -316,6 +321,7 @@ endif
>> >>  ifeq ($(uname_S),Windows)
>> >>         GIT_VERSION := $(GIT_VERSION).MSVC
>> >>         pathsep = ;
>> >> +       HAVE_ALLOCA_H = YesPlease
>> >>         NO_PREAD = YesPlease
>> >>         NEEDS_CRYPTO_WITH_SSL = YesPlease
>> >>         NO_LIBGEN_H = YesPlease
>> >
>> > In MSVC, alloca is defined in in malloc.h, not alloca.h:
>> >
>> > http://msdn.microsoft.com/en-us/library/wb1s57t5.aspx
>> >
>> > In fact, it has no alloca.h at all. But we don't have malloca.h in
>> > mingw either, so creating a compat/win32/alloca.h that includes
>> > malloc.h is probably sufficient.
>>
>> "But we don't have alloca.h in mingw either", sorry.
>
> Don't we have that for MSVC already in
>
>     compat/vcbuild/include/alloca.h
>
> and
>
>     ifeq ($(uname_S),Windows)
>         ...
>         BASIC_CFLAGS = ... -Icompat/vcbuild/include ...
>
>
> in config.mak.uname ?

Ah, of course. Thanks for setting me straight!

> And as I've not touched MINGW part in config.mak.uname the patch stays
> valid as it is :) and we can incrementally update what platforms have
> working alloca with follow-up patches.
>
> In fact that would be maybe preferred, for maintainers to enable alloca
> with knowledge and testing, as one person can't have them all at hand.

Yeah, you're probably right.

Re: [PATCH 17/19] Portable alloca for Git

[Kirill Smelkov]

On Fri, Feb 28, 2014 at 06:19:58PM +0100, Erik Faye-Lund wrote:
> On Fri, Feb 28, 2014 at 6:00 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
> > On Fri, Feb 28, 2014 at 02:50:04PM +0100, Erik Faye-Lund wrote:
> >> On Fri, Feb 28, 2014 at 2:44 PM, Erik Faye-Lund <kusmabite@gmail.com> wrote:
> >> > On Mon, Feb 24, 2014 at 5:21 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
> >> >> diff --git a/Makefile b/Makefile
> >> >> index dddaf4f..0334806 100644
> >> >> --- a/Makefile
> >> >> +++ b/Makefile
> >> >> @@ -316,6 +321,7 @@ endif
> >> >>  ifeq ($(uname_S),Windows)
> >> >>         GIT_VERSION := $(GIT_VERSION).MSVC
> >> >>         pathsep = ;
> >> >> +       HAVE_ALLOCA_H = YesPlease
> >> >>         NO_PREAD = YesPlease
> >> >>         NEEDS_CRYPTO_WITH_SSL = YesPlease
> >> >>         NO_LIBGEN_H = YesPlease
> >> >
> >> > In MSVC, alloca is defined in in malloc.h, not alloca.h:
> >> >
> >> > http://msdn.microsoft.com/en-us/library/wb1s57t5.aspx
> >> >
> >> > In fact, it has no alloca.h at all. But we don't have malloca.h in
> >> > mingw either, so creating a compat/win32/alloca.h that includes
> >> > malloc.h is probably sufficient.
> >>
> >> "But we don't have alloca.h in mingw either", sorry.
> >
> > Don't we have that for MSVC already in
> >
> >     compat/vcbuild/include/alloca.h
> >
> > and
> >
> >     ifeq ($(uname_S),Windows)
> >         ...
> >         BASIC_CFLAGS = ... -Icompat/vcbuild/include ...
> >
> >
> > in config.mak.uname ?
> 
> Ah, of course. Thanks for setting me straight!
> 
> > And as I've not touched MINGW part in config.mak.uname the patch stays
> > valid as it is :) and we can incrementally update what platforms have
> > working alloca with follow-up patches.
> >
> > In fact that would be maybe preferred, for maintainers to enable alloca
> > with knowledge and testing, as one person can't have them all at hand.
> 
> Yeah, you're probably right.

Erik, the patch has been merged into pu today. Would you please
follow-up with tested MINGW change?

Thanks beforehand,
Kirill

Re: [PATCH 12/19] tree-diff: remove special-case diff-emitting code for empty-tree cases

[Junio C Hamano]

Kirill Smelkov <kirr@mns.spb.ru> writes:

> via teaching tree_entry_pathcmp() how to compare empty tree descriptors:

Drop this line, as you explain the "pretend empty compares bigger
than anything else" idea later anyway?  This early part of the
proposed log message made me hiccup while reading it.

> While walking trees, we iterate their entries from lowest to highest in
> sort order, so empty tree means all entries were already went over.
>
> If we artificially assign +infinity value to such tree "entry", it will
> go after all usual entries, and through the usual driver loop we will be
> taking the same actions, which were hand-coded for special cases, i.e.
>
>     t1 empty, t2 non-empty
>         pathcmp(+∞, t2) -> +1
>         show_path(/*t1=*/NULL, t2);     /* = t1 > t2 case in main loop */
>
>     t1 non-empty, t2-empty
>         pathcmp(t1, +∞) -> -1
>         show_path(t1, /*t2=*/NULL);     /* = t1 < t2 case in main loop */

Sounds good.  I would have phrased a bit differently, though:

    When we have T1 and T2, we return a sign that tells the caller
    to indicate the "earlier" one to be emitted, and by returning
    the sign that causes the non-empty side to be emitted, we will
    automatically cause the entries from the remaining side to be
    emitted, without attempting to touch the empty side at all.  We
    can teach tree_entry_pathcmp() to pretend that an empty tree has
    an element that sorts after anything else to achieve this.

without saying "infinity".

> Right now we never go to when compared tree descriptors are infinity,...

Sorry, but I cannot parse this.

> as
> this condition is checked in the loop beginning as finishing criteria,

What condition and which loop?  The loop that immediately surrounds
the callsite of tree_entry_pathcmp() is the infinite "for (;;) {" loop,
and after it prepares t1 and t2 by skipping paths outside pathspec,
we check if both are empty (i.e. we ran out).  Is that the condition
you are referring to?

> but will do in the future, when there will be several parents iterated
> simultaneously, and some pair of them would run to the end.
>
> Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
>
> ( re-posting without change )
>
>  tree-diff.c | 21 +++++++++------------
>  1 file changed, 9 insertions(+), 12 deletions(-)
>
> diff --git a/tree-diff.c b/tree-diff.c
> index cf96ad7..2fd6d0e 100644
> --- a/tree-diff.c
> +++ b/tree-diff.c
> @@ -12,12 +12,19 @@
>   *
>   * NOTE files and directories *always* compare differently, even when having
>   *      the same name - thanks to base_name_compare().
> + *
> + * NOTE empty (=invalid) descriptor(s) take part in comparison as +infty.

The basic idea is very sane.  It is a nice (and obvious---once you
are told about the trick) and clean restructuring of the code.

>   */
>  static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
>  {
>  	struct name_entry *e1, *e2;
>  	int cmp;
>  
> +	if (!t1->size)
> +		return t2->size ? +1 /* +∞ > c */  : 0 /* +∞ = +∞ */;
> +	else if (!t2->size)
> +		return -1;	/* c < +∞ */

Where do these "c" come from?  I somehow feel that these comments
are making it harder to understand what is going on.

>  	e1 = &t1->entry;
>  	e2 = &t2->entry;
>  	cmp = base_name_compare(e1->path, tree_entry_len(e1), e1->mode,
> @@ -151,18 +158,8 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
>  			skip_uninteresting(t1, &base, opt);
>  			skip_uninteresting(t2, &base, opt);
>  		}
> -		if (!t1->size) {
> -			if (!t2->size)
> -				break;
> -			show_path(&base, opt, /*t1=*/NULL, t2);
> -			update_tree_entry(t2);
> -			continue;
> -		}
> -		if (!t2->size) {
> -			show_path(&base, opt, t1, /*t2=*/NULL);
> -			update_tree_entry(t1);
> -			continue;
> -		}
> +		if (!t1->size && !t2->size)
> +			break;
>  
>  		cmp = tree_entry_pathcmp(t1, t2);

Re: [PATCH 11/19] tree-diff: simplify tree_entry_pathcmp

[Junio C Hamano]

Kirill Smelkov <kirr@mns.spb.ru> writes:

> Since an earlier "Finally switch over tree descriptors to contain a
> pre-parsed entry", we can safely access all tree_desc->entry fields
> directly instead of first "extracting" them through
> tree_entry_extract.
>
> Use it. The code generated stays the same - only it now visually looks
> cleaner.
>
> Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
>
> ( re-posting without change )

Thanks.

Hopefully I'll be merging the series up to this point to 'next'
soonish.

>
>  tree-diff.c | 17 ++++++-----------
>  1 file changed, 6 insertions(+), 11 deletions(-)
>
> diff --git a/tree-diff.c b/tree-diff.c
> index 20a4fda..cf96ad7 100644
> --- a/tree-diff.c
> +++ b/tree-diff.c
> @@ -15,18 +15,13 @@
>   */
>  static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
>  {
> -	unsigned mode1, mode2;
> -	const char *path1, *path2;
> -	const unsigned char *sha1, *sha2;
> -	int cmp, pathlen1, pathlen2;
> +	struct name_entry *e1, *e2;
> +	int cmp;
>  
> -	sha1 = tree_entry_extract(t1, &path1, &mode1);
> -	sha2 = tree_entry_extract(t2, &path2, &mode2);
> -
> -	pathlen1 = tree_entry_len(&t1->entry);
> -	pathlen2 = tree_entry_len(&t2->entry);
> -
> -	cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
> +	e1 = &t1->entry;
> +	e2 = &t2->entry;
> +	cmp = base_name_compare(e1->path, tree_entry_len(e1), e1->mode,
> +				e2->path, tree_entry_len(e2), e2->mode);
>  	return cmp;
>  }

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Junio C Hamano]

Kirill Smelkov <kirr@mns.spb.ru> writes:

> The downside is that try_to_follow_renames(), if active, we cause
> re-reading of 2 initial trees, which was negligible based on my timings,

That would depend on how often the codepath triggered in your test
case, but is totally understandable.  It fires only when the path we
have been following disappears from the parent, and the processing
of try-to-follow itself is very compute-intensive (it needs to run
find-copies-harder logic) that will end up reading many subtrees of
the two initial trees; two more reading of tree objects will be
dwarfed by the actual processing.

> and which is outweighed cogently by the upsides.

> Changes since v1:
>
>  - don't need to touch diff.h, as diff_tree() became static.

Nice.  I wonder if it is an option to let the function keep its name
diff_tree() without renaming it to __diff_tree_whatever(), though.

>  tree-diff.c | 60 ++++++++++++++++++++++++++++--------------------------------
>  1 file changed, 28 insertions(+), 32 deletions(-)
>
> diff --git a/tree-diff.c b/tree-diff.c
> index b99622c..f90acf5 100644
> --- a/tree-diff.c
> +++ b/tree-diff.c
> @@ -137,12 +137,17 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
>  	}
>  }
>  
> -static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
> -		     const char *base_str, struct diff_options *opt)
> +static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
> +			    const char *base_str, struct diff_options *opt)
>  {
> +	struct tree_desc t1, t2;
> +	void *t1tree, *t2tree;
>  	struct strbuf base;
>  	int baselen = strlen(base_str);
>  
> +	t1tree = fill_tree_descriptor(&t1, old);
> +	t2tree = fill_tree_descriptor(&t2, new);
> +
>  	/* Enable recursion indefinitely */
>  	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
>  
> @@ -155,39 +160,41 @@ static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
>  		if (diff_can_quit_early(opt))
>  			break;
>  		if (opt->pathspec.nr) {
> -			skip_uninteresting(t1, &base, opt);
> -			skip_uninteresting(t2, &base, opt);
> +			skip_uninteresting(&t1, &base, opt);
> +			skip_uninteresting(&t2, &base, opt);
>  		}
> -		if (!t1->size && !t2->size)
> +		if (!t1.size && !t2.size)
>  			break;
>  
> -		cmp = tree_entry_pathcmp(t1, t2);
> +		cmp = tree_entry_pathcmp(&t1, &t2);
>  
>  		/* t1 = t2 */
>  		if (cmp == 0) {
>  			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
> -			    hashcmp(t1->entry.sha1, t2->entry.sha1) ||
> -			    (t1->entry.mode != t2->entry.mode))
> -				show_path(&base, opt, t1, t2);
> +			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
> +			    (t1.entry.mode != t2.entry.mode))
> +				show_path(&base, opt, &t1, &t2);
>  
> -			update_tree_entry(t1);
> -			update_tree_entry(t2);
> +			update_tree_entry(&t1);
> +			update_tree_entry(&t2);
>  		}
>  
>  		/* t1 < t2 */
>  		else if (cmp < 0) {
> -			show_path(&base, opt, t1, /*t2=*/NULL);
> -			update_tree_entry(t1);
> +			show_path(&base, opt, &t1, /*t2=*/NULL);
> +			update_tree_entry(&t1);
>  		}
>  
>  		/* t1 > t2 */
>  		else {
> -			show_path(&base, opt, /*t1=*/NULL, t2);
> -			update_tree_entry(t2);
> +			show_path(&base, opt, /*t1=*/NULL, &t2);
> +			update_tree_entry(&t2);
>  		}
>  	}
>  
>  	strbuf_release(&base);
> +	free(t2tree);
> +	free(t1tree);
>  	return 0;
>  }
>  
> @@ -202,7 +209,7 @@ static inline int diff_might_be_rename(void)
>  		!DIFF_FILE_VALID(diff_queued_diff.queue[0]->one);
>  }
>  
> -static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, const char *base, struct diff_options *opt)
> +static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
>  {
>  	struct diff_options diff_opts;
>  	struct diff_queue_struct *q = &diff_queued_diff;
> @@ -240,7 +247,7 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
>  	diff_opts.break_opt = opt->break_opt;
>  	diff_opts.rename_score = opt->rename_score;
>  	diff_setup_done(&diff_opts);
> -	diff_tree(t1, t2, base, &diff_opts);
> +	__diff_tree_sha1(old, new, base, &diff_opts);
>  	diffcore_std(&diff_opts);
>  	free_pathspec(&diff_opts.pathspec);
>  
> @@ -301,23 +308,12 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
>  
>  int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
>  {
> -	void *tree1, *tree2;
> -	struct tree_desc t1, t2;
> -	unsigned long size1, size2;
>  	int retval;
>  
> -	tree1 = fill_tree_descriptor(&t1, old);
> -	tree2 = fill_tree_descriptor(&t2, new);
> -	size1 = t1.size;
> -	size2 = t2.size;
> -	retval = diff_tree(&t1, &t2, base, opt);
> -	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename()) {
> -		init_tree_desc(&t1, tree1, size1);
> -		init_tree_desc(&t2, tree2, size2);
> -		try_to_follow_renames(&t1, &t2, base, opt);
> -	}
> -	free(tree1);
> -	free(tree2);
> +	retval = __diff_tree_sha1(old, new, base, opt);
> +	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
> +		try_to_follow_renames(old, new, base, opt);
> +
>  	return retval;
>  }

Re: [PATCH v2 16/19] tree-diff: reuse base str(buf) memory on sub-tree recursion

[Junio C Hamano]

Kirill Smelkov <kirr@mns.spb.ru> writes:

> instead of allocating it all the time for every subtree in
> __diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all
> callee just use it in stacking style, without memory allocations.
>
> This should be faster, and for me this change gives the following
> slight speedups for
>
>     git log --raw --no-abbrev --no-renames --format='%H'
>
>                 navy.git    linux.git v3.10..v3.11
>
>     before      0.618s      1.903s
>     after       0.611s      1.889s
>     speedup     1.1%        0.7%
>
> Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
> ---
>
> Changes since v1:
>
>  - don't need to touch diff.h, as the function we are changing became static.
>
>  tree-diff.c | 36 ++++++++++++++++++------------------
>  1 file changed, 18 insertions(+), 18 deletions(-)
>
> diff --git a/tree-diff.c b/tree-diff.c
> index aea0297..c76821d 100644
> --- a/tree-diff.c
> +++ b/tree-diff.c
> @@ -115,7 +115,7 @@ static void show_path(struct strbuf *base, struct diff_options *opt,
>  	if (recurse) {
>  		strbuf_addch(base, '/');
>  		__diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
> -				 t2 ? t2->entry.sha1 : NULL, base->buf, opt);
> +				 t2 ? t2->entry.sha1 : NULL, base, opt);
>  	}
>  
>  	strbuf_setlen(base, old_baselen);

I was scratching my head for a while, after seeing that there does
not seem to be any *new* code added by this patch in order to
store-away the original length and restore the singleton base buffer
to the original length after using addch/addstr to extend it.

But I see that the code has already been prepared to do this
conversion.  I wonder why we didn't do this earlier ;-)

Looks good.  Thanks.

> @@ -138,12 +138,10 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
>  }
>  
>  static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
> -			    const char *base_str, struct diff_options *opt)
> +			    struct strbuf *base, struct diff_options *opt)
>  {
>  	struct tree_desc t1, t2;
>  	void *t1tree, *t2tree;
> -	struct strbuf base;
> -	int baselen = strlen(base_str);
>  
>  	t1tree = fill_tree_descriptor(&t1, old);
>  	t2tree = fill_tree_descriptor(&t2, new);
> @@ -151,17 +149,14 @@ static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
>  	/* Enable recursion indefinitely */
>  	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
>  
> -	strbuf_init(&base, PATH_MAX);
> -	strbuf_add(&base, base_str, baselen);
> -
>  	for (;;) {
>  		int cmp;
>  
>  		if (diff_can_quit_early(opt))
>  			break;
>  		if (opt->pathspec.nr) {
> -			skip_uninteresting(&t1, &base, opt);
> -			skip_uninteresting(&t2, &base, opt);
> +			skip_uninteresting(&t1, base, opt);
> +			skip_uninteresting(&t2, base, opt);
>  		}
>  		if (!t1.size && !t2.size)
>  			break;
> @@ -173,7 +168,7 @@ static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
>  			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
>  			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
>  			    (t1.entry.mode != t2.entry.mode))
> -				show_path(&base, opt, &t1, &t2);
> +				show_path(base, opt, &t1, &t2);
>  
>  			update_tree_entry(&t1);
>  			update_tree_entry(&t2);
> @@ -181,18 +176,17 @@ static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
>  
>  		/* t1 < t2 */
>  		else if (cmp < 0) {
> -			show_path(&base, opt, &t1, /*t2=*/NULL);
> +			show_path(base, opt, &t1, /*t2=*/NULL);
>  			update_tree_entry(&t1);
>  		}
>  
>  		/* t1 > t2 */
>  		else {
> -			show_path(&base, opt, /*t1=*/NULL, &t2);
> +			show_path(base, opt, /*t1=*/NULL, &t2);
>  			update_tree_entry(&t2);
>  		}
>  	}
>  
> -	strbuf_release(&base);
>  	free(t2tree);
>  	free(t1tree);
>  	return 0;
> @@ -209,7 +203,7 @@ static inline int diff_might_be_rename(void)
>  		!DIFF_FILE_VALID(diff_queued_diff.queue[0]->one);
>  }
>  
> -static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
> +static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, struct strbuf *base, struct diff_options *opt)
>  {
>  	struct diff_options diff_opts;
>  	struct diff_queue_struct *q = &diff_queued_diff;
> @@ -306,13 +300,19 @@ static void try_to_follow_renames(const unsigned char *old, const unsigned char
>  	q->nr = 1;
>  }
>  
> -int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
> +int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base_str, struct diff_options *opt)
>  {
> +	struct strbuf base;
>  	int retval;
>  
> -	retval = __diff_tree_sha1(old, new, base, opt);
> -	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
> -		try_to_follow_renames(old, new, base, opt);
> +	strbuf_init(&base, PATH_MAX);
> +	strbuf_addstr(&base, base_str);
> +
> +	retval = __diff_tree_sha1(old, new, &base, opt);
> +	if (!*base_str && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
> +		try_to_follow_renames(old, new, &base, opt);
> +
> +	strbuf_release(&base);
>  
>  	return retval;
>  }

Re: [PATCH 17/19] Portable alloca for Git

[Junio C Hamano]

Kirill Smelkov <kirr@mns.spb.ru> writes:

> On Fri, Feb 28, 2014 at 06:19:58PM +0100, Erik Faye-Lund wrote:
>> On Fri, Feb 28, 2014 at 6:00 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
>> ...
>> > In fact that would be maybe preferred, for maintainers to enable alloca
>> > with knowledge and testing, as one person can't have them all at hand.
>> 
>> Yeah, you're probably right.
>
> Erik, the patch has been merged into pu today. Would you please
> follow-up with tested MINGW change?

Sooo.... I lost track but this discussion seems to have petered out
around here.  I think the copy we have had for a while on 'pu' is
basically sound, and can easily built on by platform folks by adding
or removing the -DHAVE_ALLOCA_H from the Makefile.

Re: [PATCH 12/19] tree-diff: remove special-case diff-emitting code for empty-tree cases

[Kirill Smelkov]

On Mon, Mar 24, 2014 at 02:18:10PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@mns.spb.ru> writes:
> 
> > via teaching tree_entry_pathcmp() how to compare empty tree descriptors:
> 
> Drop this line, as you explain the "pretend empty compares bigger
> than anything else" idea later anyway?  This early part of the
> proposed log message made me hiccup while reading it.

Hmm, I was trying to show the big picture first and only then details...


> > While walking trees, we iterate their entries from lowest to highest in
> > sort order, so empty tree means all entries were already went over.
> >
> > If we artificially assign +infinity value to such tree "entry", it will
> > go after all usual entries, and through the usual driver loop we will be
> > taking the same actions, which were hand-coded for special cases, i.e.
> >
> >     t1 empty, t2 non-empty
> >         pathcmp(+∞, t2) -> +1
> >         show_path(/*t1=*/NULL, t2);     /* = t1 > t2 case in main loop */
> >
> >     t1 non-empty, t2-empty
> >         pathcmp(t1, +∞) -> -1
> >         show_path(t1, /*t2=*/NULL);     /* = t1 < t2 case in main loop */
> 
> Sounds good.  I would have phrased a bit differently, though:
> 
>     When we have T1 and T2, we return a sign that tells the caller
>     to indicate the "earlier" one to be emitted, and by returning
>     the sign that causes the non-empty side to be emitted, we will
>     automatically cause the entries from the remaining side to be
>     emitted, without attempting to touch the empty side at all.  We
>     can teach tree_entry_pathcmp() to pretend that an empty tree has
>     an element that sorts after anything else to achieve this.
> 
> without saying "infinity".

Doesn't your description, especially "an element that sorts after
anything else" match what "infinity" is pretty exactly? :)

I agree it could read more clearly to those new to the concept, but we
are basically talking about the same thing and once someone is familiar
with infinity and its friends the second description imho is less
obvious.

Let's maybe as a compromise add your text as "In other words <textual
description ...>" ?

This way, it will hopefully be good both ways...


> > Right now we never go to when compared tree descriptors are infinity,...
> 
> Sorry, but I cannot parse this.

Sorry, I've omitted one word here. It should read

    "Right now we never go to when compared tree descriptors are _both_ infinity,..."

i.e. right now we never call tree_entry_pathcmp with both t1 and t2
being empty.


> > as
> > this condition is checked in the loop beginning as finishing criteria,
> 
> What condition and which loop?  The loop that immediately surrounds
> the callsite of tree_entry_pathcmp() is the infinite "for (;;) {" loop,
> and after it prepares t1 and t2 by skipping paths outside pathspec,
> we check if both are empty (i.e. we ran out).  Is that the condition
> you are referring to?

Yes exactly. Modulo diff_can_quit_early() logic, we break from loop in
diff_tree (the loop in which special-case diff-tree emitting code was)
when both trees were scanned to the end, i.e.

        if (!t1->size && !t2->size)
                break;

in other words when both t1 and t2 are "+∞".

Because of that, at this stage we will never go into tree_entry_pathcmp
with (+∞,+∞) arguments, which could mean (!t1->size && !t2->size) case
could be unnecessary in tree_entry_pathcmp and should not be coded at
all...

> > but will do in the future, when there will be several parents iterated
> > simultaneously, and some pair of them would run to the end.

... I was trying to say this case will probably be needed later, and that
it is better to have it for generality.

I hope this should be more clear once that prologue with "both" included
is not confusing.


> > Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
> > Signed-off-by: Junio C Hamano <gitster@pobox.com>
> > ---
> >
> > ( re-posting without change )
> >
> >  tree-diff.c | 21 +++++++++------------
> >  1 file changed, 9 insertions(+), 12 deletions(-)
> >
> > diff --git a/tree-diff.c b/tree-diff.c
> > index cf96ad7..2fd6d0e 100644
> > --- a/tree-diff.c
> > +++ b/tree-diff.c
> > @@ -12,12 +12,19 @@
> >   *
> >   * NOTE files and directories *always* compare differently, even when having
> >   *      the same name - thanks to base_name_compare().
> > + *
> > + * NOTE empty (=invalid) descriptor(s) take part in comparison as +infty.
> 
> The basic idea is very sane.  It is a nice (and obvious---once you
> are told about the trick) and clean restructuring of the code.

Thanks. I was surprised it is seen as a trick, as infinity is very handy
and common concept in many areas and in sorting too.

> 
> >   */
> >  static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
> >  {
> >  	struct name_entry *e1, *e2;
> >  	int cmp;
> >  
> > +	if (!t1->size)
> > +		return t2->size ? +1 /* +∞ > c */  : 0 /* +∞ = +∞ */;
> > +	else if (!t2->size)
> > +		return -1;	/* c < +∞ */
> 
> Where do these "c" come from?  I somehow feel that these comments
> are making it harder to understand what is going on.

"c" means some finite "c"onstant here. When I was studying at school and
at the university, it was common to denote constants via this letter -
i.e. in algebra and operators they often show scalar multiplication as

    c·A     (or α·A)

etc. I understand it could maybe be confusing (but it came to me as
surprise), so would the following be maybe better:

        if (!t1->size)
        	return t2->size ? +1 /* +∞ > const */  : 0 /* +∞ = +∞ */;
        else if (!t2->size)
        	return -1;	/* const < +∞ */

?


Thanks,
Kirill

> >  	e1 = &t1->entry;
> >  	e2 = &t2->entry;
> >  	cmp = base_name_compare(e1->path, tree_entry_len(e1), e1->mode,
> > @@ -151,18 +158,8 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
> >  			skip_uninteresting(t1, &base, opt);
> >  			skip_uninteresting(t2, &base, opt);
> >  		}
> > -		if (!t1->size) {
> > -			if (!t2->size)
> > -				break;
> > -			show_path(&base, opt, /*t1=*/NULL, t2);
> > -			update_tree_entry(t2);
> > -			continue;
> > -		}
> > -		if (!t2->size) {
> > -			show_path(&base, opt, t1, /*t2=*/NULL);
> > -			update_tree_entry(t1);
> > -			continue;
> > -		}
> > +		if (!t1->size && !t2->size)
> > +			break;
> >  
> >  		cmp = tree_entry_pathcmp(t1, t2);

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Kirill Smelkov]

On Mon, Mar 24, 2014 at 02:36:22PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@mns.spb.ru> writes:
> 
> > The downside is that try_to_follow_renames(), if active, we cause
> > re-reading of 2 initial trees, which was negligible based on my timings,
> 
> That would depend on how often the codepath triggered in your test
> case, but is totally understandable.  It fires only when the path we
> have been following disappears from the parent, and the processing
> of try-to-follow itself is very compute-intensive (it needs to run
> find-copies-harder logic) that will end up reading many subtrees of
> the two initial trees; two more reading of tree objects will be
> dwarfed by the actual processing.

I agree and thanks for the explanation.

> > and which is outweighed cogently by the upsides.
> 
> > Changes since v1:
> >
> >  - don't need to touch diff.h, as diff_tree() became static.
> 
> Nice.  I wonder if it is an option to let the function keep its name
> diff_tree() without renaming it to __diff_tree_whatever(), though.

As I see it, in Git for functions operating on trees, there is convention
to accept either `struct tree_desc *` and be named simply, or sha1 and
be named with _sha1 suffix. From this point of view for new diff_tree()
accepting sha1's and staying with its old name would be confusing.

Besides, in the end we'll have two function with high-level wrapper, and
lower-lever worker:

    - diff_tree_sha1(), and
    - diff_tree_paths().

So it's not about this only particular case.  Both do some simple
preparation, call worker, and perform some cleanup.

So the question is how to name the worker?

In Linux they use "__" prefix. We could also use some other prefix or
suffix, e.g. "_bh" (for bottom-half), "_worker", "_low", "_raw", etc...


To me, personally, the cleanest is "__" prefix, but maybe I'm too used
to Linux etc... I'm open to other naming scheme, only it should be
consistent.

What are the downsides of "__" prefix by the way?


> >  tree-diff.c | 60 ++++++++++++++++++++++++++++--------------------------------
> >  1 file changed, 28 insertions(+), 32 deletions(-)
> >
> > diff --git a/tree-diff.c b/tree-diff.c
> > index b99622c..f90acf5 100644
> > --- a/tree-diff.c
> > +++ b/tree-diff.c
> > @@ -137,12 +137,17 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
> >  	}
> >  }
> >  
> > -static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
> > -		     const char *base_str, struct diff_options *opt)
> > +static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
> > +			    const char *base_str, struct diff_options *opt)
> >  {
> > +	struct tree_desc t1, t2;
> > +	void *t1tree, *t2tree;
> >  	struct strbuf base;
> >  	int baselen = strlen(base_str);
> >  
> > +	t1tree = fill_tree_descriptor(&t1, old);
> > +	t2tree = fill_tree_descriptor(&t2, new);
> > +
> >  	/* Enable recursion indefinitely */
> >  	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
> >  
> > @@ -155,39 +160,41 @@ static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
> >  		if (diff_can_quit_early(opt))
> >  			break;
> >  		if (opt->pathspec.nr) {
> > -			skip_uninteresting(t1, &base, opt);
> > -			skip_uninteresting(t2, &base, opt);
> > +			skip_uninteresting(&t1, &base, opt);
> > +			skip_uninteresting(&t2, &base, opt);
> >  		}
> > -		if (!t1->size && !t2->size)
> > +		if (!t1.size && !t2.size)
> >  			break;
> >  
> > -		cmp = tree_entry_pathcmp(t1, t2);
> > +		cmp = tree_entry_pathcmp(&t1, &t2);
> >  
> >  		/* t1 = t2 */
> >  		if (cmp == 0) {
> >  			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
> > -			    hashcmp(t1->entry.sha1, t2->entry.sha1) ||
> > -			    (t1->entry.mode != t2->entry.mode))
> > -				show_path(&base, opt, t1, t2);
> > +			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
> > +			    (t1.entry.mode != t2.entry.mode))
> > +				show_path(&base, opt, &t1, &t2);
> >  
> > -			update_tree_entry(t1);
> > -			update_tree_entry(t2);
> > +			update_tree_entry(&t1);
> > +			update_tree_entry(&t2);
> >  		}
> >  
> >  		/* t1 < t2 */
> >  		else if (cmp < 0) {
> > -			show_path(&base, opt, t1, /*t2=*/NULL);
> > -			update_tree_entry(t1);
> > +			show_path(&base, opt, &t1, /*t2=*/NULL);
> > +			update_tree_entry(&t1);
> >  		}
> >  
> >  		/* t1 > t2 */
> >  		else {
> > -			show_path(&base, opt, /*t1=*/NULL, t2);
> > -			update_tree_entry(t2);
> > +			show_path(&base, opt, /*t1=*/NULL, &t2);
> > +			update_tree_entry(&t2);
> >  		}
> >  	}
> >  
> >  	strbuf_release(&base);
> > +	free(t2tree);
> > +	free(t1tree);
> >  	return 0;
> >  }
> >  
> > @@ -202,7 +209,7 @@ static inline int diff_might_be_rename(void)
> >  		!DIFF_FILE_VALID(diff_queued_diff.queue[0]->one);
> >  }
> >  
> > -static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, const char *base, struct diff_options *opt)
> > +static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
> >  {
> >  	struct diff_options diff_opts;
> >  	struct diff_queue_struct *q = &diff_queued_diff;
> > @@ -240,7 +247,7 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
> >  	diff_opts.break_opt = opt->break_opt;
> >  	diff_opts.rename_score = opt->rename_score;
> >  	diff_setup_done(&diff_opts);
> > -	diff_tree(t1, t2, base, &diff_opts);
> > +	__diff_tree_sha1(old, new, base, &diff_opts);
> >  	diffcore_std(&diff_opts);
> >  	free_pathspec(&diff_opts.pathspec);
> >  
> > @@ -301,23 +308,12 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
> >  
> >  int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
> >  {
> > -	void *tree1, *tree2;
> > -	struct tree_desc t1, t2;
> > -	unsigned long size1, size2;
> >  	int retval;
> >  
> > -	tree1 = fill_tree_descriptor(&t1, old);
> > -	tree2 = fill_tree_descriptor(&t2, new);
> > -	size1 = t1.size;
> > -	size2 = t2.size;
> > -	retval = diff_tree(&t1, &t2, base, opt);
> > -	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename()) {
> > -		init_tree_desc(&t1, tree1, size1);
> > -		init_tree_desc(&t2, tree2, size2);
> > -		try_to_follow_renames(&t1, &t2, base, opt);
> > -	}
> > -	free(tree1);
> > -	free(tree2);
> > +	retval = __diff_tree_sha1(old, new, base, opt);
> > +	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
> > +		try_to_follow_renames(old, new, base, opt);
> > +
> >  	return retval;
> >  }

Re: [PATCH v2 16/19] tree-diff: reuse base str(buf) memory on sub-tree recursion

[Kirill Smelkov]

On Mon, Mar 24, 2014 at 02:43:36PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@mns.spb.ru> writes:
> 
> > instead of allocating it all the time for every subtree in
> > __diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all
> > callee just use it in stacking style, without memory allocations.
> >
> > This should be faster, and for me this change gives the following
> > slight speedups for
> >
> >     git log --raw --no-abbrev --no-renames --format='%H'
> >
> >                 navy.git    linux.git v3.10..v3.11
> >
> >     before      0.618s      1.903s
> >     after       0.611s      1.889s
> >     speedup     1.1%        0.7%
> >
> > Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
> > ---
> >
> > Changes since v1:
> >
> >  - don't need to touch diff.h, as the function we are changing became static.
> >
> >  tree-diff.c | 36 ++++++++++++++++++------------------
> >  1 file changed, 18 insertions(+), 18 deletions(-)
> >
> > diff --git a/tree-diff.c b/tree-diff.c
> > index aea0297..c76821d 100644
> > --- a/tree-diff.c
> > +++ b/tree-diff.c
> > @@ -115,7 +115,7 @@ static void show_path(struct strbuf *base, struct diff_options *opt,
> >  	if (recurse) {
> >  		strbuf_addch(base, '/');
> >  		__diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
> > -				 t2 ? t2->entry.sha1 : NULL, base->buf, opt);
> > +				 t2 ? t2->entry.sha1 : NULL, base, opt);
> >  	}
> >  
> >  	strbuf_setlen(base, old_baselen);
> 
> I was scratching my head for a while, after seeing that there does
> not seem to be any *new* code added by this patch in order to
> store-away the original length and restore the singleton base buffer
> to the original length after using addch/addstr to extend it.
> 
> But I see that the code has already been prepared to do this
> conversion.  I wonder why we didn't do this earlier ;-)

The conversion to reusing memory started in 48932677 "diff-tree: convert
base+baselen to writable strbuf" which allowed to avoid "quite a bit of
malloc() and memcpy()", but for this to work allocation at diff_tree()
entry had to be there.

In particular it had to be there, because diff_tree() accepted base as C
string, not strbuf, and since diff_tree() was calling itself
recursively - oops - new allocation on every subtree.

I've opened the door for avoiding allocations via splitting diff_tree
into high-level and low-level parts. The high-level part still accepts
`char *base`, but low-level function operates on strbuf and recurses
into low-level self.

The high-level diff_tree_sha1() still allocates memory for every
diff(tree1,tree2), but that is significantly lower compared to
allocating memory on every subtree...

The lesson here is: better use strbuf for api unless there is a reason
not to.


> Looks good.  Thanks.

Thanks.

> > @@ -138,12 +138,10 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
> >  }
> >  
> >  static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
> > -			    const char *base_str, struct diff_options *opt)
> > +			    struct strbuf *base, struct diff_options *opt)
> >  {
> >  	struct tree_desc t1, t2;
> >  	void *t1tree, *t2tree;
> > -	struct strbuf base;
> > -	int baselen = strlen(base_str);
> >  
> >  	t1tree = fill_tree_descriptor(&t1, old);
> >  	t2tree = fill_tree_descriptor(&t2, new);
> > @@ -151,17 +149,14 @@ static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
> >  	/* Enable recursion indefinitely */
> >  	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
> >  
> > -	strbuf_init(&base, PATH_MAX);
> > -	strbuf_add(&base, base_str, baselen);
> > -
> >  	for (;;) {
> >  		int cmp;
> >  
> >  		if (diff_can_quit_early(opt))
> >  			break;
> >  		if (opt->pathspec.nr) {
> > -			skip_uninteresting(&t1, &base, opt);
> > -			skip_uninteresting(&t2, &base, opt);
> > +			skip_uninteresting(&t1, base, opt);
> > +			skip_uninteresting(&t2, base, opt);
> >  		}
> >  		if (!t1.size && !t2.size)
> >  			break;
> > @@ -173,7 +168,7 @@ static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
> >  			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
> >  			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
> >  			    (t1.entry.mode != t2.entry.mode))
> > -				show_path(&base, opt, &t1, &t2);
> > +				show_path(base, opt, &t1, &t2);
> >  
> >  			update_tree_entry(&t1);
> >  			update_tree_entry(&t2);
> > @@ -181,18 +176,17 @@ static int __diff_tree_sha1(const unsigned char *old, const unsigned char *new,
> >  
> >  		/* t1 < t2 */
> >  		else if (cmp < 0) {
> > -			show_path(&base, opt, &t1, /*t2=*/NULL);
> > +			show_path(base, opt, &t1, /*t2=*/NULL);
> >  			update_tree_entry(&t1);
> >  		}
> >  
> >  		/* t1 > t2 */
> >  		else {
> > -			show_path(&base, opt, /*t1=*/NULL, &t2);
> > +			show_path(base, opt, /*t1=*/NULL, &t2);
> >  			update_tree_entry(&t2);
> >  		}
> >  	}
> >  
> > -	strbuf_release(&base);
> >  	free(t2tree);
> >  	free(t1tree);
> >  	return 0;
> > @@ -209,7 +203,7 @@ static inline int diff_might_be_rename(void)
> >  		!DIFF_FILE_VALID(diff_queued_diff.queue[0]->one);
> >  }
> >  
> > -static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
> > +static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, struct strbuf *base, struct diff_options *opt)
> >  {
> >  	struct diff_options diff_opts;
> >  	struct diff_queue_struct *q = &diff_queued_diff;
> > @@ -306,13 +300,19 @@ static void try_to_follow_renames(const unsigned char *old, const unsigned char
> >  	q->nr = 1;
> >  }
> >  
> > -int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
> > +int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base_str, struct diff_options *opt)
> >  {
> > +	struct strbuf base;
> >  	int retval;
> >  
> > -	retval = __diff_tree_sha1(old, new, base, opt);
> > -	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
> > -		try_to_follow_renames(old, new, base, opt);
> > +	strbuf_init(&base, PATH_MAX);
> > +	strbuf_addstr(&base, base_str);
> > +
> > +	retval = __diff_tree_sha1(old, new, &base, opt);
> > +	if (!*base_str && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
> > +		try_to_follow_renames(old, new, &base, opt);
> > +
> > +	strbuf_release(&base);
> >  
> >  	return retval;
> >  }

Re: [PATCH 11/19] tree-diff: simplify tree_entry_pathcmp

[Kirill Smelkov]

On Mon, Mar 24, 2014 at 02:25:04PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@mns.spb.ru> writes:
> 
> > Since an earlier "Finally switch over tree descriptors to contain a
> > pre-parsed entry", we can safely access all tree_desc->entry fields
> > directly instead of first "extracting" them through
> > tree_entry_extract.
> >
> > Use it. The code generated stays the same - only it now visually looks
> > cleaner.
> >
> > Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
> > Signed-off-by: Junio C Hamano <gitster@pobox.com>
> > ---
> >
> > ( re-posting without change )
> 
> Thanks.
> 
> Hopefully I'll be merging the series up to this point to 'next'
> soonish.

Thanks a lot!


> >  tree-diff.c | 17 ++++++-----------
> >  1 file changed, 6 insertions(+), 11 deletions(-)
> >
> > diff --git a/tree-diff.c b/tree-diff.c
> > index 20a4fda..cf96ad7 100644
> > --- a/tree-diff.c
> > +++ b/tree-diff.c
> > @@ -15,18 +15,13 @@
> >   */
> >  static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
> >  {
> > -	unsigned mode1, mode2;
> > -	const char *path1, *path2;
> > -	const unsigned char *sha1, *sha2;
> > -	int cmp, pathlen1, pathlen2;
> > +	struct name_entry *e1, *e2;
> > +	int cmp;
> >  
> > -	sha1 = tree_entry_extract(t1, &path1, &mode1);
> > -	sha2 = tree_entry_extract(t2, &path2, &mode2);
> > -
> > -	pathlen1 = tree_entry_len(&t1->entry);
> > -	pathlen2 = tree_entry_len(&t2->entry);
> > -
> > -	cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
> > +	e1 = &t1->entry;
> > +	e2 = &t2->entry;
> > +	cmp = base_name_compare(e1->path, tree_entry_len(e1), e1->mode,
> > +				e2->path, tree_entry_len(e2), e2->mode);
> >  	return cmp;
> >  }

Re: [PATCH 12/19] tree-diff: remove special-case diff-emitting code for empty-tree cases

[Junio C Hamano]

Kirill Smelkov <kirr@navytux.spb.ru> writes:

>> >  static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
>> >  {
>> >  	struct name_entry *e1, *e2;
>> >  	int cmp;
>> >  
>> > +	if (!t1->size)
>> > +		return t2->size ? +1 /* +∞ > c */  : 0 /* +∞ = +∞ */;
>> > +	else if (!t2->size)
>> > +		return -1;	/* c < +∞ */
>> 
>> Where do these "c" come from?  I somehow feel that these comments
>> are making it harder to understand what is going on.
>
> "c" means some finite "c"onstant here. When I was studying at school and
> at the university, it was common to denote constants via this letter -
> i.e. in algebra and operators they often show scalar multiplication as
>
>     c·A     (or α·A)
>
> etc. I understand it could maybe be confusing (but it came to me as
> surprise), so would the following be maybe better:
>
>         if (!t1->size)
>         	return t2->size ? +1 /* +∞ > const */  : 0 /* +∞ = +∞ */;
>         else if (!t2->size)
>         	return -1;	/* const < +∞ */
>
> ?

Not better at all, I am afraid.  A "const" in the code usually means
"something that does not change, as opposed to a variable", but what
you are saying here is "t1 does not have an element but t2 still
does. Pretend as if t1 has a virtual/fake element that is larger
than any real element t2 may happen to have at the head of its
queue", and you are labeling that "real element at the head of t2"
as "const", but as the walker advances, the head element in t1 and
t2 will change---they are not "const" in that sense, and the reader
is left scratching his head seeing "const" there, wondering what the
author of the comment meant.

"real" or "concrete" might be better a phrasing, but I do not think
having "/* +inf > concrete */" there helps the reader understand
what is going on in the first place.  Perhaps:

        /*
         * When one side is empty, pretend that it has an element
         * that sorts later than what the other non-empty side has,
         * so that the caller advances the non-empty side without
         * touching the empty side.
         */
        if (!t1->size)
                return !t2->size ? 0 : 1;
        else if (!t2->size)
                return -1;

or something?

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Junio C Hamano]

Kirill Smelkov <kirr@navytux.spb.ru> writes:

> What are the downsides of "__" prefix by the way?

Aren't these names reserved for compiler/runtime implementations?

Re: [PATCH 12/19] tree-diff: remove special-case diff-emitting code for empty-tree cases

[Junio C Hamano]

Kirill Smelkov <kirr@navytux.spb.ru> writes:

> On Mon, Mar 24, 2014 at 02:18:10PM -0700, Junio C Hamano wrote:
>> Kirill Smelkov <kirr@mns.spb.ru> writes:
>> 
>> > via teaching tree_entry_pathcmp() how to compare empty tree descriptors:
>> 
>> Drop this line, as you explain the "pretend empty compares bigger
>> than anything else" idea later anyway?  This early part of the
>> proposed log message made me hiccup while reading it.
>
> Hmm, I was trying to show the big picture first and only then details...

The subject should be sufficient for the big picture.  "OK, we are
removing the special casing" is what we expect the reader to get.
Then, this

>> > While walking trees, we iterate their entries from lowest to highest in
>> > sort order, so empty tree means all entries were already went over.

sets the background.  "OK, the code walks two trees, both have
sorted elements, in parallel." is what we want the reader to
understand.  Then the next part gives the idea of pretending that
the empty-side always compare later than the non-empty side while
doing that parallel walking (similar to "merge").

So, yes, I think it is a good presentation order to give big picture
punch-line first on the subject, some background and then the
solution.

Re: [PATCH 12/19] tree-diff: remove special-case diff-emitting code for empty-tree cases

[Kirill Smelkov]

On Tue, Mar 25, 2014 at 10:45:01AM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@navytux.spb.ru> writes:
> 
> >> >  static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
> >> >  {
> >> >  	struct name_entry *e1, *e2;
> >> >  	int cmp;
> >> >  
> >> > +	if (!t1->size)
> >> > +		return t2->size ? +1 /* +∞ > c */  : 0 /* +∞ = +∞ */;
> >> > +	else if (!t2->size)
> >> > +		return -1;	/* c < +∞ */
> >> 
> >> Where do these "c" come from?  I somehow feel that these comments
> >> are making it harder to understand what is going on.
> >
> > "c" means some finite "c"onstant here. When I was studying at school and
> > at the university, it was common to denote constants via this letter -
> > i.e. in algebra and operators they often show scalar multiplication as
> >
> >     c·A     (or α·A)
> >
> > etc. I understand it could maybe be confusing (but it came to me as
> > surprise), so would the following be maybe better:
> >
> >         if (!t1->size)
> >         	return t2->size ? +1 /* +∞ > const */  : 0 /* +∞ = +∞ */;
> >         else if (!t2->size)
> >         	return -1;	/* const < +∞ */
> >
> > ?
> 
> Not better at all, I am afraid.  A "const" in the code usually means
> "something that does not change, as opposed to a variable", but what
> you are saying here is "t1 does not have an element but t2 still
> does. Pretend as if t1 has a virtual/fake element that is larger
> than any real element t2 may happen to have at the head of its
> queue", and you are labeling that "real element at the head of t2"
> as "const", but as the walker advances, the head element in t1 and
> t2 will change---they are not "const" in that sense, and the reader
> is left scratching his head seeing "const" there, wondering what the
> author of the comment meant.

I agree.


> "real" or "concrete" might be better a phrasing, but I do not think
> having "/* +inf > concrete */" there helps the reader understand
> what is going on in the first place.  Perhaps:
> 
>         /*
>          * When one side is empty, pretend that it has an element
>          * that sorts later than what the other non-empty side has,
>          * so that the caller advances the non-empty side without
>          * touching the empty side.
>          */
>         if (!t1->size)
>                 return !t2->size ? 0 : 1;
>         else if (!t2->size)
>                 return -1;
> 
> or something?

Yes, that describe the reasoning without stranger symbols. How about
taking it further with

          * NOTE empty (=invalid) descriptor(s) take part in comparison as +infty,
          *      so that they sort *after* valid tree entries.
          *
          *      Due to this convention, if trees are scanned in sorted order, all
          *      non-empty descriptors will be processed first.
          */
         static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
         {
                struct name_entry *e1, *e2;
                int cmp;
         
                /* empty descriptors sort after valid tree entries */
                if (!t1->size)
                        return t2->size ? +1 : 0;
                else if (!t2->size)
                        return -1;

?

On Tue, Mar 25, 2014 at 03:07:33PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@navytux.spb.ru> writes:
> 
> > On Mon, Mar 24, 2014 at 02:18:10PM -0700, Junio C Hamano wrote:
> >> Kirill Smelkov <kirr@mns.spb.ru> writes:
> >> 
> >> > via teaching tree_entry_pathcmp() how to compare empty tree descriptors:
> >> 
> >> Drop this line, as you explain the "pretend empty compares bigger
> >> than anything else" idea later anyway?  This early part of the
> >> proposed log message made me hiccup while reading it.
> >
> > Hmm, I was trying to show the big picture first and only then details...
> 
> The subject should be sufficient for the big picture.  "OK, we are
> removing the special casing" is what we expect the reader to get.
> Then, this
> 
> >> > While walking trees, we iterate their entries from lowest to highest in
> >> > sort order, so empty tree means all entries were already went over.
> 
> sets the background.  "OK, the code walks two trees, both have
> sorted elements, in parallel." is what we want the reader to
> understand.  Then the next part gives the idea of pretending that
> the empty-side always compare later than the non-empty side while
> doing that parallel walking (similar to "merge").
> 
> So, yes, I think it is a good presentation order to give big picture
> punch-line first on the subject, some background and then the
> solution.

Ok, let it be this way and let's drop it.

Here is updated patch:
(please keep author email)

---- 8< ----
From: Kirill Smelkov <kirr@mns.spb.ru>
Date: Mon, 24 Feb 2014 20:21:44 +0400
Subject: [PATCH v2] tree-diff: remove special-case diff-emitting code for empty-tree cases

While walking trees, we iterate their entries from lowest to highest in
sort order, so empty tree means all entries were already went over.

If we artificially assign +infinity value to such tree "entry", it will
go after all usual entries, and through the usual driver loop we will be
taking the same actions, which were hand-coded for special cases, i.e.

    t1 empty, t2 non-empty
        pathcmp(+∞, t2) -> +1
        show_path(/*t1=*/NULL, t2);     /* = t1 > t2 case in main loop */

    t1 non-empty, t2-empty
        pathcmp(t1, +∞) -> -1
        show_path(t1, /*t2=*/NULL);     /* = t1 < t2 case in main loop */

In other words when we have t1 and t2, we return a sign that tells the
caller to indicate the "earlier" one to be emitted, and by returning the
sign that causes the non-empty side to be emitted, we will automatically
cause the entries from the remaining side to be emitted, without
attempting to touch the empty side at all.  We can teach
tree_entry_pathcmp() to pretend that an empty tree has an element that
sorts after anything else to achieve this.

Right now we never go to when compared tree descriptors are both
infinity, as this condition is checked in the loop beginning as
finishing criteria, but will do so in the future, when there will be
several parents iterated simultaneously, and some pair of them would run
to the end.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

Changes since v1:

 - reworked commit log and comments as per Junio suggestions.


 tree-diff.c | 26 ++++++++++++++------------
 1 file changed, 14 insertions(+), 12 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index f8b2607..6177658 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -12,12 +12,24 @@
  *
  * NOTE files and directories *always* compare differently, even when having
  *      the same name - thanks to base_name_compare().
+ *
+ * NOTE empty (=invalid) descriptor(s) take part in comparison as +infty,
+ *      so that they sort *after* valid tree entries.
+ *
+ *      Due to this convention, if trees are scanned in sorted order, all
+ *      non-empty descriptors will be processed first.
  */
 static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 {
 	struct name_entry *e1, *e2;
 	int cmp;
 
+	/* empty descriptors sort after valid tree entries */
+	if (!t1->size)
+		return t2->size ? +1 : 0;
+	else if (!t2->size)
+		return -1;
+
 	e1 = &t1->entry;
 	e2 = &t2->entry;
 	cmp = base_name_compare(e1->path, tree_entry_len(e1), e1->mode,
@@ -150,18 +162,8 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 			skip_uninteresting(t1, &base, opt);
 			skip_uninteresting(t2, &base, opt);
 		}
-		if (!t1->size) {
-			if (!t2->size)
-				break;
-			show_path(&base, opt, /*t1=*/NULL, t2);
-			update_tree_entry(t2);
-			continue;
-		}
-		if (!t2->size) {
-			show_path(&base, opt, t1, /*t2=*/NULL);
-			update_tree_entry(t1);
-			continue;
-		}
+		if (!t1->size && !t2->size)
+			break;
 
 		cmp = tree_entry_pathcmp(t1, t2);
 
-- 
1.9.rc0.143.g6fd479e

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Kirill Smelkov]

On Tue, Mar 25, 2014 at 10:46:32AM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@navytux.spb.ru> writes:
> 
> > What are the downsides of "__" prefix by the way?
> 
> Aren't these names reserved for compiler/runtime implementations?

Yes, but there are precedents when people don't obey it widely and
in practice everything works :) Let it be something portable anyway -
how about diff_tree_sha1_low() ?

So corrected patch is below. If such suffixing will be accepted, I will
send follow-up patches corrected similiary.

  ( or please pull them from
    git://repo.or.cz/git/kirr.git y6/tree-diff-walk-multitree )

Thanks,
Kirill

---- 8< ----
From: Kirill Smelkov <kirr@mns.spb.ru>
Date: Mon, 24 Feb 2014 20:21:46 +0400
Subject: [PATCH v3] tree-diff: rework diff_tree interface to be sha1 based

In the next commit this will allow to reduce intermediate calls, when
recursing into subtrees - at that stage we know only subtree sha1, and
it is natural for tree walker to start from that phase. For now we do

    diff_tree
        show_path
            diff_tree_sha1
                diff_tree
                    ...

and the change will allow to reduce it to

    diff_tree
        show_path
            diff_tree

Also, it will allow to omit allocating strbuf for each subtree, and just
reuse the common strbuf via playing with its len.

The above-mentioned improvements go in the next 2 patches.

The downside is that try_to_follow_renames(), if active, we cause
re-reading of 2 initial trees, which was negligible based on my timings,
and which is outweighed cogently by the upsides.

NOTE To keep with the current interface and semantics, I needed to
rename the function from diff_tree() to diff_tree_sha1(). As
diff_tree_sha1() was already used, and the function we are talking here
is its more low-level helper, let's use convention for suffixing
such helpers with "_low". So the final renaming is

    diff_tree() -> diff_tree_sha1_low()

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

Changes since v2:

 - renamed __diff_tree_sha1() -> diff_tree_sha1_low() as the former
   overlaps with reserved-for-implementation identifiers namespace.


 tree-diff.c | 60 ++++++++++++++++++++++++++++--------------------------------
 1 file changed, 28 insertions(+), 32 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index f137f39..0277c5c 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -141,12 +141,17 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
 	}
 }
 
-static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
-		     const char *base_str, struct diff_options *opt)
+static int diff_tree_sha1_low(const unsigned char *old, const unsigned char *new,
+			      const char *base_str, struct diff_options *opt)
 {
+	struct tree_desc t1, t2;
+	void *t1tree, *t2tree;
 	struct strbuf base;
 	int baselen = strlen(base_str);
 
+	t1tree = fill_tree_descriptor(&t1, old);
+	t2tree = fill_tree_descriptor(&t2, new);
+
 	/* Enable recursion indefinitely */
 	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
@@ -159,39 +164,41 @@ static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 		if (diff_can_quit_early(opt))
 			break;
 		if (opt->pathspec.nr) {
-			skip_uninteresting(t1, &base, opt);
-			skip_uninteresting(t2, &base, opt);
+			skip_uninteresting(&t1, &base, opt);
+			skip_uninteresting(&t2, &base, opt);
 		}
-		if (!t1->size && !t2->size)
+		if (!t1.size && !t2.size)
 			break;
 
-		cmp = tree_entry_pathcmp(t1, t2);
+		cmp = tree_entry_pathcmp(&t1, &t2);
 
 		/* t1 = t2 */
 		if (cmp == 0) {
 			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
-			    hashcmp(t1->entry.sha1, t2->entry.sha1) ||
-			    (t1->entry.mode != t2->entry.mode))
-				show_path(&base, opt, t1, t2);
+			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
+			    (t1.entry.mode != t2.entry.mode))
+				show_path(&base, opt, &t1, &t2);
 
-			update_tree_entry(t1);
-			update_tree_entry(t2);
+			update_tree_entry(&t1);
+			update_tree_entry(&t2);
 		}
 
 		/* t1 < t2 */
 		else if (cmp < 0) {
-			show_path(&base, opt, t1, /*t2=*/NULL);
-			update_tree_entry(t1);
+			show_path(&base, opt, &t1, /*t2=*/NULL);
+			update_tree_entry(&t1);
 		}
 
 		/* t1 > t2 */
 		else {
-			show_path(&base, opt, /*t1=*/NULL, t2);
-			update_tree_entry(t2);
+			show_path(&base, opt, /*t1=*/NULL, &t2);
+			update_tree_entry(&t2);
 		}
 	}
 
 	strbuf_release(&base);
+	free(t2tree);
+	free(t1tree);
 	return 0;
 }
 
@@ -206,7 +213,7 @@ static inline int diff_might_be_rename(void)
 		!DIFF_FILE_VALID(diff_queued_diff.queue[0]->one);
 }
 
-static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, const char *base, struct diff_options *opt)
+static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
 {
 	struct diff_options diff_opts;
 	struct diff_queue_struct *q = &diff_queued_diff;
@@ -244,7 +251,7 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
 	diff_opts.break_opt = opt->break_opt;
 	diff_opts.rename_score = opt->rename_score;
 	diff_setup_done(&diff_opts);
-	diff_tree(t1, t2, base, &diff_opts);
+	diff_tree_sha1_low(old, new, base, &diff_opts);
 	diffcore_std(&diff_opts);
 	free_pathspec(&diff_opts.pathspec);
 
@@ -305,23 +312,12 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
 
 int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
 {
-	void *tree1, *tree2;
-	struct tree_desc t1, t2;
-	unsigned long size1, size2;
 	int retval;
 
-	tree1 = fill_tree_descriptor(&t1, old);
-	tree2 = fill_tree_descriptor(&t2, new);
-	size1 = t1.size;
-	size2 = t2.size;
-	retval = diff_tree(&t1, &t2, base, opt);
-	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename()) {
-		init_tree_desc(&t1, tree1, size1);
-		init_tree_desc(&t2, tree2, size2);
-		try_to_follow_renames(&t1, &t2, base, opt);
-	}
-	free(tree1);
-	free(tree2);
+	retval = diff_tree_sha1_low(old, new, base, opt);
+	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
+		try_to_follow_renames(old, new, base, opt);
+
 	return retval;
 }
 
-- 
1.9.rc0.143.g6fd479e

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Junio C Hamano]

Kirill Smelkov <kirr@navytux.spb.ru> writes:

> On Tue, Mar 25, 2014 at 10:46:32AM -0700, Junio C Hamano wrote:
>> Kirill Smelkov <kirr@navytux.spb.ru> writes:
>> 
>> > What are the downsides of "__" prefix by the way?
>> 
>> Aren't these names reserved for compiler/runtime implementations?
>
> Yes, but there are precedents when people don't obey it widely and
> in practice everything works :)

I think you are alluding to the practice in the Linux kernel, but
their requirement is vastly different---their product do not even
link with libc and they always compile with specific selected
versions of gcc, no?

> Let it be something portable anyway -
> how about diff_tree_sha1_low() ?

Sure.

As this is a file-scope static, I do not think the exact naming
matters that much.  Just FYI, we seem to use ll_ prefix (standing
for low-level) in some places.

Thanks.

Re: [PATCH 15/19] tree-diff: no need to call "full" diff_tree_sha1 from show_path()

[Kirill Smelkov]

On Mon, Feb 24, 2014 at 08:21:47PM +0400, Kirill Smelkov wrote:
> As described in previous commit, when recursing into sub-trees, we can
> use lower-level tree walker, since its interface is now sha1 based.
> 
> The change is ok, because diff_tree_sha1() only invokes
> __diff_tree_sha1(), and also, if base is empty, try_to_follow_renames().
> But base is not empty here, as we have added a path and '/' before
> recursing.
> 
> Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
> 
> ( re-posting without change )
> 
>  tree-diff.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tree-diff.c b/tree-diff.c
> index f90acf5..aea0297 100644
> --- a/tree-diff.c
> +++ b/tree-diff.c
> @@ -114,8 +114,8 @@ static void show_path(struct strbuf *base, struct diff_options *opt,
>  
>  	if (recurse) {
>  		strbuf_addch(base, '/');
> -		diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
> -			       t2 ? t2->entry.sha1 : NULL, base->buf, opt);
> +		__diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
> +				 t2 ? t2->entry.sha1 : NULL, base->buf, opt);
>  	}
>  
>  	strbuf_setlen(base, old_baselen);

I've found this does not compile as I've forgot to add __diff_tree_sha1
prototype, and also we are changing naming for __diff_tree_sha1() to
ll_diff_tree_sha1() to follow Git coding style for consistency and
corrections to previous patch, so here goes v2:

(please keep author email)
---- 8< ----
From: Kirill Smelkov <kirr@mns.spb.ru>
Date: Mon, 24 Feb 2014 20:21:47 +0400
Subject: [PATCH v2] tree-diff: no need to call "full" diff_tree_sha1 from show_path()

As described in previous commit, when recursing into sub-trees, we can
use lower-level tree walker, since its interface is now sha1 based.

The change is ok, because diff_tree_sha1() only invokes
ll_diff_tree_sha1(), and also, if base is empty, try_to_follow_renames().
But base is not empty here, as we have added a path and '/' before
recursing.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

Changes since v1:

 - adjust to renaming __diff_tree_sha1 -> ll_diff_tree_sha1;
 - added ll_diff_tree_sha1 prototype as the function is defined below
   here-introduced call-site.

 tree-diff.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 1d02e43..7fbb022 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -6,6 +6,10 @@
 #include "diffcore.h"
 #include "tree.h"
 
+
+static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
+			     const char *base_str, struct diff_options *opt);
+
 /*
  * Compare two tree entries, taking into account only path/S_ISDIR(mode),
  * but not their sha1's.
@@ -118,8 +122,8 @@ static void show_path(struct strbuf *base, struct diff_options *opt,
 
 	if (recurse) {
 		strbuf_addch(base, '/');
-		diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
-			       t2 ? t2->entry.sha1 : NULL, base->buf, opt);
+		ll_diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
+				  t2 ? t2->entry.sha1 : NULL, base->buf, opt);
 	}
 
 	strbuf_setlen(base, old_baselen);
-- 
1.9.rc0.143.g6fd479e

Re: [PATCH v2 16/19] tree-diff: reuse base str(buf) memory on sub-tree recursion

[Kirill Smelkov]

On Tue, Mar 25, 2014 at 01:23:20PM +0400, Kirill Smelkov wrote:
> On Mon, Mar 24, 2014 at 02:43:36PM -0700, Junio C Hamano wrote:
> > Kirill Smelkov <kirr@mns.spb.ru> writes:
> > 
> > > instead of allocating it all the time for every subtree in
> > > __diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all
> > > callee just use it in stacking style, without memory allocations.
> > >
> > > This should be faster, and for me this change gives the following
> > > slight speedups for
> > >
> > >     git log --raw --no-abbrev --no-renames --format='%H'
> > >
> > >                 navy.git    linux.git v3.10..v3.11
> > >
> > >     before      0.618s      1.903s
> > >     after       0.611s      1.889s
> > >     speedup     1.1%        0.7%
> > >
> > > Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
> > > ---
> > >
> > > Changes since v1:
> > >
> > >  - don't need to touch diff.h, as the function we are changing became static.
> > >
> > >  tree-diff.c | 36 ++++++++++++++++++------------------
> > >  1 file changed, 18 insertions(+), 18 deletions(-)
> > >
> > > diff --git a/tree-diff.c b/tree-diff.c
> > > index aea0297..c76821d 100644
> > > --- a/tree-diff.c
> > > +++ b/tree-diff.c
> > > @@ -115,7 +115,7 @@ static void show_path(struct strbuf *base, struct diff_options *opt,
> > >  	if (recurse) {
> > >  		strbuf_addch(base, '/');
> > >  		__diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
> > > -				 t2 ? t2->entry.sha1 : NULL, base->buf, opt);
> > > +				 t2 ? t2->entry.sha1 : NULL, base, opt);
> > >  	}
> > >  
> > >  	strbuf_setlen(base, old_baselen);
> > 
> > I was scratching my head for a while, after seeing that there does
> > not seem to be any *new* code added by this patch in order to
> > store-away the original length and restore the singleton base buffer
> > to the original length after using addch/addstr to extend it.
> > 
> > But I see that the code has already been prepared to do this
> > conversion.  I wonder why we didn't do this earlier ;-)
> 
> The conversion to reusing memory started in 48932677 "diff-tree: convert
> base+baselen to writable strbuf" which allowed to avoid "quite a bit of
> malloc() and memcpy()", but for this to work allocation at diff_tree()
> entry had to be there.
> 
> In particular it had to be there, because diff_tree() accepted base as C
> string, not strbuf, and since diff_tree() was calling itself
> recursively - oops - new allocation on every subtree.
> 
> I've opened the door for avoiding allocations via splitting diff_tree
> into high-level and low-level parts. The high-level part still accepts
> `char *base`, but low-level function operates on strbuf and recurses
> into low-level self.
> 
> The high-level diff_tree_sha1() still allocates memory for every
> diff(tree1,tree2), but that is significantly lower compared to
> allocating memory on every subtree...
> 
> The lesson here is: better use strbuf for api unless there is a reason
> not to.
> 
> 
> > Looks good.  Thanks.
> 
> Thanks.

Thanks again. Here it goes adjusted to __diff_tree_sha1 -> ll_diff_tree_sha1 renaming:

(please keep author email)
---- 8< ----
From: Kirill Smelkov <kirr@mns.spb.ru>
Date: Mon, 24 Feb 2014 20:21:48 +0400
Subject: [PATCH v3] tree-diff: reuse base str(buf) memory on sub-tree recursion

instead of allocating it all the time for every subtree in
ll_diff_tree_sha1, let's allocate it once in diff_tree_sha1, and then all
callee just use it in stacking style, without memory allocations.

This should be faster, and for me this change gives the following
slight speedups for

    git log --raw --no-abbrev --no-renames --format='%H'

                navy.git    linux.git v3.10..v3.11

    before      0.618s      1.903s
    after       0.611s      1.889s
    speedup     1.1%        0.7%

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

Changes since v2:

 - adjust to __diff_tree_sha1 -> ll_diff_tree_sha1 renaming.

Changes since v1:

 - don't need to touch diff.h, as the function we are changing became
   static.

 tree-diff.c | 38 +++++++++++++++++++-------------------
 1 file changed, 19 insertions(+), 19 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index 7fbb022..8c8bde6 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -8,7 +8,7 @@
 
 
 static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
-			     const char *base_str, struct diff_options *opt);
+			     struct strbuf *base, struct diff_options *opt);
 
 /*
  * Compare two tree entries, taking into account only path/S_ISDIR(mode),
@@ -123,7 +123,7 @@ static void show_path(struct strbuf *base, struct diff_options *opt,
 	if (recurse) {
 		strbuf_addch(base, '/');
 		ll_diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
-				  t2 ? t2->entry.sha1 : NULL, base->buf, opt);
+				  t2 ? t2->entry.sha1 : NULL, base, opt);
 	}
 
 	strbuf_setlen(base, old_baselen);
@@ -146,12 +146,10 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
 }
 
 static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
-			     const char *base_str, struct diff_options *opt)
+			     struct strbuf *base, struct diff_options *opt)
 {
 	struct tree_desc t1, t2;
 	void *t1tree, *t2tree;
-	struct strbuf base;
-	int baselen = strlen(base_str);
 
 	t1tree = fill_tree_descriptor(&t1, old);
 	t2tree = fill_tree_descriptor(&t2, new);
@@ -159,17 +157,14 @@ static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 	/* Enable recursion indefinitely */
 	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
-	strbuf_init(&base, PATH_MAX);
-	strbuf_add(&base, base_str, baselen);
-
 	for (;;) {
 		int cmp;
 
 		if (diff_can_quit_early(opt))
 			break;
 		if (opt->pathspec.nr) {
-			skip_uninteresting(&t1, &base, opt);
-			skip_uninteresting(&t2, &base, opt);
+			skip_uninteresting(&t1, base, opt);
+			skip_uninteresting(&t2, base, opt);
 		}
 		if (!t1.size && !t2.size)
 			break;
@@ -181,7 +176,7 @@ static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
 			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
 			    (t1.entry.mode != t2.entry.mode))
-				show_path(&base, opt, &t1, &t2);
+				show_path(base, opt, &t1, &t2);
 
 			update_tree_entry(&t1);
 			update_tree_entry(&t2);
@@ -189,18 +184,17 @@ static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 
 		/* t1 < t2 */
 		else if (cmp < 0) {
-			show_path(&base, opt, &t1, /*t2=*/NULL);
+			show_path(base, opt, &t1, /*t2=*/NULL);
 			update_tree_entry(&t1);
 		}
 
 		/* t1 > t2 */
 		else {
-			show_path(&base, opt, /*t1=*/NULL, &t2);
+			show_path(base, opt, /*t1=*/NULL, &t2);
 			update_tree_entry(&t2);
 		}
 	}
 
-	strbuf_release(&base);
 	free(t2tree);
 	free(t1tree);
 	return 0;
@@ -217,7 +211,7 @@ static inline int diff_might_be_rename(void)
 		!DIFF_FILE_VALID(diff_queued_diff.queue[0]->one);
 }
 
-static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
+static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, struct strbuf *base, struct diff_options *opt)
 {
 	struct diff_options diff_opts;
 	struct diff_queue_struct *q = &diff_queued_diff;
@@ -314,13 +308,19 @@ static void try_to_follow_renames(const unsigned char *old, const unsigned char
 	q->nr = 1;
 }
 
-int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
+int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base_str, struct diff_options *opt)
 {
+	struct strbuf base;
 	int retval;
 
-	retval = ll_diff_tree_sha1(old, new, base, opt);
-	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
-		try_to_follow_renames(old, new, base, opt);
+	strbuf_init(&base, PATH_MAX);
+	strbuf_addstr(&base, base_str);
+
+	retval = ll_diff_tree_sha1(old, new, &base, opt);
+	if (!*base_str && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
+		try_to_follow_renames(old, new, &base, opt);
+
+	strbuf_release(&base);
 
 	return retval;
 }
-- 
1.9.rc0.143.g6fd479e

Re: [PATCH 17/19] Portable alloca for Git

[Kirill Smelkov]

On Mon, Mar 24, 2014 at 02:47:24PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@mns.spb.ru> writes:
> 
> > On Fri, Feb 28, 2014 at 06:19:58PM +0100, Erik Faye-Lund wrote:
> >> On Fri, Feb 28, 2014 at 6:00 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
> >> ...
> >> > In fact that would be maybe preferred, for maintainers to enable alloca
> >> > with knowledge and testing, as one person can't have them all at hand.
> >> 
> >> Yeah, you're probably right.
> >
> > Erik, the patch has been merged into pu today. Would you please
> > follow-up with tested MINGW change?
> 
> Sooo.... I lost track but this discussion seems to have petered out
> around here.  I think the copy we have had for a while on 'pu' is
> basically sound, and can easily built on by platform folks by adding
> or removing the -DHAVE_ALLOCA_H from the Makefile.

Yes, that is all correct - that version works and we can improve it in
the future with platform-specific follow-up patches, if needed.

Please pick up the patch with ack from Thomas Schwinge.

Thanks,
Kirill

(please keep author email)
---- 8< ----
From: Kirill Smelkov <kirr@mns.spb.ru>
Date: Mon, 24 Feb 2014 20:21:49 +0400
Subject: [PATCH v1a] Portable alloca for Git

In the next patch we'll have to use alloca() for performance reasons,
but since alloca is non-standardized and is not portable, let's have a
trick with compatibility wrappers:

1. at configure time, determine, do we have working alloca() through
   alloca.h, and define

    #define HAVE_ALLOCA_H

   if yes.

2. in code

    #ifdef HAVE_ALLOCA_H
    # include <alloca.h>
    # define xalloca(size)      (alloca(size))
    # define xalloca_free(p)    do {} while(0)
    #else
    # define xalloca(size)      (xmalloc(size))
    # define xalloca_free(p)    (free(p))
    #endif

   and use it like

   func() {
       p = xalloca(size);
       ...

       xalloca_free(p);
   }

This way, for systems, where alloca is available, we'll have optimal
on-stack allocations with fast executions. On the other hand, on
systems, where alloca is not available, this gracefully fallbacks to
xmalloc/free.

Both autoconf and config.mak.uname configurations were updated. For
autoconf, we are not bothering considering cases, when no alloca.h is
available, but alloca() works some other way - its simply alloca.h is
available and works or not, everything else is deep legacy.

For config.mak.uname, I've tried to make my almost-sure guess for where
alloca() is available, but since I only have access to Linux it is the
only change I can be sure about myself, with relevant to other changed
systems people Cc'ed.

NOTE

SunOS and Windows had explicit -DHAVE_ALLOCA_H in their configurations.
I've changed that to now-common HAVE_ALLOCA_H=YesPlease which should be
correct.

Cc: Brandon Casey <drafnel@gmail.com>
Cc: Marius Storm-Olsen <mstormo@gmail.com>
Cc: Johannes Sixt <j6t@kdbg.org>
Cc: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Cc: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Cc: Gerrit Pape <pape@smarden.org>
Cc: Petr Salinger <Petr.Salinger@seznam.cz>
Cc: Jonathan Nieder <jrnieder@gmail.com>
Cc: Thomas Schwinge <tschwinge@gnu.org>
Acked-by: Thomas Schwinge <thomas@codesourcery.com> (GNU Hurd changes)
Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

Changes since v1:

 - added ack for GNU/Hurd.

 Makefile          |  6 ++++++
 config.mak.uname  | 10 ++++++++--
 configure.ac      |  8 ++++++++
 git-compat-util.h |  8 ++++++++
 4 files changed, 30 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index dddaf4f..0334806 100644
--- a/Makefile
+++ b/Makefile
@@ -30,6 +30,8 @@ all::
 # Define LIBPCREDIR=/foo/bar if your libpcre header and library files are in
 # /foo/bar/include and /foo/bar/lib directories.
 #
+# Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.
+#
 # Define NO_CURL if you do not have libcurl installed.  git-http-fetch and
 # git-http-push are not built, and you cannot use http:// and https://
 # transports (neither smart nor dumb).
@@ -1099,6 +1101,10 @@ ifdef USE_LIBPCRE
 	EXTLIBS += -lpcre
 endif
 
+ifdef HAVE_ALLOCA_H
+	BASIC_CFLAGS += -DHAVE_ALLOCA_H
+endif
+
 ifdef NO_CURL
 	BASIC_CFLAGS += -DNO_CURL
 	REMOTE_CURL_PRIMARY =
diff --git a/config.mak.uname b/config.mak.uname
index 7d31fad..71602ee 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -28,6 +28,7 @@ ifeq ($(uname_S),OSF1)
 	NO_NSEC = YesPlease
 endif
 ifeq ($(uname_S),Linux)
+	HAVE_ALLOCA_H = YesPlease
 	NO_STRLCPY = YesPlease
 	NO_MKSTEMPS = YesPlease
 	HAVE_PATHS_H = YesPlease
@@ -35,6 +36,7 @@ ifeq ($(uname_S),Linux)
 	HAVE_DEV_TTY = YesPlease
 endif
 ifeq ($(uname_S),GNU/kFreeBSD)
+	HAVE_ALLOCA_H = YesPlease
 	NO_STRLCPY = YesPlease
 	NO_MKSTEMPS = YesPlease
 	HAVE_PATHS_H = YesPlease
@@ -103,6 +105,7 @@ ifeq ($(uname_S),SunOS)
 	NEEDS_NSL = YesPlease
 	SHELL_PATH = /bin/bash
 	SANE_TOOL_PATH = /usr/xpg6/bin:/usr/xpg4/bin
+	HAVE_ALLOCA_H = YesPlease
 	NO_STRCASESTR = YesPlease
 	NO_MEMMEM = YesPlease
 	NO_MKDTEMP = YesPlease
@@ -146,7 +149,7 @@ ifeq ($(uname_S),SunOS)
 	endif
 	INSTALL = /usr/ucb/install
 	TAR = gtar
-	BASIC_CFLAGS += -D__EXTENSIONS__ -D__sun__ -DHAVE_ALLOCA_H
+	BASIC_CFLAGS += -D__EXTENSIONS__ -D__sun__
 endif
 ifeq ($(uname_O),Cygwin)
 	ifeq ($(shell expr "$(uname_R)" : '1\.[1-6]\.'),4)
@@ -166,6 +169,7 @@ ifeq ($(uname_O),Cygwin)
 	else
 		NO_REGEX = UnfortunatelyYes
 	endif
+	HAVE_ALLOCA_H = YesPlease
 	NEEDS_LIBICONV = YesPlease
 	NO_FAST_WORKING_DIRECTORY = UnfortunatelyYes
 	NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease
@@ -239,6 +243,7 @@ ifeq ($(uname_S),AIX)
 endif
 ifeq ($(uname_S),GNU)
 	# GNU/Hurd
+	HAVE_ALLOCA_H = YesPlease
 	NO_STRLCPY = YesPlease
 	NO_MKSTEMPS = YesPlease
 	HAVE_PATHS_H = YesPlease
@@ -316,6 +321,7 @@ endif
 ifeq ($(uname_S),Windows)
 	GIT_VERSION := $(GIT_VERSION).MSVC
 	pathsep = ;
+	HAVE_ALLOCA_H = YesPlease
 	NO_PREAD = YesPlease
 	NEEDS_CRYPTO_WITH_SSL = YesPlease
 	NO_LIBGEN_H = YesPlease
@@ -363,7 +369,7 @@ ifeq ($(uname_S),Windows)
 	COMPAT_OBJS = compat/msvc.o compat/winansi.o \
 		compat/win32/pthread.o compat/win32/syslog.o \
 		compat/win32/dirent.o
-	COMPAT_CFLAGS = -D__USE_MINGW_ACCESS -DNOGDI -DHAVE_STRING_H -DHAVE_ALLOCA_H -Icompat -Icompat/regex -Icompat/win32 -DSTRIP_EXTENSION=\".exe\"
+	COMPAT_CFLAGS = -D__USE_MINGW_ACCESS -DNOGDI -DHAVE_STRING_H -Icompat -Icompat/regex -Icompat/win32 -DSTRIP_EXTENSION=\".exe\"
 	BASIC_LDFLAGS = -IGNORE:4217 -IGNORE:4049 -NOLOGO -SUBSYSTEM:CONSOLE -NODEFAULTLIB:MSVCRT.lib
 	EXTLIBS = user32.lib advapi32.lib shell32.lib wininet.lib ws2_32.lib
 	PTHREAD_LIBS =
diff --git a/configure.ac b/configure.ac
index 2f43393..0eae704 100644
--- a/configure.ac
+++ b/configure.ac
@@ -272,6 +272,14 @@ AS_HELP_STRING([],           [ARG can be also prefix for libpcre library and hea
 	GIT_CONF_SUBST([LIBPCREDIR])
     fi)
 #
+# Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.
+AC_FUNC_ALLOCA
+case $ac_cv_working_alloca_h in
+    yes)    HAVE_ALLOCA_H=YesPlease;;
+    *)      HAVE_ALLOCA_H='';;
+esac
+GIT_CONF_SUBST([HAVE_ALLOCA_H])
+#
 # Define NO_CURL if you do not have curl installed.  git-http-pull and
 # git-http-push are not built, and you cannot use http:// and https://
 # transports.
diff --git a/git-compat-util.h b/git-compat-util.h
index cbd86c3..63b2b3b 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -526,6 +526,14 @@ extern void release_pack_memory(size_t);
 typedef void (*try_to_free_t)(size_t);
 extern try_to_free_t set_try_to_free_routine(try_to_free_t);
 
+#ifdef HAVE_ALLOCA_H
+# include <alloca.h>
+# define xalloca(size)      (alloca(size))
+# define xalloca_free(p)    do {} while (0)
+#else
+# define xalloca(size)      (xmalloc(size))
+# define xalloca_free(p)    (free(p))
+#endif
 extern char *xstrdup(const char *str);
 extern void *xmalloc(size_t size);
 extern void *xmallocz(size_t size);
-- 
1.9.rc0.143.g6fd479e

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Kirill Smelkov]

On Mon, Feb 24, 2014 at 08:21:50PM +0400, Kirill Smelkov wrote:
[...]
> not changed:
> 
> - low-level helpers are still named with "__" prefix as, imho, that is the best
>   convention to name such helpers, without sacrificing signal/noise ratio. All
>   of them are now static though.

Please find attached corrected version of this patch with
__diff_tree_sha1() renamed to ll_diff_tree_sha1() and other identifiers
corrected similarly for consistency with Git codebase style.

Thanks,
Kirill

(please keep author email)
---- 8< ----
From: Kirill Smelkov <kirr@mns.spb.ru>
Date: Mon, 24 Feb 2014 20:21:50 +0400
Subject: [PATCH v3] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

Previously diff_tree(), which is now named ll_diff_tree_sha1(), was
generating diff_filepair(s) for two trees t1 and t2, and that was
usually used for a commit as t1=HEAD~, and t2=HEAD - i.e. to see changes
a commit introduces.

In Git, however, we have fundamentally built flexibility in that a
commit can have many parents - 1 for a plain commit, 2 for a simple merge,
but also more than 2 for merging several heads at once.

For merges there is a so called combine-diff, which shows diff, a merge
introduces by itself, omitting changes done by any parent. That works
through first finding paths, that are different to all parents, and then
showing generalized diff, with separate columns for +/- for each parent.
The code lives in combine-diff.c .

There is an impedance mismatch, however, in that a commit could
generally have any number of parents, and that while diffing trees, we
divide cases for 2-tree diffs and more-than-2-tree diffs. I mean there
is no special casing for multiple parents commits in e.g.
revision-walker .

That impedance mismatch *hurts* *performance* *badly* for generating
combined diffs - in "combine-diff: optimize combine_diff_path
sets intersection" I've already removed some slowness from it, but from
the timings provided there, it could be seen, that combined diffs still
cost more than an order of magnitude more cpu time, compared to diff for
usual commits, and that would only be an optimistic estimate, if we take
into account that for e.g. linux.git there is only one merge for several
dozens of plain commits.

That slowness comes from the fact that currently, while generating
combined diff, a lot of time is spent computing diff(commit,commit^2)
just to only then intersect that huge diff to almost small set of files
from diff(commit,commit^1).

That's because at present, to compute combine-diff, for first finding
paths, that "every parent touches", we use the following combine-diff
property/definition:

D(A,P1...Pn) = D(A,P1) ^ ... ^ D(A,Pn)      (w.r.t. paths)

where

D(A,P1...Pn) is combined diff between commit A, and parents Pi

and

D(A,Pi) is usual two-tree diff Pi..A

So if any of that D(A,Pi) is huge, tracting 1 n-parent combine-diff as n
1-parent diffs and intersecting results will be slow.

And usually, for linux.git and other topic-based workflows, that
D(A,P2) is huge, because, if merge-base of A and P2, is several dozens
of merges (from A, via first parent) below, that D(A,P2) will be diffing
sum of merges from several subsystems to 1 subsystem.

The solution is to avoid computing n 1-parent diffs, and to find
changed-to-all-parents paths via scanning A's and all Pi's trees
simultaneously, at each step comparing their entries, and based on that
comparison, populate paths result, and deduce we could *skip*
*recursing* into subdirectories, if at least for 1 parent, sha1 of that
dir tree is the same as in A. That would save us from doing significant
amount of needless work.

Such approach is very similar to what diff_tree() does, only there we
deal with scanning only 2 trees simultaneously, and for n+1 tree, the
logic is a bit more complex:

D(A,X1...Xn) calculation scheme
-------------------------------

D(A,X1...Xn) = D(A,X1) ^ ... ^ D(A,Xn)       (regarding resulting paths set)

     D(A,Xj)         - diff between A..Xj
     D(A,X1...Xn)    - combined diff from A to parents X1,...,Xn

We start from all trees, which are sorted, and compare their entries in
lock-step:

      A     X1       Xn
      -     -        -
     |a|   |x1|     |xn|
     |-|   |--| ... |--|      i = argmin(x1...xn)
     | |   |  |     |  |
     |-|   |--|     |--|
     |.|   |. |     |. |
      .     .        .
      .     .        .

at any time there could be 3 cases:

     1)  a < xi;
     2)  a > xi;
     3)  a = xi.

Schematic deduction of what every case means, and what to do, follows:

1)  a < xi  ->  ∀j a ∉ Xj  ->  "+a" ∈ D(A,Xj)  ->  D += "+a";  a↓

2)  a > xi

    2.1) ∃j: xj > xi  ->  "-xi" ∉ D(A,Xj)  ->  D += ø;  ∀ xk=xi  xk↓
    2.2) ∀j  xj = xi  ->  xj ∉ A  ->  "-xj" ∈ D(A,Xj)  ->  D += "-xi";  ∀j xj↓

3)  a = xi

    3.1) ∃j: xj > xi  ->  "+a" ∈ D(A,Xj)  ->  only xk=xi remains to investigate
    3.2) xj = xi  ->  investigate δ(a,xj)
     |
     |
     v

    3.1+3.2) looking at δ(a,xk) ∀k: xk=xi - if all != ø  ->

                      ⎧δ(a,xk)  - if xk=xi
             ->  D += ⎨
                      ⎩"+a"     - if xk>xi

    in any case a↓  ∀ xk=xi  xk↓

~

For comparison, here is how diff_tree() works:

D(A,B) calculation scheme
-------------------------

    A     B
    -     -
   |a|   |b|    a < b   ->  a ∉ B   ->   D(A,B) +=  +a    a↓
   |-|   |-|    a > b   ->  b ∉ A   ->   D(A,B) +=  -b    b↓
   | |   | |    a = b   ->  investigate δ(a,b)            a↓ b↓
   |-|   |-|
   |.|   |.|
    .     .
    .     .

~~~~~~~~

This patch generalizes diff tree-walker to work with arbitrary number of
parents as described above - i.e. now there is a resulting tree t, and
some parents trees tp[i] i=[0..nparent). The generalization builds on
the fact that usual diff

D(A,B)

is by definition the same as combined diff

D(A,[B]),

so if we could rework the code for common case and make it be not slower
for nparent=1 case, usual diff(t1,t2) generation will not be slower, and
multiparent diff tree-walker would greatly benefit generating
combine-diff.

What we do is as follows:

1) diff tree-walker ll_diff_tree_sha1() is internally reworked to be
   a paths generator (new name diff_tree_paths()), with each generated path
   being `struct combine_diff_path` with info for path, new sha1,mode and for
   every parent which sha1,mode it was in it.

2) From that info, we can still generate usual diff queue with
   struct diff_filepairs, via "exporting" generated
   combine_diff_path, if we know we run for nparent=1 case.
   (see emit_diff() which is now named emit_diff_first_parent_only())

3) In order for diff_can_quit_early(), which checks

       DIFF_OPT_TST(opt, HAS_CHANGES))

   to work, that exporting have to be happening not in bulk, but
   incrementally, one diff path at a time.

   For such consumers, there is a new callback in diff_options
   introduced:

       ->pathchange(opt, struct combine_diff_path *)

   which, if set to !NULL, is called for every generated path.

   (see new compat ll_diff_tree_sha1() wrapper around new paths
    generator for setup)

4) The paths generation itself, is reworked from previous
   ll_diff_tree_sha1() code according to "D(A,X1...Xn) calculation
   scheme" provided above:

   On the start we allocate [nparent] arrays in place what was
   earlier just for one parent tree.

   then we just generalize loops, and comparison according to the
   algorithm.

Some notes(*):

1) alloca(), for small arrays, is used for "runs not slower for
   nparent=1 case than before" goal - if we change it to xmalloc()/free()
   the timings get ~1% worse. For alloca() we use just-introduced
   xalloca/xalloca_free compatibility wrappers, so it should not be a
   portability problem.

2) For every parent tree, we need to keep a tag, whether entry from that
   parent equals to entry from minimal parent. For performance reasons I'm
   keeping that tag in entry's mode field in unused bit - see S_IFXMIN_NEQ.
   Not doing so, we'd need to alloca another [nparent] array, which hurts
   performance.

3) For emitted paths, memory could be reused, if we know the path was
   processed via callback and will not be needed later. We use efficient
   hand-made realloc-style path_appendnew(), that saves us from ~1-1.5%
   of potential additional slowdown.

4) goto(s) are used in several places, as the code executes a little bit
   faster with lowered register pressure.

Also

- we should now check for FIND_COPIES_HARDER not only when two entries
  names are the same, and their hashes are equal, but also for a case,
  when a path was removed from some of all parents having it.

  The reason is, if we don't, that path won't be emitted at all (see
  "a > xi" case), and we'll just skip it, and FIND_COPIES_HARDER wants
  all paths - with diff or without - to be emitted, to be later analyzed
  for being copies sources.

  The new check is only necessary for nparent >1, as for nparent=1 case
  xmin_eqtotal always =1 =nparent, and a path is always added to diff as
  removal.

~~~~~~~~

Timings for

    # without -c, i.e. testing only nparent=1 case
    `git log --raw --no-abbrev --no-renames`

before and after the patch are as follows:

                navy.git        linux.git v3.10..v3.11

    before      0.611s          1.889s
    after       0.619s          1.907s
    slowdown    1.3%            0.9%

This timings show we did no harm to usual diff(tree1,tree2) generation.
From the table we can see that we actually did ~1% slowdown, but I think
I've "earned" that 1% in the previous patch ("tree-diff: reuse base
str(buf) memory on sub-tree recursion", HEAD~~) so for nparent=1 case,
net timings stays approximately the same.

The output also stayed the same.

(*) If we revert 1)-4) to more usual techniques, for nparent=1 case,
    we'll get ~2-2.5% of additional slowdown, which I've tried to avoid, as
   "do no harm for nparent=1 case" rule.

For linux.git, combined diff will run an order of magnitude faster and
appropriate timings will be provided in the next commit, as we'll be
taking advantage of the new diff tree-walker for combined-diff
generation there.

P.S. and combined diff is not some exotic/for-play-only stuff - for
example for a program I write to represent Git archives as readonly
filesystem, there is initial scan with

    `git log --reverse --raw --no-abbrev --no-renames -c`

to extract log of what was created/changed when, as a result building a
map

    {}  sha1    ->  in which commit (and date) a content was added

that `-c` means also show combined diff for merges, and without them, if
a merge is non-trivial (merges changes from two parents with both having
separate changes to a file), or an evil one, the map will not be full,
i.e. some valid sha1 would be absent from it.

That case was my initial motivation for combined diffs speedup.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

Changes since v2:

- rename:
    __diff_tree_paths   -> ll_diff_tree_paths
    __path_append_new   -> path_append_new

  and adjust to previous renaming
    __diff_tree_sha1() -> ll_diff_tree_sha1()

  to be consistent with Git coding style for naming low-level helpers
  (ll_ prefix) and not using "__" as it overlaps with
  reserved-for-implementation identifier namespace.

  path_append_new goes without "ll_" prefix as it is the only function
  with no high-level counterpart.

Changes since v1:

- fixed last-minute thinko/bug last time introduced on my side (sorry) with
  opt->pathchange manipulation in __diff_tree_sha1() - we were forgetting to
  restore opt->pathchange, which led to incorrect log -c (merges _and_ plain
  diff-tree) output;

  This time, I've verified several times, log output stays really the same.

- direct use of alloca() changed to portability wrappers xalloca/xalloca_free
  which gracefully degrade to xmalloc/free on systems, where alloca is not
  available (see new patch 17).

- "i = 0; do { ... } while (++i < nparent)" is back to usual looping
  "for (i = 0; i < nparent; ++)", as I've re-measured timings and the
  difference is negligible.

  ( Initially, when I was fighting for every cycle it made sense, but real
    no-slowdown turned out to be related to avoiding mallocs, load trees in correct
    order and reducing register pressure. )

- S_IFXMIN_NEQ definition moved out to cache.h, to have all modes registry in one place;


- p0 -> first_parent; corrected comments about how emit_diff_first_parent_only
  behaves;
 cache.h     |  15 ++
 diff.c      |   1 +
 diff.h      |  10 ++
 tree-diff.c | 505 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------
 4 files changed, 467 insertions(+), 64 deletions(-)

diff --git a/cache.h b/cache.h
index dc040fb..e7f5a0c 100644
--- a/cache.h
+++ b/cache.h
@@ -75,6 +75,21 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);
 #define S_ISGITLINK(m)	(((m) & S_IFMT) == S_IFGITLINK)
 
 /*
+ * Some mode bits are also used internally for computations.
+ *
+ * They *must* not overlap with any valid modes, and they *must* not be emitted
+ * to outside world - i.e. appear on disk or network. In other words, it's just
+ * temporary fields, which we internally use, but they have to stay in-house.
+ *
+ * ( such approach is valid, as standard S_IF* fits into 16 bits, and in Git
+ *   codebase mode is `unsigned int` which is assumed to be at least 32 bits )
+ */
+
+/* used internally in tree-diff */
+#define S_DIFFTREE_IFXMIN_NEQ	0x80000000
+
+
+/*
  * Intensive research over the course of many years has shown that
  * port 9418 is totally unused by anything else. Or
  *
diff --git a/diff.c b/diff.c
index 8e4a6a9..cda4aa8 100644
--- a/diff.c
+++ b/diff.c
@@ -3216,6 +3216,7 @@ void diff_setup(struct diff_options *options)
 	options->context = diff_context_default;
 	DIFF_OPT_SET(options, RENAME_EMPTY);
 
+	/* pathchange left =NULL by default */
 	options->change = diff_change;
 	options->add_remove = diff_addremove;
 	options->use_color = diff_use_color_default;
diff --git a/diff.h b/diff.h
index 5d7b9f7..732dca7 100644
--- a/diff.h
+++ b/diff.h
@@ -15,6 +15,10 @@ struct diff_filespec;
 struct userdiff_driver;
 struct sha1_array;
 struct commit;
+struct combine_diff_path;
+
+typedef int (*pathchange_fn_t)(struct diff_options *options,
+		 struct combine_diff_path *path);
 
 typedef void (*change_fn_t)(struct diff_options *options,
 		 unsigned old_mode, unsigned new_mode,
@@ -157,6 +161,7 @@ struct diff_options {
 	int close_file;
 
 	struct pathspec pathspec;
+	pathchange_fn_t pathchange;
 	change_fn_t change;
 	add_remove_fn_t add_remove;
 	diff_format_fn_t format_callback;
@@ -189,6 +194,11 @@ const char *diff_line_prefix(struct diff_options *);
 
 extern const char mime_boundary_leader[];
 
+extern
+struct combine_diff_path *diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parent_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt);
 extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 			  const char *base, struct diff_options *opt);
 extern int diff_root_tree_sha1(const unsigned char *new, const char *base,
diff --git a/tree-diff.c b/tree-diff.c
index 8c8bde6..4a497f7 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -6,7 +6,19 @@
 #include "diffcore.h"
 #include "tree.h"
 
+/*
+ * internal mode marker, saying a tree entry != entry of tp[imin]
+ * (see ll_diff_tree_paths for what it means there)
+ *
+ * we will update/use/emit entry for diff only with it unset.
+ */
+#define S_IFXMIN_NEQ	S_DIFFTREE_IFXMIN_NEQ
+
 
+static struct combine_diff_path *ll_diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parents_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt);
 static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 			     struct strbuf *base, struct diff_options *opt);
 
@@ -42,71 +54,151 @@ static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 }
 
 
-/* convert path, t1/t2 -> opt->diff_*() callbacks */
-static void emit_diff(struct diff_options *opt, struct strbuf *path,
-		      struct tree_desc *t1, struct tree_desc *t2)
+/*
+ * convert path -> opt->diff_*() callbacks
+ *
+ * emits diff to first parent only, and tells diff tree-walker that we are done
+ * with p and it can be freed.
+ */
+static int emit_diff_first_parent_only(struct diff_options *opt, struct combine_diff_path *p)
 {
-	unsigned int mode1 = t1 ? t1->entry.mode : 0;
-	unsigned int mode2 = t2 ? t2->entry.mode : 0;
-
-	if (mode1 && mode2) {
-		opt->change(opt, mode1, mode2, t1->entry.sha1, t2->entry.sha1,
-			1, 1, path->buf, 0, 0);
+	struct combine_diff_parent *p0 = &p->parent[0];
+	if (p->mode && p0->mode) {
+		opt->change(opt, p0->mode, p->mode, p0->sha1, p->sha1,
+			1, 1, p->path, 0, 0);
 	}
 	else {
 		const unsigned char *sha1;
 		unsigned int mode;
 		int addremove;
 
-		if (mode2) {
+		if (p->mode) {
 			addremove = '+';
-			sha1 = t2->entry.sha1;
-			mode = mode2;
+			sha1 = p->sha1;
+			mode = p->mode;
 		} else {
 			addremove = '-';
-			sha1 = t1->entry.sha1;
-			mode = mode1;
+			sha1 = p0->sha1;
+			mode = p0->mode;
 		}
 
-		opt->add_remove(opt, addremove, mode, sha1, 1, path->buf, 0);
+		opt->add_remove(opt, addremove, mode, sha1, 1, p->path, 0);
 	}
+
+	return 0;	/* we are done with p */
 }
 
 
-/* new path should be added to diff
+/*
+ * Make a new combine_diff_path from path/mode/sha1
+ * and append it to paths list tail.
+ *
+ * Memory for created elements could be reused:
+ *
+ *	- if last->next == NULL, the memory is allocated;
+ *
+ *	- if last->next != NULL, it is assumed that p=last->next was returned
+ *	  earlier by this function, and p->next was *not* modified.
+ *	  The memory is then reused from p.
+ *
+ * so for clients,
+ *
+ * - if you do need to keep the element
+ *
+ *	p = path_appendnew(p, ...);
+ *	process(p);
+ *	p->next = NULL;
+ *
+ * - if you don't need to keep the element after processing
+ *
+ *	pprev = p;
+ *	p = path_appendnew(p, ...);
+ *	process(p);
+ *	p = pprev;
+ *	; don't forget to free tail->next in the end
+ *
+ * p->parent[] remains uninitialized.
+ */
+static struct combine_diff_path *path_appendnew(struct combine_diff_path *last,
+	int nparent, const struct strbuf *base, const char *path, int pathlen,
+	unsigned mode, const unsigned char *sha1)
+{
+	struct combine_diff_path *p;
+	int len = base->len + pathlen;
+	int alloclen = combine_diff_path_size(nparent, len);
+
+	/* if last->next is !NULL - it is a pre-allocated memory, we can reuse */
+	p = last->next;
+	if (p && (alloclen > (intptr_t)p->next)) {
+		free(p);
+		p = NULL;
+	}
+
+	if (!p) {
+		p = xmalloc(alloclen);
+
+		/*
+		 * until we go to it next round, .next holds how many bytes we
+		 * allocated (for faster realloc - we don't need copying old data).
+		 */
+		p->next = (struct combine_diff_path *)(intptr_t)alloclen;
+	}
+
+	last->next = p;
+
+	p->path = (char *)&(p->parent[nparent]);
+	memcpy(p->path, base->buf, base->len);
+	memcpy(p->path + base->len, path, pathlen);
+	p->path[len] = 0;
+	p->mode = mode;
+	hashcpy(p->sha1, sha1 ? sha1 : null_sha1);
+
+	return p;
+}
+
+/*
+ * new path should be added to combine diff
  *
  * 3 cases on how/when it should be called and behaves:
  *
- *	!t1,  t2	-> path added, parent lacks it
- *	 t1, !t2	-> path removed from parent
- *	 t1,  t2	-> path modified
+ *	 t, !tp		-> path added, all parents lack it
+ *	!t,  tp		-> path removed from all parents
+ *	 t,  tp		-> path modified/added
+ *			   (M for tp[i]=tp[imin], A otherwise)
  */
-static void show_path(struct strbuf *base, struct diff_options *opt,
-		      struct tree_desc *t1, struct tree_desc *t2)
+static struct combine_diff_path *emit_path(struct combine_diff_path *p,
+	struct strbuf *base, struct diff_options *opt, int nparent,
+	struct tree_desc *t, struct tree_desc *tp,
+	int imin)
 {
 	unsigned mode;
 	const char *path;
+	const unsigned char *sha1;
 	int pathlen;
 	int old_baselen = base->len;
-	int isdir, recurse = 0, emitthis = 1;
+	int i, isdir, recurse = 0, emitthis = 1;
 
 	/* at least something has to be valid */
-	assert(t1 || t2);
+	assert(t || tp);
 
-	if (t2) {
+	if (t) {
 		/* path present in resulting tree */
-		tree_entry_extract(t2, &path, &mode);
-		pathlen = tree_entry_len(&t2->entry);
+		sha1 = tree_entry_extract(t, &path, &mode);
+		pathlen = tree_entry_len(&t->entry);
 		isdir = S_ISDIR(mode);
 	} else {
 		/*
-		 * a path was removed - take path from parent. Also take
-		 * mode from parent, to decide on recursion.
+		 * a path was removed - take path from imin parent. Also take
+		 * mode from that parent, to decide on recursion(1).
+		 *
+		 * 1) all modes for tp[k]=tp[imin] should be the same wrt
+		 *    S_ISDIR, thanks to base_name_compare().
 		 */
-		tree_entry_extract(t1, &path, &mode);
-		pathlen = tree_entry_len(&t1->entry);
+		tree_entry_extract(&tp[imin], &path, &mode);
+		pathlen = tree_entry_len(&tp[imin].entry);
 
 		isdir = S_ISDIR(mode);
+		sha1 = NULL;
 		mode = 0;
 	}
 
@@ -115,18 +207,81 @@ static void show_path(struct strbuf *base, struct diff_options *opt,
 		emitthis = DIFF_OPT_TST(opt, TREE_IN_RECURSIVE);
 	}
 
-	strbuf_add(base, path, pathlen);
+	if (emitthis) {
+		int keep;
+		struct combine_diff_path *pprev = p;
+		p = path_appendnew(p, nparent, base, path, pathlen, mode, sha1);
+
+		for (i = 0; i < nparent; ++i) {
+			/*
+			 * tp[i] is valid, if present and if tp[i]==tp[imin] -
+			 * otherwise, we should ignore it.
+			 */
+			int tpi_valid = tp && !(tp[i].entry.mode & S_IFXMIN_NEQ);
+
+			const unsigned char *sha1_i;
+			unsigned mode_i;
+
+			p->parent[i].status =
+				!t ? DIFF_STATUS_DELETED :
+					tpi_valid ?
+						DIFF_STATUS_MODIFIED :
+						DIFF_STATUS_ADDED;
+
+			if (tpi_valid) {
+				sha1_i = tp[i].entry.sha1;
+				mode_i = tp[i].entry.mode;
+			}
+			else {
+				sha1_i = NULL;
+				mode_i = 0;
+			}
+
+			p->parent[i].mode = mode_i;
+			hashcpy(p->parent[i].sha1, sha1_i ? sha1_i : null_sha1);
+		}
 
-	if (emitthis)
-		emit_diff(opt, base, t1, t2);
+		keep = 1;
+		if (opt->pathchange)
+			keep = opt->pathchange(opt, p);
+
+		/*
+		 * If a path was filtered or consumed - we don't need to add it
+		 * to the list and can reuse its memory, leaving it as
+		 * pre-allocated element on the tail.
+		 *
+		 * On the other hand, if path needs to be kept, we need to
+		 * correct its .next to NULL, as it was pre-initialized to how
+		 * much memory was allocated.
+		 *
+		 * see path_appendnew() for details.
+		 */
+		if (!keep)
+			p = pprev;
+		else
+			p->next = NULL;
+	}
 
 	if (recurse) {
+		const unsigned char **parents_sha1;
+
+		parents_sha1 = xalloca(nparent * sizeof(parents_sha1[0]));
+		for (i = 0; i < nparent; ++i) {
+			/* same rule as in emitthis */
+			int tpi_valid = tp && !(tp[i].entry.mode & S_IFXMIN_NEQ);
+
+			parents_sha1[i] = tpi_valid ? tp[i].entry.sha1
+						    : NULL;
+		}
+
+		strbuf_add(base, path, pathlen);
 		strbuf_addch(base, '/');
-		ll_diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
-				  t2 ? t2->entry.sha1 : NULL, base, opt);
+		p = ll_diff_tree_paths(p, sha1, parents_sha1, nparent, base, opt);
+		xalloca_free(parents_sha1);
 	}
 
 	strbuf_setlen(base, old_baselen);
+	return p;
 }
 
 static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
@@ -145,59 +300,260 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
 	}
 }
 
-static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
-			     struct strbuf *base, struct diff_options *opt)
+
+/*
+ * generate paths for combined diff D(sha1,parents_sha1[])
+ *
+ * Resulting paths are appended to combine_diff_path linked list, and also, are
+ * emitted on the go via opt->pathchange() callback, so it is possible to
+ * process the result as batch or incrementally.
+ *
+ * The paths are generated scanning new tree and all parents trees
+ * simultaneously, similarly to what diff_tree() was doing for 2 trees.
+ * The theory behind such scan is as follows:
+ *
+ *
+ * D(A,X1...Xn) calculation scheme
+ * -------------------------------
+ *
+ * D(A,X1...Xn) = D(A,X1) ^ ... ^ D(A,Xn)	(regarding resulting paths set)
+ *
+ *	D(A,Xj)		- diff between A..Xj
+ *	D(A,X1...Xn)	- combined diff from A to parents X1,...,Xn
+ *
+ *
+ * We start from all trees, which are sorted, and compare their entries in
+ * lock-step:
+ *
+ *	 A     X1       Xn
+ *	 -     -        -
+ *	|a|   |x1|     |xn|
+ *	|-|   |--| ... |--|      i = argmin(x1...xn)
+ *	| |   |  |     |  |
+ *	|-|   |--|     |--|
+ *	|.|   |. |     |. |
+ *	 .     .        .
+ *	 .     .        .
+ *
+ * at any time there could be 3 cases:
+ *
+ *	1)  a < xi;
+ *	2)  a > xi;
+ *	3)  a = xi.
+ *
+ * Schematic deduction of what every case means, and what to do, follows:
+ *
+ * 1)  a < xi  ->  ∀j a ∉ Xj  ->  "+a" ∈ D(A,Xj)  ->  D += "+a";  a↓
+ *
+ * 2)  a > xi
+ *
+ *     2.1) ∃j: xj > xi  ->  "-xi" ∉ D(A,Xj)  ->  D += ø;  ∀ xk=xi  xk↓
+ *     2.2) ∀j  xj = xi  ->  xj ∉ A  ->  "-xj" ∈ D(A,Xj)  ->  D += "-xi";  ∀j xj↓
+ *
+ * 3)  a = xi
+ *
+ *     3.1) ∃j: xj > xi  ->  "+a" ∈ D(A,Xj)  ->  only xk=xi remains to investigate
+ *     3.2) xj = xi  ->  investigate δ(a,xj)
+ *      |
+ *      |
+ *      v
+ *
+ *     3.1+3.2) looking at δ(a,xk) ∀k: xk=xi - if all != ø  ->
+ *
+ *                       ⎧δ(a,xk)  - if xk=xi
+ *              ->  D += ⎨
+ *                       ⎩"+a"     - if xk>xi
+ *
+ *
+ *     in any case a↓  ∀ xk=xi  xk↓
+ *
+ *
+ * ~~~~~~~~
+ *
+ * NOTE
+ *
+ *	Usual diff D(A,B) is by definition the same as combined diff D(A,[B]),
+ *	so this diff paths generator can, and is used, for plain diffs
+ *	generation too.
+ *
+ *	Please keep attention to the common D(A,[B]) case when working on the
+ *	code, in order not to slow it down.
+ *
+ * NOTE
+ *	nparent must be > 0.
+ */
+
+
+/* ∀ xk=xi  xk↓ */
+static inline void update_tp_entries(struct tree_desc *tp, int nparent)
 {
-	struct tree_desc t1, t2;
-	void *t1tree, *t2tree;
+	int i;
+	for (i = 0; i < nparent; ++i)
+		if (!(tp[i].entry.mode & S_IFXMIN_NEQ))
+			update_tree_entry(&tp[i]);
+}
 
-	t1tree = fill_tree_descriptor(&t1, old);
-	t2tree = fill_tree_descriptor(&t2, new);
+static struct combine_diff_path *ll_diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parents_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt)
+{
+	struct tree_desc t, *tp;
+	void *ttree, **tptree;
+	int i;
+
+	tp     = xalloca(nparent * sizeof(tp[0]));
+	tptree = xalloca(nparent * sizeof(tptree[0]));
+
+	/*
+	 * load parents first, as they are probably already cached.
+	 *
+	 * ( log_tree_diff() parses commit->parent before calling here via
+	 *   diff_tree_sha1(parent, commit) )
+	 */
+	for (i = 0; i < nparent; ++i)
+		tptree[i] = fill_tree_descriptor(&tp[i], parents_sha1[i]);
+	ttree = fill_tree_descriptor(&t, sha1);
 
 	/* Enable recursion indefinitely */
 	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
 	for (;;) {
-		int cmp;
+		int imin, cmp;
 
 		if (diff_can_quit_early(opt))
 			break;
+
 		if (opt->pathspec.nr) {
-			skip_uninteresting(&t1, base, opt);
-			skip_uninteresting(&t2, base, opt);
+			skip_uninteresting(&t, base, opt);
+			for (i = 0; i < nparent; i++)
+				skip_uninteresting(&tp[i], base, opt);
 		}
-		if (!t1.size && !t2.size)
-			break;
 
-		cmp = tree_entry_pathcmp(&t1, &t2);
+		/* comparing is finished when all trees are done */
+		if (!t.size) {
+			int done = 1;
+			for (i = 0; i < nparent; ++i)
+				if (tp[i].size) {
+					done = 0;
+					break;
+				}
+			if (done)
+				break;
+		}
+
+		/*
+		 * lookup imin = argmin(x1...xn),
+		 * mark entries whether they =tp[imin] along the way
+		 */
+		imin = 0;
+		tp[0].entry.mode &= ~S_IFXMIN_NEQ;
+
+		for (i = 1; i < nparent; ++i) {
+			cmp = tree_entry_pathcmp(&tp[i], &tp[imin]);
+			if (cmp < 0) {
+				imin = i;
+				tp[i].entry.mode &= ~S_IFXMIN_NEQ;
+			}
+			else if (cmp == 0) {
+				tp[i].entry.mode &= ~S_IFXMIN_NEQ;
+			}
+			else {
+				tp[i].entry.mode |= S_IFXMIN_NEQ;
+			}
+		}
+
+		/* fixup markings for entries before imin */
+		for (i = 0; i < imin; ++i)
+			tp[i].entry.mode |= S_IFXMIN_NEQ;	/* x[i] > x[imin] */
+
 
-		/* t1 = t2 */
-		if (cmp == 0) {
-			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
-			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
-			    (t1.entry.mode != t2.entry.mode))
-				show_path(base, opt, &t1, &t2);
 
-			update_tree_entry(&t1);
-			update_tree_entry(&t2);
+		/* compare a vs x[imin] */
+		cmp = tree_entry_pathcmp(&t, &tp[imin]);
+
+		/* a = xi */
+		if (cmp == 0) {
+			/* are either xk > xi or diff(a,xk) != ø ? */
+			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
+				for (i = 0; i < nparent; ++i) {
+					/* x[i] > x[imin] */
+					if (tp[i].entry.mode & S_IFXMIN_NEQ)
+						continue;
+
+					/* diff(a,xk) != ø */
+					if (hashcmp(t.entry.sha1, tp[i].entry.sha1) ||
+					    (t.entry.mode != tp[i].entry.mode))
+						continue;
+
+					goto skip_emit_t_tp;
+				}
+			}
+
+			/* D += {δ(a,xk) if xk=xi;  "+a" if xk > xi} */
+			p = emit_path(p, base, opt, nparent,
+					&t, tp, imin);
+
+		skip_emit_t_tp:
+			/* a↓,  ∀ xk=ximin  xk↓ */
+			update_tree_entry(&t);
+			update_tp_entries(tp, nparent);
 		}
 
-		/* t1 < t2 */
+		/* a < xi */
 		else if (cmp < 0) {
-			show_path(base, opt, &t1, /*t2=*/NULL);
-			update_tree_entry(&t1);
+			/* D += "+a" */
+			p = emit_path(p, base, opt, nparent,
+					&t, /*tp=*/NULL, -1);
+
+			/* a↓ */
+			update_tree_entry(&t);
 		}
 
-		/* t1 > t2 */
+		/* a > xi */
 		else {
-			show_path(base, opt, /*t1=*/NULL, &t2);
-			update_tree_entry(&t2);
+			/* ∀j xj=ximin -> D += "-xi" */
+			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
+				for (i = 0; i < nparent; ++i)
+					if (tp[i].entry.mode & S_IFXMIN_NEQ)
+						goto skip_emit_tp;
+			}
+
+			p = emit_path(p, base, opt, nparent,
+					/*t=*/NULL, tp, imin);
+
+		skip_emit_tp:
+			/* ∀ xk=ximin  xk↓ */
+			update_tp_entries(tp, nparent);
 		}
 	}
 
-	free(t2tree);
-	free(t1tree);
-	return 0;
+	free(ttree);
+	for (i = nparent-1; i >= 0; i--)
+		free(tptree[i]);
+	xalloca_free(tptree);
+	xalloca_free(tp);
+
+	return p;
+}
+
+struct combine_diff_path *diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parents_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt)
+{
+	p = ll_diff_tree_paths(p, sha1, parents_sha1, nparent, base, opt);
+
+	/*
+	 * free pre-allocated last element, if any
+	 * (see path_appendnew() for details about why)
+	 */
+	if (p->next) {
+		free(p->next);
+		p->next = NULL;
+	}
+
+	return p;
 }
 
 /*
@@ -308,6 +664,27 @@ static void try_to_follow_renames(const unsigned char *old, const unsigned char
 	q->nr = 1;
 }
 
+static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
+			     struct strbuf *base, struct diff_options *opt)
+{
+	struct combine_diff_path phead, *p;
+	const unsigned char *parents_sha1[1] = {old};
+	pathchange_fn_t pathchange_old = opt->pathchange;
+
+	phead.next = NULL;
+	opt->pathchange = emit_diff_first_parent_only;
+	diff_tree_paths(&phead, new, parents_sha1, 1, base, opt);
+
+	for (p = phead.next; p;) {
+		struct combine_diff_path *pprev = p;
+		p = p->next;
+		free(pprev);
+	}
+
+	opt->pathchange = pathchange_old;
+	return 0;
+}
+
 int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base_str, struct diff_options *opt)
 {
 	struct strbuf base;
-- 
1.9.rc0.143.g6fd479e

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Kirill Smelkov]

On Wed, Mar 26, 2014 at 02:34:24PM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@navytux.spb.ru> writes:
> 
> > On Tue, Mar 25, 2014 at 10:46:32AM -0700, Junio C Hamano wrote:
> >> Kirill Smelkov <kirr@navytux.spb.ru> writes:
> >> 
> >> > What are the downsides of "__" prefix by the way?
> >> 
> >> Aren't these names reserved for compiler/runtime implementations?
> >
> > Yes, but there are precedents when people don't obey it widely and
> > in practice everything works :)
> 
> I think you are alluding to the practice in the Linux kernel, but
> their requirement is vastly different---their product do not even
> link with libc and they always compile with specific selected
> versions of gcc, no?

Yes, that is correct. Only "__" was so visually appealing that there was
a temptation to break the rules, but...


> > Let it be something portable anyway -
> > how about diff_tree_sha1_low() ?
> 
> Sure.
> 
> As this is a file-scope static, I do not think the exact naming
> matters that much.  Just FYI, we seem to use ll_ prefix (standing
> for low-level) in some places.

... let's then use this "ll_" prefix scheme for consistency.

Corrected patch is below, and I've sent corrections to follow-up
patches as well.

Thanks,
Kirill

(please keep author email)
---- 8< ----
From: Kirill Smelkov <kirr@mns.spb.ru>
Date: Mon, 24 Feb 2014 20:21:46 +0400
Subject: [PATCH v3a] tree-diff: rework diff_tree interface to be sha1 based

In the next commit this will allow to reduce intermediate calls, when
recursing into subtrees - at that stage we know only subtree sha1, and
it is natural for tree walker to start from that phase. For now we do

    diff_tree
        show_path
            diff_tree_sha1
                diff_tree
                    ...

and the change will allow to reduce it to

    diff_tree
        show_path
            diff_tree

Also, it will allow to omit allocating strbuf for each subtree, and just
reuse the common strbuf via playing with its len.

The above-mentioned improvements go in the next 2 patches.

The downside is that try_to_follow_renames(), if active, we cause
re-reading of 2 initial trees, which was negligible based on my timings,
and which is outweighed cogently by the upsides.

NOTE To keep with the current interface and semantics, I needed to
rename the function from diff_tree() to diff_tree_sha1(). As
diff_tree_sha1() was already used, and the function we are talking here
is its more low-level helper, let's use convention for prefixing
such helpers with "ll_". So the final renaming is

    diff_tree() -> ll_diff_tree_sha1()

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---

Changes since v3:

 - further rename diff_tree_sha1_low() -> ll_diff_tree_sha1() to follow Git
   style for naming low-level helpers.

Changes since v2:

 - renamed __diff_tree_sha1() -> diff_tree_sha1_low() as the former
   overlaps with reserved-for-implementation identifiers namespace.

Changes since v1:

 - don't need to touch diff.h, as diff_tree() became static.


 tree-diff.c | 60 ++++++++++++++++++++++++++++--------------------------------
 1 file changed, 28 insertions(+), 32 deletions(-)

diff --git a/tree-diff.c b/tree-diff.c
index f137f39..1d02e43 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -141,12 +141,17 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
 	}
 }
 
-static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
-		     const char *base_str, struct diff_options *opt)
+static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
+			     const char *base_str, struct diff_options *opt)
 {
+	struct tree_desc t1, t2;
+	void *t1tree, *t2tree;
 	struct strbuf base;
 	int baselen = strlen(base_str);
 
+	t1tree = fill_tree_descriptor(&t1, old);
+	t2tree = fill_tree_descriptor(&t2, new);
+
 	/* Enable recursion indefinitely */
 	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
@@ -159,39 +164,41 @@ static int diff_tree(struct tree_desc *t1, struct tree_desc *t2,
 		if (diff_can_quit_early(opt))
 			break;
 		if (opt->pathspec.nr) {
-			skip_uninteresting(t1, &base, opt);
-			skip_uninteresting(t2, &base, opt);
+			skip_uninteresting(&t1, &base, opt);
+			skip_uninteresting(&t2, &base, opt);
 		}
-		if (!t1->size && !t2->size)
+		if (!t1.size && !t2.size)
 			break;
 
-		cmp = tree_entry_pathcmp(t1, t2);
+		cmp = tree_entry_pathcmp(&t1, &t2);
 
 		/* t1 = t2 */
 		if (cmp == 0) {
 			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
-			    hashcmp(t1->entry.sha1, t2->entry.sha1) ||
-			    (t1->entry.mode != t2->entry.mode))
-				show_path(&base, opt, t1, t2);
+			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
+			    (t1.entry.mode != t2.entry.mode))
+				show_path(&base, opt, &t1, &t2);
 
-			update_tree_entry(t1);
-			update_tree_entry(t2);
+			update_tree_entry(&t1);
+			update_tree_entry(&t2);
 		}
 
 		/* t1 < t2 */
 		else if (cmp < 0) {
-			show_path(&base, opt, t1, /*t2=*/NULL);
-			update_tree_entry(t1);
+			show_path(&base, opt, &t1, /*t2=*/NULL);
+			update_tree_entry(&t1);
 		}
 
 		/* t1 > t2 */
 		else {
-			show_path(&base, opt, /*t1=*/NULL, t2);
-			update_tree_entry(t2);
+			show_path(&base, opt, /*t1=*/NULL, &t2);
+			update_tree_entry(&t2);
 		}
 	}
 
 	strbuf_release(&base);
+	free(t2tree);
+	free(t1tree);
 	return 0;
 }
 
@@ -206,7 +213,7 @@ static inline int diff_might_be_rename(void)
 		!DIFF_FILE_VALID(diff_queued_diff.queue[0]->one);
 }
 
-static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, const char *base, struct diff_options *opt)
+static void try_to_follow_renames(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
 {
 	struct diff_options diff_opts;
 	struct diff_queue_struct *q = &diff_queued_diff;
@@ -244,7 +251,7 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
 	diff_opts.break_opt = opt->break_opt;
 	diff_opts.rename_score = opt->rename_score;
 	diff_setup_done(&diff_opts);
-	diff_tree(t1, t2, base, &diff_opts);
+	ll_diff_tree_sha1(old, new, base, &diff_opts);
 	diffcore_std(&diff_opts);
 	free_pathspec(&diff_opts.pathspec);
 
@@ -305,23 +312,12 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
 
 int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
 {
-	void *tree1, *tree2;
-	struct tree_desc t1, t2;
-	unsigned long size1, size2;
 	int retval;
 
-	tree1 = fill_tree_descriptor(&t1, old);
-	tree2 = fill_tree_descriptor(&t2, new);
-	size1 = t1.size;
-	size2 = t2.size;
-	retval = diff_tree(&t1, &t2, base, opt);
-	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename()) {
-		init_tree_desc(&t1, tree1, size1);
-		init_tree_desc(&t2, tree2, size2);
-		try_to_follow_renames(&t1, &t2, base, opt);
-	}
-	free(tree1);
-	free(tree2);
+	retval = ll_diff_tree_sha1(old, new, base, opt);
+	if (!*base && DIFF_OPT_TST(opt, FOLLOW_RENAMES) && diff_might_be_rename())
+		try_to_follow_renames(old, new, base, opt);
+
 	return retval;
 }
 
-- 
1.9.rc0.143.g6fd479e

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Junio C Hamano]

Kirill Smelkov <kirr@navytux.spb.ru> writes:

> (please keep author email)
> ---- 8< ----
> From: Kirill Smelkov <kirr@mns.spb.ru>
> Date: Mon, 24 Feb 2014 20:21:46 +0400
> Subject: [PATCH v3a] tree-diff: rework diff_tree interface to be sha1 based

"git am -c" will discard everything above the scissors and then
start parsing the in-body headers from there, so the above From:
will be used.

But you have a few entries in .mailmap; do you want to update them
as well?

By the way, in general I do not appreciate people lying on the Date:
with an in-body header in their patches, either in the original or
in rerolls.

Thanks.

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Kirill Smelkov]

+stefanbeller

On Thu, Mar 27, 2014 at 11:48:11AM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@navytux.spb.ru> writes:
> 
> > (please keep author email)
> > ---- 8< ----
> > From: Kirill Smelkov <kirr@mns.spb.ru>
> > Date: Mon, 24 Feb 2014 20:21:46 +0400
> > Subject: [PATCH v3a] tree-diff: rework diff_tree interface to be sha1 based
> 
> "git am -c" will discard everything above the scissors and then
> start parsing the in-body headers from there, so the above From:
> will be used.

Thanks.

> But you have a few entries in .mailmap; do you want to update them
> as well?

When Stefan Beller was contacting me on emails, if I recall correctly, I
told him all those kirr@... entries are mine, but the one this patch is
authored with indicates that something was done at work, and I'd prefer to
acknowledge that. So maybe

---- 8< ----
From: Kirill Smelkov <kirr@navytux.spb.ru>
Date: Thu, 27 Mar 2014 23:32:14 +0400
Subject: [PATCH] .mailmap: Separate Kirill Smelkov personal and work addresses

The address kirr@mns.spb.ru indicates that a patch was done at work and
I'd like to acknowledge that.

The address kirr@navytux.spb.ru is my personal email and indicates that
a contribution is done completely on my own time and resources.

kirr@landau.phys.spbu.ru is old university account which no longer works
(sigh, to much spam "because of me" on the server) and maps to
kirr@navytux.spb.ru which should be considered as primary.

Signed-off-by: Kirill Smelkov <kirr@navytux.spb.ru>
---
 .mailmap | 1 -
 1 file changed, 1 deletion(-)

diff --git a/.mailmap b/.mailmap
index 11057cb..0be5e02 100644
--- a/.mailmap
+++ b/.mailmap
@@ -117,7 +117,6 @@ Keith Cascio <keith@CS.UCLA.EDU> <keith@cs.ucla.edu>
 Kent Engstrom <kent@lysator.liu.se>
 Kevin Leung <kevinlsk@gmail.com>
 Kirill Smelkov <kirr@navytux.spb.ru> <kirr@landau.phys.spbu.ru>
-Kirill Smelkov <kirr@navytux.spb.ru> <kirr@mns.spb.ru>
 Knut Franke <Knut.Franke@gmx.de> <k.franke@science-computing.de>
 Lars Doelle <lars.doelle@on-line ! de>
 Lars Doelle <lars.doelle@on-line.de>
-- 
1.9.rc0.143.g6fd479e
---- 8< ----

On the other hand, it is still all me, and the main address (navytux) is
indicated correctly, so I dunno...

> By the way, in general I do not appreciate people lying on the Date:
> with an in-body header in their patches, either in the original or
> in rerolls.
> 
> Thanks.

I see. Somehow it is pity that the date of original work is lost via
this approach, as now we are only changing cosmetics etc, and the bulk
of the work was done earlier.

Anyway, we can drop the date, but please keep the email, as it is used
for the acknowledgment.

Thanks,
Kirill

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Johannes Sixt]

Am 3/27/2014 19:48, schrieb Junio C Hamano:
>> From: Kirill Smelkov <kirr@mns.spb.ru>
>> Date: Mon, 24 Feb 2014 20:21:46 +0400
>> ...
> 
> By the way, in general I do not appreciate people lying on the Date:
> with an in-body header in their patches, either in the original or
> in rerolls.

format-patch is not very cooperative in this aspect. When I prepare a
patch series with format-patch, I find myself editing out the Date: line
from all patches it produces again and again. :-(

-- Hannes

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Junio C Hamano]

Johannes Sixt <j.sixt@viscovery.net> writes:

> Am 3/27/2014 19:48, schrieb Junio C Hamano:
>>> From: Kirill Smelkov <kirr@mns.spb.ru>
>>> Date: Mon, 24 Feb 2014 20:21:46 +0400
>>> ...
>> 
>> By the way, in general I do not appreciate people lying on the Date:
>> with an in-body header in their patches, either in the original or
>> in rerolls.
>
> format-patch is not very cooperative in this aspect. When I prepare a
> patch series with format-patch, I find myself editing out the Date: line
> from all patches it produces again and again. :-(

I am not sure what you mean.  If you are pasting the format-patch
output into an editor your MUA is using to receive the body of the
message from you, you would remove all the non-body lines, not just
Date: but Subject: and From:, no?

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Johannes Sixt]

Am 28.03.2014 18:06, schrieb Junio C Hamano:
> Johannes Sixt <j.sixt@viscovery.net> writes:
> 
>> Am 3/27/2014 19:48, schrieb Junio C Hamano:
>>>> From: Kirill Smelkov <kirr@mns.spb.ru>
>>>> Date: Mon, 24 Feb 2014 20:21:46 +0400
>>>> ...
>>>
>>> By the way, in general I do not appreciate people lying on the Date:
>>> with an in-body header in their patches, either in the original or
>>> in rerolls.
>>
>> format-patch is not very cooperative in this aspect. When I prepare a
>> patch series with format-patch, I find myself editing out the Date: line
>> from all patches it produces again and again. :-(
> 
> I am not sure what you mean.  If you are pasting the format-patch
> output into an editor your MUA is using to receive the body of the
> message from you, you would remove all the non-body lines, not just
> Date: but Subject: and From:, no?

Correct. So I should add that my gripe is about when I want to send a
patch series with git-send-email that was prepared with git-format-patch.

-- Hannes

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

> Am 28.03.2014 18:06, schrieb Junio C Hamano:
>> Johannes Sixt <j.sixt@viscovery.net> writes:
>> 
>>> Am 3/27/2014 19:48, schrieb Junio C Hamano:
>>>>> From: Kirill Smelkov <kirr@mns.spb.ru>
>>>>> Date: Mon, 24 Feb 2014 20:21:46 +0400
>>>>> ...
>>>>
>>>> By the way, in general I do not appreciate people lying on the Date:
>>>> with an in-body header in their patches, either in the original or
>>>> in rerolls.
>>>
>>> format-patch is not very cooperative in this aspect. When I prepare a
>>> patch series with format-patch, I find myself editing out the Date: line
>>> from all patches it produces again and again. :-(
>> 
>> I am not sure what you mean.  If you are pasting the format-patch
>> output into an editor your MUA is using to receive the body of the
>> message from you, you would remove all the non-body lines, not just
>> Date: but Subject: and From:, no?
>
> Correct. So I should add that my gripe is about when I want to send a
> patch series with git-send-email that was prepared with git-format-patch.

Hmph.  Don't you get fresh timestamps for your messages in such a
case, ignoring whatever is at the beginning of the input files?

My reading of git-send-email is:

 * "$time = time - scalar $#files" prepares the initial "timestamp",
   so that running two "git send-email" back to back will give
   timestamps to the series sent out by the first invocation that
   are older than the ones the second series will get;

 * "sub send_message" calls "format_2822_time($time++)" to send the
   first message with that initial "timestamp", incrementing the
   timestamps by 1 second intervals (without having to actually wait
   1 second in between messages) for each patch.

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Johannes Sixt]

Am 28.03.2014 19:36, schrieb Junio C Hamano:
> Johannes Sixt <j6t@kdbg.org> writes:
> 
>> Am 28.03.2014 18:06, schrieb Junio C Hamano:
>>> Johannes Sixt <j.sixt@viscovery.net> writes:
>>>
>>>> Am 3/27/2014 19:48, schrieb Junio C Hamano:
>>>>>> From: Kirill Smelkov <kirr@mns.spb.ru>
>>>>>> Date: Mon, 24 Feb 2014 20:21:46 +0400
>>>>>> ...
>>>>>
>>>>> By the way, in general I do not appreciate people lying on the Date:
>>>>> with an in-body header in their patches, either in the original or
>>>>> in rerolls.
>>>>
>>>> format-patch is not very cooperative in this aspect. When I prepare a
>>>> patch series with format-patch, I find myself editing out the Date: line
>>>> from all patches it produces again and again. :-(
>>>
>>> I am not sure what you mean.  If you are pasting the format-patch
>>> output into an editor your MUA is using to receive the body of the
>>> message from you, you would remove all the non-body lines, not just
>>> Date: but Subject: and From:, no?
>>
>> Correct. So I should add that my gripe is about when I want to send a
>> patch series with git-send-email that was prepared with git-format-patch.
> 
> Hmph.  Don't you get fresh timestamps for your messages in such a
> case, ignoring whatever is at the beginning of the input files?
> 
> My reading of git-send-email is:
> 
>  * "$time = time - scalar $#files" prepares the initial "timestamp",
>    so that running two "git send-email" back to back will give
>    timestamps to the series sent out by the first invocation that
>    are older than the ones the second series will get;
> 
>  * "sub send_message" calls "format_2822_time($time++)" to send the
>    first message with that initial "timestamp", incrementing the
>    timestamps by 1 second intervals (without having to actually wait
>    1 second in between messages) for each patch.

Ah, nice! I didn't know that. I never dared to leave an old author date
(or any date) in the patches, and assumed that it would be kept and
disrupt the email time line.

Thanks for the hint, and sorry for the noise.

-- Hannes

Re: [PATCH v2 14/19] tree-diff: rework diff_tree interface to be sha1 based

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

>> My reading of git-send-email is:
>> 
>>  * "$time = time - scalar $#files" prepares the initial "timestamp",
>>    so that running two "git send-email" back to back will give
>>    timestamps to the series sent out by the first invocation that
>>    are older than the ones the second series will get;

A completely irrelevant tangent, but I was being an idiot here.  The
"-scaler #$files" is not about two send-email running back to back.
A second invocation that sends out a long series will start its
timestamp #$files in the past, that will overlap with the timestamp
of the last one in the first invocation.  And that is not what the
code attempts to address.  It wants to merely avoid timestamps from
the future.

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Junio C Hamano]

Kirill Smelkov <kirr@navytux.spb.ru> writes:

> +extern
> +struct combine_diff_path *diff_tree_paths(

These two on the same line, please.

> +	struct combine_diff_path *p, const unsigned char *sha1,
> +	const unsigned char **parent_sha1, int nparent,
> +	struct strbuf *base, struct diff_options *opt);
>  extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
>  			  const char *base, struct diff_options *opt);
> ...
> +/*
> + * convert path -> opt->diff_*() callbacks
> + *
> + * emits diff to first parent only, and tells diff tree-walker that we are done
> + * with p and it can be freed.
> + */
> +static int emit_diff_first_parent_only(struct diff_options *opt, struct combine_diff_path *p)
>  {

Very straight-forward; good.

> +static struct combine_diff_path *path_appendnew(struct combine_diff_path *last,
> +	int nparent, const struct strbuf *base, const char *path, int pathlen,
> +	unsigned mode, const unsigned char *sha1)
> +{
> +	struct combine_diff_path *p;
> +	int len = base->len + pathlen;
> +	int alloclen = combine_diff_path_size(nparent, len);
> +
> +	/* if last->next is !NULL - it is a pre-allocated memory, we can reuse */
> +	p = last->next;
> +	if (p && (alloclen > (intptr_t)p->next)) {
> +		free(p);
> +		p = NULL;
> +	}
> +
> +	if (!p) {
> +		p = xmalloc(alloclen);
> +
> +		/*
> +		 * until we go to it next round, .next holds how many bytes we
> +		 * allocated (for faster realloc - we don't need copying old data).
> +		 */
> +		p->next = (struct combine_diff_path *)(intptr_t)alloclen;

This reuse of the .next field is somewhat yucky, but it is very
localized inside a function that has a single callsite to this
function, so let's let it pass.

> +static struct combine_diff_path *emit_path(struct combine_diff_path *p,
> +	struct strbuf *base, struct diff_options *opt, int nparent,
> +	struct tree_desc *t, struct tree_desc *tp,
> +	int imin)
>  {

Again, fairly straight-forward and good.

> +/*
> + * generate paths for combined diff D(sha1,parents_sha1[])
> + ...
> +static struct combine_diff_path *ll_diff_tree_paths(
> +	struct combine_diff_path *p, const unsigned char *sha1,
> +	const unsigned char **parents_sha1, int nparent,
> +	struct strbuf *base, struct diff_options *opt)
> +{
> +	struct tree_desc t, *tp;
> +	void *ttree, **tptree;
> +	int i;
> +
> +	tp     = xalloca(nparent * sizeof(tp[0]));
> +	tptree = xalloca(nparent * sizeof(tptree[0]));
> +
> +	/*
> +	 * load parents first, as they are probably already cached.
> +	 *
> +	 * ( log_tree_diff() parses commit->parent before calling here via
> +	 *   diff_tree_sha1(parent, commit) )
> +	 */
> +	for (i = 0; i < nparent; ++i)
> +		tptree[i] = fill_tree_descriptor(&tp[i], parents_sha1[i]);
> +	ttree = fill_tree_descriptor(&t, sha1);
>  
>  	/* Enable recursion indefinitely */
>  	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
>  
>  	for (;;) {
> -		int cmp;
> +		int imin, cmp;
>  
>  		if (diff_can_quit_early(opt))
>  			break;
> +
>  		if (opt->pathspec.nr) {
> -			skip_uninteresting(&t1, base, opt);
> -			skip_uninteresting(&t2, base, opt);
> +			skip_uninteresting(&t, base, opt);
> +			for (i = 0; i < nparent; i++)
> +				skip_uninteresting(&tp[i], base, opt);
>  		}
> -		if (!t1.size && !t2.size)
> -			break;
>  
> -		cmp = tree_entry_pathcmp(&t1, &t2);
> +		/* comparing is finished when all trees are done */
> +		if (!t.size) {
> +			int done = 1;
> +			for (i = 0; i < nparent; ++i)
> +				if (tp[i].size) {
> +					done = 0;
> +					break;
> +				}
> +			if (done)
> +				break;
> +		}
> +
> +		/*
> +		 * lookup imin = argmin(x1...xn),
> +		 * mark entries whether they =tp[imin] along the way
> +		 */
> +		imin = 0;
> +		tp[0].entry.mode &= ~S_IFXMIN_NEQ;
> +
> +		for (i = 1; i < nparent; ++i) {
> +			cmp = tree_entry_pathcmp(&tp[i], &tp[imin]);
> +			if (cmp < 0) {
> +				imin = i;
> +				tp[i].entry.mode &= ~S_IFXMIN_NEQ;
> +			}
> +			else if (cmp == 0) {
> +				tp[i].entry.mode &= ~S_IFXMIN_NEQ;
> +			}
> +			else {
> +				tp[i].entry.mode |= S_IFXMIN_NEQ;
> +			}
> +		}
> +
> +		/* fixup markings for entries before imin */
> +		for (i = 0; i < imin; ++i)
> +			tp[i].entry.mode |= S_IFXMIN_NEQ;	/* x[i] > x[imin] */
> +

These two loop made my reading hiccup for a while.  With these you
are scanning the tp[] array 1.5 times (and doing the bitwise
assignment to entry.mode 1.5 * nparent times), but I suspect it may
have been a lot easier to read if the first loop only identified the
imin, and the second loop only did the entry.mode for _all_ nparents.

> +		/* compare a vs x[imin] */
> +		cmp = tree_entry_pathcmp(&t, &tp[imin]);
> +
> +		/* a = xi */
> +		if (cmp == 0) {
> +			/* are either xk > xi or diff(a,xk) != ø ? */
> +			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
> +				for (i = 0; i < nparent; ++i) {
> +					/* x[i] > x[imin] */
> +					if (tp[i].entry.mode & S_IFXMIN_NEQ)
> +						continue;
> +
> +					/* diff(a,xk) != ø */
> +					if (hashcmp(t.entry.sha1, tp[i].entry.sha1) ||
> +					    (t.entry.mode != tp[i].entry.mode))
> +						continue;
> +
> +					goto skip_emit_t_tp;
> +				}
> +			}

Please bear with me.  The notation scares me as I am not good at math.

In short, the above loop is about:

    We are looking at path in 't' and some parents have the same
    path.  If any of these parents have that path with the contents
    identical to 't', then do not emit this path.

which makes sense to me, but these notation also made my reading
hiccup, especially because it is hard to guess what "xk" refers to
(e.g. "any k where 0 <= k < nparent && i != k"? "all such k"?).  I
still haven't figured out what you meant to say with "xk", but I
think I got what the code wants to do.

How does the "the (virtual) path from a tree that has ran out of
entries sorts later than anything else" comparison rule influence
the picture?  A parent that has ran out would have _NEQ bit set and
would not count as having the same contents as the path from 't'.
If 't' has ran out, the only way t and tp[imin] could compare equal
is when tp[imin] has also ran out, but that can happen only when all
the parents are done with, so we would have broken out of the loop
even before we try to figure out imin.  So there is no funnies
there, which is good.

> +			/* D += {δ(a,xk) if xk=xi;  "+a" if xk > xi} */
> +			p = emit_path(p, base, opt, nparent,
> +					&t, tp, imin);
> +
> +		skip_emit_t_tp:
> +			/* a↓,  ∀ xk=ximin  xk↓ */
> +			update_tree_entry(&t);
> +			update_tp_entries(tp, nparent);
>  		}
>  
> -		/* t1 < t2 */
> +		/* a < xi */
>  		else if (cmp < 0) {
> -			show_path(base, opt, &t1, /*t2=*/NULL);
> -			update_tree_entry(&t1);
> +			/* D += "+a" */
> +			p = emit_path(p, base, opt, nparent,
> +					&t, /*tp=*/NULL, -1);
> +
> +			/* a↓ */
> +			update_tree_entry(&t);

This is straight-forward.  No parent has path 't' has, so only the
entry from 't' is given, and we deal with the next entry in 't'
without touching any of the parents in the next iteration.  Good.

>  		}
>  
> -		/* t1 > t2 */
> +		/* a > xi */
>  		else {
> -			show_path(base, opt, /*t1=*/NULL, &t2);
> -			update_tree_entry(&t2);
> +			/* ∀j xj=ximin -> D += "-xi" */

Did you mean "-xj"?

> +			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
> +				for (i = 0; i < nparent; ++i)
> +					if (tp[i].entry.mode & S_IFXMIN_NEQ)
> +						goto skip_emit_tp;
> +			}
> +
> +			p = emit_path(p, base, opt, nparent,
> +					/*t=*/NULL, tp, imin);
> +
> +		skip_emit_tp:
> +			/* ∀ xk=ximin  xk↓ */
> +			update_tp_entries(tp, nparent);

There are parents whose path sort earlier than what is in 't'
(i.e. they were lost in the result---we would want to show
removal).  What makes us jump to the skip label?

    We are looking at path in 't', and some parents have paths that
    sort earlier than that path.  We will not go to skip label if
    any one of the parent's entry sorts after some other parent (or
    the parent in question has ran out its entries), which means we
    show the entry from the parents only when all the parents have
    that same path, which is missing from 't'.

I am not sure if I am reading this correctly, though.

For the two-way diff, the above degenerates to "show all parent
entries that come before the first entry in 't'", which is correct.
For the combined diff, the current intersect_paths() makes sure that
each path appears in all the pair-wise diff between t and tp[],
which again means that the above logic match the current behaviour.


> +struct combine_diff_path *diff_tree_paths(
> +	struct combine_diff_path *p, const unsigned char *sha1,
> +	const unsigned char **parents_sha1, int nparent,
> +	struct strbuf *base, struct diff_options *opt)
> +{
> +	p = ll_diff_tree_paths(p, sha1, parents_sha1, nparent, base, opt);
> +
> +	/*
> +	 * free pre-allocated last element, if any
> +	 * (see path_appendnew() for details about why)
> +	 */
> +	if (p->next) {
> +		free(p->next);
> +		p->next = NULL;
> +	}
> +
> +	return p;
>  }
>  
>  /*
> @@ -308,6 +664,27 @@ static void try_to_follow_renames(const unsigned char *old, const unsigned char
>  	q->nr = 1;
>  }
>  
> +static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
> +			     struct strbuf *base, struct diff_options *opt)
> +{
> +	struct combine_diff_path phead, *p;
> +	const unsigned char *parents_sha1[1] = {old};
> +	pathchange_fn_t pathchange_old = opt->pathchange;
> +
> +	phead.next = NULL;
> +	opt->pathchange = emit_diff_first_parent_only;
> +	diff_tree_paths(&phead, new, parents_sha1, 1, base, opt);

Hmph.  I would have expected

	const unsigned char **parents_sha1 = &old;

or even

	diff_tree_paths(&phead, new, &old, 1, base, opt);

here.


Thanks.

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Kirill Smelkov]

Junio,

First of all thanks a lot for reviewing this patch. I'll reply inline
with corrected version attached in the end.

On Fri, Apr 04, 2014 at 11:42:39AM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@navytux.spb.ru> writes:
> 
> > +extern
> > +struct combine_diff_path *diff_tree_paths(
> 
> These two on the same line, please.

Ok

> > +	struct combine_diff_path *p, const unsigned char *sha1,
> > +	const unsigned char **parent_sha1, int nparent,
> > +	struct strbuf *base, struct diff_options *opt);
> >  extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
> >  			  const char *base, struct diff_options *opt);
> > ...
> > +/*
> > + * convert path -> opt->diff_*() callbacks
> > + *
> > + * emits diff to first parent only, and tells diff tree-walker that we are done
> > + * with p and it can be freed.
> > + */
> > +static int emit_diff_first_parent_only(struct diff_options *opt, struct combine_diff_path *p)
> >  {
> 
> Very straight-forward; good.

Thanks

> > +static struct combine_diff_path *path_appendnew(struct combine_diff_path *last,
> > +	int nparent, const struct strbuf *base, const char *path, int pathlen,
> > +	unsigned mode, const unsigned char *sha1)
> > +{
> > +	struct combine_diff_path *p;
> > +	int len = base->len + pathlen;
> > +	int alloclen = combine_diff_path_size(nparent, len);
> > +
> > +	/* if last->next is !NULL - it is a pre-allocated memory, we can reuse */
> > +	p = last->next;
> > +	if (p && (alloclen > (intptr_t)p->next)) {
> > +		free(p);
> > +		p = NULL;
> > +	}
> > +
> > +	if (!p) {
> > +		p = xmalloc(alloclen);
> > +
> > +		/*
> > +		 * until we go to it next round, .next holds how many bytes we
> > +		 * allocated (for faster realloc - we don't need copying old data).
> > +		 */
> > +		p->next = (struct combine_diff_path *)(intptr_t)alloclen;
> 
> This reuse of the .next field is somewhat yucky, but it is very
> localized inside a function that has a single callsite to this
> function, so let's let it pass.

I agree it is not pretty, but it was the best approach I could find
for avoiding memory re-allocation without introducing new fields into
`struct combine_diff_path`. And yes, the trick is localized, so let's
let it live.


> > +static struct combine_diff_path *emit_path(struct combine_diff_path *p,
> > +	struct strbuf *base, struct diff_options *opt, int nparent,
> > +	struct tree_desc *t, struct tree_desc *tp,
> > +	int imin)
> >  {
> 
> Again, fairly straight-forward and good.

Thanks again.


> > +/*
> > + * generate paths for combined diff D(sha1,parents_sha1[])
> > + ...
> > +static struct combine_diff_path *ll_diff_tree_paths(
> > +	struct combine_diff_path *p, const unsigned char *sha1,
> > +	const unsigned char **parents_sha1, int nparent,
> > +	struct strbuf *base, struct diff_options *opt)
> > +{
> > +	struct tree_desc t, *tp;
> > +	void *ttree, **tptree;
> > +	int i;
> > +
> > +	tp     = xalloca(nparent * sizeof(tp[0]));
> > +	tptree = xalloca(nparent * sizeof(tptree[0]));
> > +
> > +	/*
> > +	 * load parents first, as they are probably already cached.
> > +	 *
> > +	 * ( log_tree_diff() parses commit->parent before calling here via
> > +	 *   diff_tree_sha1(parent, commit) )
> > +	 */
> > +	for (i = 0; i < nparent; ++i)
> > +		tptree[i] = fill_tree_descriptor(&tp[i], parents_sha1[i]);
> > +	ttree = fill_tree_descriptor(&t, sha1);
> >  
> >  	/* Enable recursion indefinitely */
> >  	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
> >  
> >  	for (;;) {
> > -		int cmp;
> > +		int imin, cmp;
> >  
> >  		if (diff_can_quit_early(opt))
> >  			break;
> > +
> >  		if (opt->pathspec.nr) {
> > -			skip_uninteresting(&t1, base, opt);
> > -			skip_uninteresting(&t2, base, opt);
> > +			skip_uninteresting(&t, base, opt);
> > +			for (i = 0; i < nparent; i++)
> > +				skip_uninteresting(&tp[i], base, opt);
> >  		}
> > -		if (!t1.size && !t2.size)
> > -			break;
> >  
> > -		cmp = tree_entry_pathcmp(&t1, &t2);
> > +		/* comparing is finished when all trees are done */
> > +		if (!t.size) {
> > +			int done = 1;
> > +			for (i = 0; i < nparent; ++i)
> > +				if (tp[i].size) {
> > +					done = 0;
> > +					break;
> > +				}
> > +			if (done)
> > +				break;
> > +		}
> > +
> > +		/*
> > +		 * lookup imin = argmin(x1...xn),
> > +		 * mark entries whether they =tp[imin] along the way
> > +		 */
> > +		imin = 0;
> > +		tp[0].entry.mode &= ~S_IFXMIN_NEQ;
> > +
> > +		for (i = 1; i < nparent; ++i) {
> > +			cmp = tree_entry_pathcmp(&tp[i], &tp[imin]);
> > +			if (cmp < 0) {
> > +				imin = i;
> > +				tp[i].entry.mode &= ~S_IFXMIN_NEQ;
> > +			}
> > +			else if (cmp == 0) {
> > +				tp[i].entry.mode &= ~S_IFXMIN_NEQ;
> > +			}
> > +			else {
> > +				tp[i].entry.mode |= S_IFXMIN_NEQ;
> > +			}
> > +		}
> > +
> > +		/* fixup markings for entries before imin */
> > +		for (i = 0; i < imin; ++i)
> > +			tp[i].entry.mode |= S_IFXMIN_NEQ;	/* x[i] > x[imin] */
> > +
> 
> These two loop made my reading hiccup for a while.  With these you
> are scanning the tp[] array 1.5 times (and doing the bitwise
> assignment to entry.mode 1.5 * nparent times), but I suspect it may
> have been a lot easier to read if the first loop only identified the
> imin, and the second loop only did the entry.mode for _all_ nparents.

Hmm, if in the first loop, we identify imin only, then in the second
loop we would have to call tree_entry_pathcmp(tp[i], tp[imin]) again,
which, in my view, would be not better and even worse - in the original
case we are scanning the parents nparent times comparing paths, and only
then do simple fixup up-to imin entry.

The following

---- 8< ---
                /* lookup imin = argmin(p1...pn) */
                imin = 0;
                for (i = 1; i < nparent; ++i) {
                        cmp = tree_entry_pathcmp(&tp[i], &tp[imin]);
                        if (cmp < 0) 
                                imin = i;
                }

                /* mark entries whether they =p[imin] */
                for (i = 0; i < nparent; ++i) {
                        cmp = tree_entry_pathcmp(&tp[i], &tp[imin]);
                        if (cmp)
                                tp[i].entry.mode |= S_IFXMIN_NEQ;       
                        else
                                tp[i].entry.mode &= S_IFXMIN_NEQ;
                }
---- 8< ----

maybe looks a bit simpler, but calls tree_entry_pathcmp twice more times.

Besides for important nparent=1 case we were not calling
tree_entry_pathcmp at all and here we'll call it once, which would slow
execution down a bit, as base_name_compare shows measurable enough in profile.
To avoid that we'll need to add 'if (i==imin) continue' and this won't
be so simple then. And for general nparent case, as I've said, we'll be
calling tree_entry_pathcmp twice more times...

Because of all that I'd suggest to go with my original version.


> > +		/* compare a vs x[imin] */
> > +		cmp = tree_entry_pathcmp(&t, &tp[imin]);
> > +
> > +		/* a = xi */
> > +		if (cmp == 0) {
> > +			/* are either xk > xi or diff(a,xk) != ø ? */
> > +			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
> > +				for (i = 0; i < nparent; ++i) {
> > +					/* x[i] > x[imin] */
> > +					if (tp[i].entry.mode & S_IFXMIN_NEQ)
> > +						continue;
> > +
> > +					/* diff(a,xk) != ø */
> > +					if (hashcmp(t.entry.sha1, tp[i].entry.sha1) ||
> > +					    (t.entry.mode != tp[i].entry.mode))
> > +						continue;
> > +
> > +					goto skip_emit_t_tp;
> > +				}
> > +			}
> 
> Please bear with me.  The notation scares me as I am not good at math.
> 
> In short, the above loop is about:
> 
>     We are looking at path in 't' and some parents have the same
>     path.  If any of these parents have that path with the contents
>     identical to 't', then do not emit this path.

Yes, correct.

> which makes sense to me, but these notation also made my reading
> hiccup, especially because it is hard to guess what "xk" refers to
> (e.g. "any k where 0 <= k < nparent && i != k"? "all such k"?).  I
> still haven't figured out what you meant to say with "xk", but I
> think I got what the code wants to do.

Sorry about scaring you and about hiccup. After some break on the topic,
with a fresh eye I see a lot of confusion goes from the notation I've
chosen initially (because of how I was reasoning about it on paper, when
it was in flux) - i.e. xi for x[imin] and also using i as looping
variable. And also because xi was already used for x[imin] I've used
another letter 'k' denoting all other x'es, which leads to confusion...


I propose we do the following renaming to clarify things:

    A/a     ->      T/t     (to match resulting tree t name in the code)
    X/x     ->      P/p     (to match parents trees tp in the code)
    i       ->      imin    (so that i would be free for other tasks)

then the above (with a prologue) would look like

---- 8< ----
 *       T     P1       Pn
 *       -     -        -
 *      |t|   |p1|     |pn|
 *      |-|   |--| ... |--|      imin = argmin(p1...pn)
 *      | |   |  |     |  |
 *      |-|   |--|     |--|
 *      |.|   |. |     |. |
 *       .     .        .
 *       .     .        .
 *
 * at any time there could be 3 cases:
 *
 *      1)  t < p[imin];
 *      2)  t > p[imin];
 *      3)  t = p[imin].
 *
 * Schematic deduction of what every case means, and what to do, follows:
 *
 * 1)  t < p[imin]  ->  ∀j t ∉ Pj  ->  "+t" ∈ D(T,Pj)  ->  D += "+t";  t↓
 *
 * 2)  t > p[imin]
 *
 *     2.1) ∃j: pj > p[imin]  ->  "-p[imin]" ∉ D(T,Pj)  ->  D += ø;  ∀ pi=p[imin]  pi↓
 *     2.2) ∀i  pi = p[imin]  ->  pi ∉ T  ->  "-pi" ∈ D(T,Pi)  ->  D += "-p[imin]";  ∀i pi↓
 *
 * 3)  t = p[imin]
 *
 *     3.1) ∃j: pj > p[imin]  ->  "+t" ∈ D(T,Pj)  ->  only pi=p[imin] remains to investigate
 *     3.2) pi = p[imin]  ->  investigate δ(t,pi)
 *      |
 *      |
 *      v
 *
 *     3.1+3.2) looking at δ(t,pi) ∀i: pi=p[imin] - if all != ø  ->
 *
 *                       ⎧δ(t,pi)  - if pi=p[imin]
 *              ->  D += ⎨
 *                       ⎩"+t"     - if pi>p[imin]
 *
 *
 *     in any case t↓  ∀ pi=p[imin]  pi↓

 ...

                /* compare t vs p[imin] */
                cmp = tree_entry_pathcmp(&t, &tp[imin]);

                /* t = p[imin] */
                if (cmp == 0) {
                        /* are either pi > p[imin] or diff(t,pi) != ø ? */
                        if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
                                for (i = 0; i < nparent; ++i) {
                                        /* p[i] > p[imin] */
                                        if (tp[i].entry.mode & S_IFXMIN_NEQ)
                                                continue;

                                        /* diff(t,pi) != ø */
                                        if (hashcmp(t.entry.sha1, tp[i].entry.sha1) ||
                                            (t.entry.mode != tp[i].entry.mode))
                                                continue;

                                        goto skip_emit_t_tp;
                                }
                        }

                        /* D += {δ(t,pi) if pi=p[imin];  "+a" if pi > p[imin]} */
                        p = emit_path(p, base, opt, nparent,
                                        &t, tp, imin);

                skip_emit_t_tp:
                        /* t↓,  ∀ pi=p[imin]  pi↓ */
                        update_tree_entry(&t);
                        update_tp_entries(tp, nparent);
                }
---- 8< ----

now xk is gone and i matches p[i] (= pi) etc so variable names correlate
to algorithm description better.

Does that maybe clarify things?


> How does the "the (virtual) path from a tree that has ran out of
> entries sorts later than anything else" comparison rule influence
> the picture?  A parent that has ran out would have _NEQ bit set and
> would not count as having the same contents as the path from 't'.
> If 't' has ran out, the only way t and tp[imin] could compare equal
> is when tp[imin] has also ran out, but that can happen only when all
> the parents are done with, so we would have broken out of the loop
> even before we try to figure out imin.  So there is no funnies
> there, which is good.

Yes, exactly.


> > +			/* D += {δ(a,xk) if xk=xi;  "+a" if xk > xi} */
> > +			p = emit_path(p, base, opt, nparent,
> > +					&t, tp, imin);
> > +
> > +		skip_emit_t_tp:
> > +			/* a↓,  ∀ xk=ximin  xk↓ */
> > +			update_tree_entry(&t);
> > +			update_tp_entries(tp, nparent);
> >  		}
> >  
> > -		/* t1 < t2 */
> > +		/* a < xi */
> >  		else if (cmp < 0) {
> > -			show_path(base, opt, &t1, /*t2=*/NULL);
> > -			update_tree_entry(&t1);
> > +			/* D += "+a" */
> > +			p = emit_path(p, base, opt, nparent,
> > +					&t, /*tp=*/NULL, -1);
> > +
> > +			/* a↓ */
> > +			update_tree_entry(&t);
> 
> This is straight-forward.  No parent has path 't' has, so only the
> entry from 't' is given, and we deal with the next entry in 't'
> without touching any of the parents in the next iteration.  Good.

Yes. I hope with the renaming it looks a bit more cleaner:

                /* t < p[imin] */
                else if (cmp < 0) {
                        /* D += "+t" */
                        p = emit_path(p, base, opt, nparent,
                                        &t, /*tp=*/NULL, -1);

                        /* t↓ */
                        update_tree_entry(&t);
                }

> 
> >  		}
> >  
> > -		/* t1 > t2 */
> > +		/* a > xi */
> >  		else {
> > -			show_path(base, opt, /*t1=*/NULL, &t2);
> > -			update_tree_entry(&t2);
> > +			/* ∀j xj=ximin -> D += "-xi" */
> 
> Did you mean "-xj"?

No, ximin, which was denoted in the earlier iterations of patch as xi -
if all parents current paths are equal and t does not have this path -
remove the path present in parents. It does not strictly differs between
ximin and xj here, but xj is looping so ximin is better to have as some
defined path.

With the renaming the code looks like this

---- 8< ----
 * 2)  t > p[imin]
 *
 *     2.1) ∃j: pj > p[imin]  ->  "-p[imin]" ∉ D(T,Pj)  ->  D += ø;  ∀ pi=p[imin]  pi↓
 *     2.2) ∀i  pi = p[imin]  ->  pi ∉ T  ->  "-pi" ∈ D(T,Pi)  ->  D += "-p[imin]";  ∀i pi↓
 ...
                /* t > p[imin] */
                else {
                        /* ∀i pi=p[imin] -> D += "-p[imin]" */
                        if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
                                for (i = 0; i < nparent; ++i)
                                        if (tp[i].entry.mode & S_IFXMIN_NEQ)
                                                goto skip_emit_tp;
                        }

                        p = emit_path(p, base, opt, nparent,
                                        /*t=*/NULL, tp, imin);

                skip_emit_tp:
                        /* ∀ pi=p[imin]  pi↓ */
                        update_tp_entries(tp, nparent);
                }
---- 8< ----

Thanks for spotting it.


> > +			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
> > +				for (i = 0; i < nparent; ++i)
> > +					if (tp[i].entry.mode & S_IFXMIN_NEQ)
> > +						goto skip_emit_tp;
> > +			}
> > +
> > +			p = emit_path(p, base, opt, nparent,
> > +					/*t=*/NULL, tp, imin);
> > +
> > +		skip_emit_tp:
> > +			/* ∀ xk=ximin  xk↓ */
> > +			update_tp_entries(tp, nparent);
> 
> There are parents whose path sort earlier than what is in 't'
> (i.e. they were lost in the result---we would want to show
> removal).  What makes us jump to the skip label?
> 
>     We are looking at path in 't', and some parents have paths that
>     sort earlier than that path.  We will not go to skip label if
>     any one of the parent's entry sorts after some other parent (or
>     the parent in question has ran out its entries), which means we
>     show the entry from the parents only when all the parents have
>     that same path, which is missing from 't'.
> 
> I am not sure if I am reading this correctly, though.
> 
> For the two-way diff, the above degenerates to "show all parent
> entries that come before the first entry in 't'", which is correct.
> For the combined diff, the current intersect_paths() makes sure that
> each path appears in all the pair-wise diff between t and tp[],
> which again means that the above logic match the current behaviour.

Yes, correct (modulo we *will* go to skip label if any one of the
parent's entry sorts after some other parent). By definition of combined
diff we show a path only if it shows in every diff D(T,Pi), and if 

    2.1) ∃j: pj > p[imin]  ->  "-p[imin]" ∉ D(T,Pj)  ->  D += ø;  ∀ pi=p[imin]  pi↓

some pj sorts after p[imin] that would mean that Pj does not have
p[imin] and since t > p[imin] (which means T does not have p[imin]
either) diff D(T,Pj) does not have p[imin]. And because of that we know
the whole combined-diff will not have p[imin] as, by definition,
combined diff is sets intersection and one of the sets does not have
that path.

  ( In usual words p[imin] is not changed between Pj..T - it was
    e.g. removed in Pj~, so merging parents to T does not bring any new
    information wrt path p[imin] and that is why we do not want to show
    p[imin] in combined-diff output - no new change about that path )

So nothing to append to the output, and update minimum tree entries,
preparing for the next step.

> > +struct combine_diff_path *diff_tree_paths(
> > +	struct combine_diff_path *p, const unsigned char *sha1,
> > +	const unsigned char **parents_sha1, int nparent,
> > +	struct strbuf *base, struct diff_options *opt)
> > +{
> > +	p = ll_diff_tree_paths(p, sha1, parents_sha1, nparent, base, opt);
> > +
> > +	/*
> > +	 * free pre-allocated last element, if any
> > +	 * (see path_appendnew() for details about why)
> > +	 */
> > +	if (p->next) {
> > +		free(p->next);
> > +		p->next = NULL;
> > +	}
> > +
> > +	return p;
> >  }
> >  
> >  /*
> > @@ -308,6 +664,27 @@ static void try_to_follow_renames(const unsigned char *old, const unsigned char
> >  	q->nr = 1;
> >  }
> >  
> > +static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
> > +			     struct strbuf *base, struct diff_options *opt)
> > +{
> > +	struct combine_diff_path phead, *p;
> > +	const unsigned char *parents_sha1[1] = {old};
> > +	pathchange_fn_t pathchange_old = opt->pathchange;
> > +
> > +	phead.next = NULL;
> > +	opt->pathchange = emit_diff_first_parent_only;
> > +	diff_tree_paths(&phead, new, parents_sha1, 1, base, opt);
> 
> Hmph.  I would have expected
> 
> 	const unsigned char **parents_sha1 = &old;
> 
> or even
> 
> 	diff_tree_paths(&phead, new, &old, 1, base, opt);
> 
> here.

I agree, the last one is better - thanks for spotting this.

I'm attaching corrected patch with renaming and fixups with smaller
issues you've mentioned.

Thanks,
Kirill

P.S. Sorry for maybe some crept-in mistakes - I've tried to verify it
thoroughly, but am too sleepy to be completely sure. On the other hand I
think and hope the patch should be ok.

---- 8< ----
From: Kirill Smelkov <kirr@mns.spb.ru>
Subject: [PATCH] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

Previously diff_tree(), which is now named ll_diff_tree_sha1(), was
generating diff_filepair(s) for two trees t1 and t2, and that was
usually used for a commit as t1=HEAD~, and t2=HEAD - i.e. to see changes
a commit introduces.

In Git, however, we have fundamentally built flexibility in that a
commit can have many parents - 1 for a plain commit, 2 for a simple merge,
but also more than 2 for merging several heads at once.

For merges there is a so called combine-diff, which shows diff, a merge
introduces by itself, omitting changes done by any parent. That works
through first finding paths, that are different to all parents, and then
showing generalized diff, with separate columns for +/- for each parent.
The code lives in combine-diff.c .

There is an impedance mismatch, however, in that a commit could
generally have any number of parents, and that while diffing trees, we
divide cases for 2-tree diffs and more-than-2-tree diffs. I mean there
is no special casing for multiple parents commits in e.g.
revision-walker .

That impedance mismatch *hurts* *performance* *badly* for generating
combined diffs - in "combine-diff: optimize combine_diff_path
sets intersection" I've already removed some slowness from it, but from
the timings provided there, it could be seen, that combined diffs still
cost more than an order of magnitude more cpu time, compared to diff for
usual commits, and that would only be an optimistic estimate, if we take
into account that for e.g. linux.git there is only one merge for several
dozens of plain commits.

That slowness comes from the fact that currently, while generating
combined diff, a lot of time is spent computing diff(commit,commit^2)
just to only then intersect that huge diff to almost small set of files
from diff(commit,commit^1).

That's because at present, to compute combine-diff, for first finding
paths, that "every parent touches", we use the following combine-diff
property/definition:

D(A,P1...Pn) = D(A,P1) ^ ... ^ D(A,Pn)      (w.r.t. paths)

where

D(A,P1...Pn) is combined diff between commit A, and parents Pi

and

D(A,Pi) is usual two-tree diff Pi..A

So if any of that D(A,Pi) is huge, tracting 1 n-parent combine-diff as n
1-parent diffs and intersecting results will be slow.

And usually, for linux.git and other topic-based workflows, that
D(A,P2) is huge, because, if merge-base of A and P2, is several dozens
of merges (from A, via first parent) below, that D(A,P2) will be diffing
sum of merges from several subsystems to 1 subsystem.

The solution is to avoid computing n 1-parent diffs, and to find
changed-to-all-parents paths via scanning A's and all Pi's trees
simultaneously, at each step comparing their entries, and based on that
comparison, populate paths result, and deduce we could *skip*
*recursing* into subdirectories, if at least for 1 parent, sha1 of that
dir tree is the same as in A. That would save us from doing significant
amount of needless work.

Such approach is very similar to what diff_tree() does, only there we
deal with scanning only 2 trees simultaneously, and for n+1 tree, the
logic is a bit more complex:

D(T,P1...Pn) calculation scheme
-------------------------------

D(T,P1...Pn) = D(T,P1) ^ ... ^ D(T,Pn)	(regarding resulting paths set)

    D(T,Pj)		- diff between T..Pj
    D(T,P1...Pn)	- combined diff from T to parents P1,...,Pn

We start from all trees, which are sorted, and compare their entries in
lock-step:

     T     P1       Pn
     -     -        -
    |t|   |p1|     |pn|
    |-|   |--| ... |--|      imin = argmin(p1...pn)
    | |   |  |     |  |
    |-|   |--|     |--|
    |.|   |. |     |. |
     .     .        .
     .     .        .

at any time there could be 3 cases:

    1)  t < p[imin];
    2)  t > p[imin];
    3)  t = p[imin].

Schematic deduction of what every case means, and what to do, follows:

1)  t < p[imin]  ->  ∀j t ∉ Pj  ->  "+t" ∈ D(T,Pj)  ->  D += "+t";  t↓

2)  t > p[imin]

    2.1) ∃j: pj > p[imin]  ->  "-p[imin]" ∉ D(T,Pj)  ->  D += ø;  ∀ pi=p[imin]  pi↓
    2.2) ∀i  pi = p[imin]  ->  pi ∉ T  ->  "-pi" ∈ D(T,Pi)  ->  D += "-p[imin]";  ∀i pi↓

3)  t = p[imin]

    3.1) ∃j: pj > p[imin]  ->  "+t" ∈ D(T,Pj)  ->  only pi=p[imin] remains to investigate
    3.2) pi = p[imin]  ->  investigate δ(t,pi)
     |
     |
     v

    3.1+3.2) looking at δ(t,pi) ∀i: pi=p[imin] - if all != ø  ->

                      ⎧δ(t,pi)  - if pi=p[imin]
             ->  D += ⎨
                      ⎩"+t"     - if pi>p[imin]

    in any case t↓  ∀ pi=p[imin]  pi↓

~

For comparison, here is how diff_tree() works:

D(A,B) calculation scheme
-------------------------

    A     B
    -     -
   |a|   |b|    a < b   ->  a ∉ B   ->   D(A,B) +=  +a    a↓
   |-|   |-|    a > b   ->  b ∉ A   ->   D(A,B) +=  -b    b↓
   | |   | |    a = b   ->  investigate δ(a,b)            a↓ b↓
   |-|   |-|
   |.|   |.|
    .     .
    .     .

~~~~~~~~

This patch generalizes diff tree-walker to work with arbitrary number of
parents as described above - i.e. now there is a resulting tree t, and
some parents trees tp[i] i=[0..nparent). The generalization builds on
the fact that usual diff

D(A,B)

is by definition the same as combined diff

D(A,[B]),

so if we could rework the code for common case and make it be not slower
for nparent=1 case, usual diff(t1,t2) generation will not be slower, and
multiparent diff tree-walker would greatly benefit generating
combine-diff.

What we do is as follows:

1) diff tree-walker ll_diff_tree_sha1() is internally reworked to be
   a paths generator (new name diff_tree_paths()), with each generated path
   being `struct combine_diff_path` with info for path, new sha1,mode and for
   every parent which sha1,mode it was in it.

2) From that info, we can still generate usual diff queue with
   struct diff_filepairs, via "exporting" generated
   combine_diff_path, if we know we run for nparent=1 case.
   (see emit_diff() which is now named emit_diff_first_parent_only())

3) In order for diff_can_quit_early(), which checks

       DIFF_OPT_TST(opt, HAS_CHANGES))

   to work, that exporting have to be happening not in bulk, but
   incrementally, one diff path at a time.

   For such consumers, there is a new callback in diff_options
   introduced:

       ->pathchange(opt, struct combine_diff_path *)

   which, if set to !NULL, is called for every generated path.

   (see new compat ll_diff_tree_sha1() wrapper around new paths
    generator for setup)

4) The paths generation itself, is reworked from previous
   ll_diff_tree_sha1() code according to "D(A,P1...Pn) calculation
   scheme" provided above:

   On the start we allocate [nparent] arrays in place what was
   earlier just for one parent tree.

   then we just generalize loops, and comparison according to the
   algorithm.

Some notes(*):

1) alloca(), for small arrays, is used for "runs not slower for
   nparent=1 case than before" goal - if we change it to xmalloc()/free()
   the timings get ~1% worse. For alloca() we use just-introduced
   xalloca/xalloca_free compatibility wrappers, so it should not be a
   portability problem.

2) For every parent tree, we need to keep a tag, whether entry from that
   parent equals to entry from minimal parent. For performance reasons I'm
   keeping that tag in entry's mode field in unused bit - see S_IFXMIN_NEQ.
   Not doing so, we'd need to alloca another [nparent] array, which hurts
   performance.

3) For emitted paths, memory could be reused, if we know the path was
   processed via callback and will not be needed later. We use efficient
   hand-made realloc-style path_appendnew(), that saves us from ~1-1.5%
   of potential additional slowdown.

4) goto(s) are used in several places, as the code executes a little bit
   faster with lowered register pressure.

Also

- we should now check for FIND_COPIES_HARDER not only when two entries
  names are the same, and their hashes are equal, but also for a case,
  when a path was removed from some of all parents having it.

  The reason is, if we don't, that path won't be emitted at all (see
  "a > xi" case), and we'll just skip it, and FIND_COPIES_HARDER wants
  all paths - with diff or without - to be emitted, to be later analyzed
  for being copies sources.

  The new check is only necessary for nparent >1, as for nparent=1 case
  xmin_eqtotal always =1 =nparent, and a path is always added to diff as
  removal.

~~~~~~~~

Timings for

    # without -c, i.e. testing only nparent=1 case
    `git log --raw --no-abbrev --no-renames`

before and after the patch are as follows:

                navy.git        linux.git v3.10..v3.11

    before      0.611s          1.889s
    after       0.619s          1.907s
    slowdown    1.3%            0.9%

This timings show we did no harm to usual diff(tree1,tree2) generation.
From the table we can see that we actually did ~1% slowdown, but I think
I've "earned" that 1% in the previous patch ("tree-diff: reuse base
str(buf) memory on sub-tree recursion", HEAD~~) so for nparent=1 case,
net timings stays approximately the same.

The output also stayed the same.

(*) If we revert 1)-4) to more usual techniques, for nparent=1 case,
    we'll get ~2-2.5% of additional slowdown, which I've tried to avoid, as
   "do no harm for nparent=1 case" rule.

For linux.git, combined diff will run an order of magnitude faster and
appropriate timings will be provided in the next commit, as we'll be
taking advantage of the new diff tree-walker for combined-diff
generation there.

P.S. and combined diff is not some exotic/for-play-only stuff - for
example for a program I write to represent Git archives as readonly
filesystem, there is initial scan with

    `git log --reverse --raw --no-abbrev --no-renames -c`

to extract log of what was created/changed when, as a result building a
map

    {}  sha1    ->  in which commit (and date) a content was added

that `-c` means also show combined diff for merges, and without them, if
a merge is non-trivial (merges changes from two parents with both having
separate changes to a file), or an evil one, the map will not be full,
i.e. some valid sha1 would be absent from it.

That case was my initial motivation for combined diffs speedup.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---
 cache.h     |  15 ++
 diff.c      |   1 +
 diff.h      |   9 ++
 tree-diff.c | 504 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------
 4 files changed, 465 insertions(+), 64 deletions(-)

diff --git a/cache.h b/cache.h
index dc040fb..e7f5a0c 100644
--- a/cache.h
+++ b/cache.h
@@ -75,6 +75,21 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);
 #define S_ISGITLINK(m)	(((m) & S_IFMT) == S_IFGITLINK)
 
 /*
+ * Some mode bits are also used internally for computations.
+ *
+ * They *must* not overlap with any valid modes, and they *must* not be emitted
+ * to outside world - i.e. appear on disk or network. In other words, it's just
+ * temporary fields, which we internally use, but they have to stay in-house.
+ *
+ * ( such approach is valid, as standard S_IF* fits into 16 bits, and in Git
+ *   codebase mode is `unsigned int` which is assumed to be at least 32 bits )
+ */
+
+/* used internally in tree-diff */
+#define S_DIFFTREE_IFXMIN_NEQ	0x80000000
+
+
+/*
  * Intensive research over the course of many years has shown that
  * port 9418 is totally unused by anything else. Or
  *
diff --git a/diff.c b/diff.c
index 8e4a6a9..cda4aa8 100644
--- a/diff.c
+++ b/diff.c
@@ -3216,6 +3216,7 @@ void diff_setup(struct diff_options *options)
 	options->context = diff_context_default;
 	DIFF_OPT_SET(options, RENAME_EMPTY);
 
+	/* pathchange left =NULL by default */
 	options->change = diff_change;
 	options->add_remove = diff_addremove;
 	options->use_color = diff_use_color_default;
diff --git a/diff.h b/diff.h
index 5d7b9f7..0abd735 100644
--- a/diff.h
+++ b/diff.h
@@ -15,6 +15,10 @@ struct diff_filespec;
 struct userdiff_driver;
 struct sha1_array;
 struct commit;
+struct combine_diff_path;
+
+typedef int (*pathchange_fn_t)(struct diff_options *options,
+		 struct combine_diff_path *path);
 
 typedef void (*change_fn_t)(struct diff_options *options,
 		 unsigned old_mode, unsigned new_mode,
@@ -157,6 +161,7 @@ struct diff_options {
 	int close_file;
 
 	struct pathspec pathspec;
+	pathchange_fn_t pathchange;
 	change_fn_t change;
 	add_remove_fn_t add_remove;
 	diff_format_fn_t format_callback;
@@ -189,6 +194,10 @@ const char *diff_line_prefix(struct diff_options *);
 
 extern const char mime_boundary_leader[];
 
+extern struct combine_diff_path *diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parent_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt);
 extern int diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 			  const char *base, struct diff_options *opt);
 extern int diff_root_tree_sha1(const unsigned char *new, const char *base,
diff --git a/tree-diff.c b/tree-diff.c
index 278acc8..e7b378c 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -6,7 +6,19 @@
 #include "diffcore.h"
 #include "tree.h"
 
+/*
+ * internal mode marker, saying a tree entry != entry of tp[imin]
+ * (see ll_diff_tree_paths for what it means there)
+ *
+ * we will update/use/emit entry for diff only with it unset.
+ */
+#define S_IFXMIN_NEQ	S_DIFFTREE_IFXMIN_NEQ
+
 
+static struct combine_diff_path *ll_diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parents_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt);
 static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
 			     struct strbuf *base, struct diff_options *opt);
 
@@ -42,71 +54,151 @@ static int tree_entry_pathcmp(struct tree_desc *t1, struct tree_desc *t2)
 }
 
 
-/* convert path, t1/t2 -> opt->diff_*() callbacks */
-static void emit_diff(struct diff_options *opt, struct strbuf *path,
-		      struct tree_desc *t1, struct tree_desc *t2)
+/*
+ * convert path -> opt->diff_*() callbacks
+ *
+ * emits diff to first parent only, and tells diff tree-walker that we are done
+ * with p and it can be freed.
+ */
+static int emit_diff_first_parent_only(struct diff_options *opt, struct combine_diff_path *p)
 {
-	unsigned int mode1 = t1 ? t1->entry.mode : 0;
-	unsigned int mode2 = t2 ? t2->entry.mode : 0;
-
-	if (mode1 && mode2) {
-		opt->change(opt, mode1, mode2, t1->entry.sha1, t2->entry.sha1,
-			1, 1, path->buf, 0, 0);
+	struct combine_diff_parent *p0 = &p->parent[0];
+	if (p->mode && p0->mode) {
+		opt->change(opt, p0->mode, p->mode, p0->sha1, p->sha1,
+			1, 1, p->path, 0, 0);
 	}
 	else {
 		const unsigned char *sha1;
 		unsigned int mode;
 		int addremove;
 
-		if (mode2) {
+		if (p->mode) {
 			addremove = '+';
-			sha1 = t2->entry.sha1;
-			mode = mode2;
+			sha1 = p->sha1;
+			mode = p->mode;
 		} else {
 			addremove = '-';
-			sha1 = t1->entry.sha1;
-			mode = mode1;
+			sha1 = p0->sha1;
+			mode = p0->mode;
 		}
 
-		opt->add_remove(opt, addremove, mode, sha1, 1, path->buf, 0);
+		opt->add_remove(opt, addremove, mode, sha1, 1, p->path, 0);
 	}
+
+	return 0;	/* we are done with p */
 }
 
 
-/* new path should be added to diff
+/*
+ * Make a new combine_diff_path from path/mode/sha1
+ * and append it to paths list tail.
+ *
+ * Memory for created elements could be reused:
+ *
+ *	- if last->next == NULL, the memory is allocated;
+ *
+ *	- if last->next != NULL, it is assumed that p=last->next was returned
+ *	  earlier by this function, and p->next was *not* modified.
+ *	  The memory is then reused from p.
+ *
+ * so for clients,
+ *
+ * - if you do need to keep the element
+ *
+ *	p = path_appendnew(p, ...);
+ *	process(p);
+ *	p->next = NULL;
+ *
+ * - if you don't need to keep the element after processing
+ *
+ *	pprev = p;
+ *	p = path_appendnew(p, ...);
+ *	process(p);
+ *	p = pprev;
+ *	; don't forget to free tail->next in the end
+ *
+ * p->parent[] remains uninitialized.
+ */
+static struct combine_diff_path *path_appendnew(struct combine_diff_path *last,
+	int nparent, const struct strbuf *base, const char *path, int pathlen,
+	unsigned mode, const unsigned char *sha1)
+{
+	struct combine_diff_path *p;
+	int len = base->len + pathlen;
+	int alloclen = combine_diff_path_size(nparent, len);
+
+	/* if last->next is !NULL - it is a pre-allocated memory, we can reuse */
+	p = last->next;
+	if (p && (alloclen > (intptr_t)p->next)) {
+		free(p);
+		p = NULL;
+	}
+
+	if (!p) {
+		p = xmalloc(alloclen);
+
+		/*
+		 * until we go to it next round, .next holds how many bytes we
+		 * allocated (for faster realloc - we don't need copying old data).
+		 */
+		p->next = (struct combine_diff_path *)(intptr_t)alloclen;
+	}
+
+	last->next = p;
+
+	p->path = (char *)&(p->parent[nparent]);
+	memcpy(p->path, base->buf, base->len);
+	memcpy(p->path + base->len, path, pathlen);
+	p->path[len] = 0;
+	p->mode = mode;
+	hashcpy(p->sha1, sha1 ? sha1 : null_sha1);
+
+	return p;
+}
+
+/*
+ * new path should be added to combine diff
  *
  * 3 cases on how/when it should be called and behaves:
  *
- *	!t1,  t2	-> path added, parent lacks it
- *	 t1, !t2	-> path removed from parent
- *	 t1,  t2	-> path modified
+ *	 t, !tp		-> path added, all parents lack it
+ *	!t,  tp		-> path removed from all parents
+ *	 t,  tp		-> path modified/added
+ *			   (M for tp[i]=tp[imin], A otherwise)
  */
-static void show_path(struct strbuf *base, struct diff_options *opt,
-		      struct tree_desc *t1, struct tree_desc *t2)
+static struct combine_diff_path *emit_path(struct combine_diff_path *p,
+	struct strbuf *base, struct diff_options *opt, int nparent,
+	struct tree_desc *t, struct tree_desc *tp,
+	int imin)
 {
 	unsigned mode;
 	const char *path;
+	const unsigned char *sha1;
 	int pathlen;
 	int old_baselen = base->len;
-	int isdir, recurse = 0, emitthis = 1;
+	int i, isdir, recurse = 0, emitthis = 1;
 
 	/* at least something has to be valid */
-	assert(t1 || t2);
+	assert(t || tp);
 
-	if (t2) {
+	if (t) {
 		/* path present in resulting tree */
-		tree_entry_extract(t2, &path, &mode);
-		pathlen = tree_entry_len(&t2->entry);
+		sha1 = tree_entry_extract(t, &path, &mode);
+		pathlen = tree_entry_len(&t->entry);
 		isdir = S_ISDIR(mode);
 	} else {
 		/*
-		 * a path was removed - take path from parent. Also take
-		 * mode from parent, to decide on recursion.
+		 * a path was removed - take path from imin parent. Also take
+		 * mode from that parent, to decide on recursion(1).
+		 *
+		 * 1) all modes for tp[i]=tp[imin] should be the same wrt
+		 *    S_ISDIR, thanks to base_name_compare().
 		 */
-		tree_entry_extract(t1, &path, &mode);
-		pathlen = tree_entry_len(&t1->entry);
+		tree_entry_extract(&tp[imin], &path, &mode);
+		pathlen = tree_entry_len(&tp[imin].entry);
 
 		isdir = S_ISDIR(mode);
+		sha1 = NULL;
 		mode = 0;
 	}
 
@@ -115,18 +207,81 @@ static void show_path(struct strbuf *base, struct diff_options *opt,
 		emitthis = DIFF_OPT_TST(opt, TREE_IN_RECURSIVE);
 	}
 
-	strbuf_add(base, path, pathlen);
+	if (emitthis) {
+		int keep;
+		struct combine_diff_path *pprev = p;
+		p = path_appendnew(p, nparent, base, path, pathlen, mode, sha1);
+
+		for (i = 0; i < nparent; ++i) {
+			/*
+			 * tp[i] is valid, if present and if tp[i]==tp[imin] -
+			 * otherwise, we should ignore it.
+			 */
+			int tpi_valid = tp && !(tp[i].entry.mode & S_IFXMIN_NEQ);
+
+			const unsigned char *sha1_i;
+			unsigned mode_i;
+
+			p->parent[i].status =
+				!t ? DIFF_STATUS_DELETED :
+					tpi_valid ?
+						DIFF_STATUS_MODIFIED :
+						DIFF_STATUS_ADDED;
+
+			if (tpi_valid) {
+				sha1_i = tp[i].entry.sha1;
+				mode_i = tp[i].entry.mode;
+			}
+			else {
+				sha1_i = NULL;
+				mode_i = 0;
+			}
+
+			p->parent[i].mode = mode_i;
+			hashcpy(p->parent[i].sha1, sha1_i ? sha1_i : null_sha1);
+		}
 
-	if (emitthis)
-		emit_diff(opt, base, t1, t2);
+		keep = 1;
+		if (opt->pathchange)
+			keep = opt->pathchange(opt, p);
+
+		/*
+		 * If a path was filtered or consumed - we don't need to add it
+		 * to the list and can reuse its memory, leaving it as
+		 * pre-allocated element on the tail.
+		 *
+		 * On the other hand, if path needs to be kept, we need to
+		 * correct its .next to NULL, as it was pre-initialized to how
+		 * much memory was allocated.
+		 *
+		 * see path_appendnew() for details.
+		 */
+		if (!keep)
+			p = pprev;
+		else
+			p->next = NULL;
+	}
 
 	if (recurse) {
+		const unsigned char **parents_sha1;
+
+		parents_sha1 = xalloca(nparent * sizeof(parents_sha1[0]));
+		for (i = 0; i < nparent; ++i) {
+			/* same rule as in emitthis */
+			int tpi_valid = tp && !(tp[i].entry.mode & S_IFXMIN_NEQ);
+
+			parents_sha1[i] = tpi_valid ? tp[i].entry.sha1
+						    : NULL;
+		}
+
+		strbuf_add(base, path, pathlen);
 		strbuf_addch(base, '/');
-		ll_diff_tree_sha1(t1 ? t1->entry.sha1 : NULL,
-				  t2 ? t2->entry.sha1 : NULL, base, opt);
+		p = ll_diff_tree_paths(p, sha1, parents_sha1, nparent, base, opt);
+		xalloca_free(parents_sha1);
 	}
 
 	strbuf_setlen(base, old_baselen);
+	return p;
 }
 
 static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
@@ -145,59 +300,260 @@ static void skip_uninteresting(struct tree_desc *t, struct strbuf *base,
 	}
 }
 
-static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
-			     struct strbuf *base, struct diff_options *opt)
+
+/*
+ * generate paths for combined diff D(sha1,parents_sha1[])
+ *
+ * Resulting paths are appended to combine_diff_path linked list, and also, are
+ * emitted on the go via opt->pathchange() callback, so it is possible to
+ * process the result as batch or incrementally.
+ *
+ * The paths are generated scanning new tree and all parents trees
+ * simultaneously, similarly to what diff_tree() was doing for 2 trees.
+ * The theory behind such scan is as follows:
+ *
+ *
+ * D(T,P1...Pn) calculation scheme
+ * -------------------------------
+ *
+ * D(T,P1...Pn) = D(T,P1) ^ ... ^ D(T,Pn)	(regarding resulting paths set)
+ *
+ *	D(T,Pj)		- diff between T..Pj
+ *	D(T,P1...Pn)	- combined diff from T to parents P1,...,Pn
+ *
+ *
+ * We start from all trees, which are sorted, and compare their entries in
+ * lock-step:
+ *
+ *	 T     P1       Pn
+ *	 -     -        -
+ *	|t|   |p1|     |pn|
+ *	|-|   |--| ... |--|      imin = argmin(p1...pn)
+ *	| |   |  |     |  |
+ *	|-|   |--|     |--|
+ *	|.|   |. |     |. |
+ *	 .     .        .
+ *	 .     .        .
+ *
+ * at any time there could be 3 cases:
+ *
+ *	1)  t < p[imin];
+ *	2)  t > p[imin];
+ *	3)  t = p[imin].
+ *
+ * Schematic deduction of what every case means, and what to do, follows:
+ *
+ * 1)  t < p[imin]  ->  ∀j t ∉ Pj  ->  "+t" ∈ D(T,Pj)  ->  D += "+t";  t↓
+ *
+ * 2)  t > p[imin]
+ *
+ *     2.1) ∃j: pj > p[imin]  ->  "-p[imin]" ∉ D(T,Pj)  ->  D += ø;  ∀ pi=p[imin]  pi↓
+ *     2.2) ∀i  pi = p[imin]  ->  pi ∉ T  ->  "-pi" ∈ D(T,Pi)  ->  D += "-p[imin]";  ∀i pi↓
+ *
+ * 3)  t = p[imin]
+ *
+ *     3.1) ∃j: pj > p[imin]  ->  "+t" ∈ D(T,Pj)  ->  only pi=p[imin] remains to investigate
+ *     3.2) pi = p[imin]  ->  investigate δ(t,pi)
+ *      |
+ *      |
+ *      v
+ *
+ *     3.1+3.2) looking at δ(t,pi) ∀i: pi=p[imin] - if all != ø  ->
+ *
+ *                       ⎧δ(t,pi)  - if pi=p[imin]
+ *              ->  D += ⎨
+ *                       ⎩"+t"     - if pi>p[imin]
+ *
+ *
+ *     in any case t↓  ∀ pi=p[imin]  pi↓
+ *
+ *
+ * ~~~~~~~~
+ *
+ * NOTE
+ *
+ *	Usual diff D(A,B) is by definition the same as combined diff D(A,[B]),
+ *	so this diff paths generator can, and is used, for plain diffs
+ *	generation too.
+ *
+ *	Please keep attention to the common D(A,[B]) case when working on the
+ *	code, in order not to slow it down.
+ *
+ * NOTE
+ *	nparent must be > 0.
+ */
+
+
+/* ∀ pi=p[imin]  pi↓ */
+static inline void update_tp_entries(struct tree_desc *tp, int nparent)
 {
-	struct tree_desc t1, t2;
-	void *t1tree, *t2tree;
+	int i;
+	for (i = 0; i < nparent; ++i)
+		if (!(tp[i].entry.mode & S_IFXMIN_NEQ))
+			update_tree_entry(&tp[i]);
+}
 
-	t1tree = fill_tree_descriptor(&t1, old);
-	t2tree = fill_tree_descriptor(&t2, new);
+static struct combine_diff_path *ll_diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parents_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt)
+{
+	struct tree_desc t, *tp;
+	void *ttree, **tptree;
+	int i;
+
+	tp     = xalloca(nparent * sizeof(tp[0]));
+	tptree = xalloca(nparent * sizeof(tptree[0]));
+
+	/*
+	 * load parents first, as they are probably already cached.
+	 *
+	 * ( log_tree_diff() parses commit->parent before calling here via
+	 *   diff_tree_sha1(parent, commit) )
+	 */
+	for (i = 0; i < nparent; ++i)
+		tptree[i] = fill_tree_descriptor(&tp[i], parents_sha1[i]);
+	ttree = fill_tree_descriptor(&t, sha1);
 
 	/* Enable recursion indefinitely */
 	opt->pathspec.recursive = DIFF_OPT_TST(opt, RECURSIVE);
 
 	for (;;) {
-		int cmp;
+		int imin, cmp;
 
 		if (diff_can_quit_early(opt))
 			break;
+
 		if (opt->pathspec.nr) {
-			skip_uninteresting(&t1, base, opt);
-			skip_uninteresting(&t2, base, opt);
+			skip_uninteresting(&t, base, opt);
+			for (i = 0; i < nparent; i++)
+				skip_uninteresting(&tp[i], base, opt);
+		}
+
+		/* comparing is finished when all trees are done */
+		if (!t.size) {
+			int done = 1;
+			for (i = 0; i < nparent; ++i)
+				if (tp[i].size) {
+					done = 0;
+					break;
+				}
+			if (done)
+				break;
+		}
+
+		/*
+		 * lookup imin = argmin(p1...pn),
+		 * mark entries whether they =p[imin] along the way
+		 */
+		imin = 0;
+		tp[0].entry.mode &= ~S_IFXMIN_NEQ;
+
+		for (i = 1; i < nparent; ++i) {
+			cmp = tree_entry_pathcmp(&tp[i], &tp[imin]);
+			if (cmp < 0) {
+				imin = i;
+				tp[i].entry.mode &= ~S_IFXMIN_NEQ;
+			}
+			else if (cmp == 0) {
+				tp[i].entry.mode &= ~S_IFXMIN_NEQ;
+			}
+			else {
+				tp[i].entry.mode |= S_IFXMIN_NEQ;
+			}
 		}
-		if (!t1.size && !t2.size)
-			break;
 
-		cmp = tree_entry_pathcmp(&t1, &t2);
+		/* fixup markings for entries before imin */
+		for (i = 0; i < imin; ++i)
+			tp[i].entry.mode |= S_IFXMIN_NEQ;	/* pi > p[imin] */
 
-		/* t1 = t2 */
-		if (cmp == 0) {
-			if (DIFF_OPT_TST(opt, FIND_COPIES_HARDER) ||
-			    hashcmp(t1.entry.sha1, t2.entry.sha1) ||
-			    (t1.entry.mode != t2.entry.mode))
-				show_path(base, opt, &t1, &t2);
 
-			update_tree_entry(&t1);
-			update_tree_entry(&t2);
+
+		/* compare t vs p[imin] */
+		cmp = tree_entry_pathcmp(&t, &tp[imin]);
+
+		/* t = p[imin] */
+		if (cmp == 0) {
+			/* are either pi > p[imin] or diff(t,pi) != ø ? */
+			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
+				for (i = 0; i < nparent; ++i) {
+					/* p[i] > p[imin] */
+					if (tp[i].entry.mode & S_IFXMIN_NEQ)
+						continue;
+
+					/* diff(t,pi) != ø */
+					if (hashcmp(t.entry.sha1, tp[i].entry.sha1) ||
+					    (t.entry.mode != tp[i].entry.mode))
+						continue;
+
+					goto skip_emit_t_tp;
+				}
+			}
+
+			/* D += {δ(t,pi) if pi=p[imin];  "+a" if pi > p[imin]} */
+			p = emit_path(p, base, opt, nparent,
+					&t, tp, imin);
+
+		skip_emit_t_tp:
+			/* t↓,  ∀ pi=p[imin]  pi↓ */
+			update_tree_entry(&t);
+			update_tp_entries(tp, nparent);
 		}
 
-		/* t1 < t2 */
+		/* t < p[imin] */
 		else if (cmp < 0) {
-			show_path(base, opt, &t1, /*t2=*/NULL);
-			update_tree_entry(&t1);
+			/* D += "+t" */
+			p = emit_path(p, base, opt, nparent,
+					&t, /*tp=*/NULL, -1);
+
+			/* t↓ */
+			update_tree_entry(&t);
 		}
 
-		/* t1 > t2 */
+		/* t > p[imin] */
 		else {
-			show_path(base, opt, /*t1=*/NULL, &t2);
-			update_tree_entry(&t2);
+			/* ∀i pi=p[imin] -> D += "-p[imin]" */
+			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
+				for (i = 0; i < nparent; ++i)
+					if (tp[i].entry.mode & S_IFXMIN_NEQ)
+						goto skip_emit_tp;
+			}
+
+			p = emit_path(p, base, opt, nparent,
+					/*t=*/NULL, tp, imin);
+
+		skip_emit_tp:
+			/* ∀ pi=p[imin]  pi↓ */
+			update_tp_entries(tp, nparent);
 		}
 	}
 
-	free(t2tree);
-	free(t1tree);
-	return 0;
+	free(ttree);
+	for (i = nparent-1; i >= 0; i--)
+		free(tptree[i]);
+	xalloca_free(tptree);
+	xalloca_free(tp);
+
+	return p;
+}
+
+struct combine_diff_path *diff_tree_paths(
+	struct combine_diff_path *p, const unsigned char *sha1,
+	const unsigned char **parents_sha1, int nparent,
+	struct strbuf *base, struct diff_options *opt)
+{
+	p = ll_diff_tree_paths(p, sha1, parents_sha1, nparent, base, opt);
+
+	/*
+	 * free pre-allocated last element, if any
+	 * (see path_appendnew() for details about why)
+	 */
+	if (p->next) {
+		free(p->next);
+		p->next = NULL;
+	}
+
+	return p;
 }
 
 /*
@@ -308,6 +664,26 @@ static void try_to_follow_renames(const unsigned char *old, const unsigned char
 	q->nr = 1;
 }
 
+static int ll_diff_tree_sha1(const unsigned char *old, const unsigned char *new,
+			     struct strbuf *base, struct diff_options *opt)
+{
+	struct combine_diff_path phead, *p;
+	pathchange_fn_t pathchange_old = opt->pathchange;
+
+	phead.next = NULL;
+	opt->pathchange = emit_diff_first_parent_only;
+	diff_tree_paths(&phead, new, &old, 1, base, opt);
+
+	for (p = phead.next; p;) {
+		struct combine_diff_path *pprev = p;
+		p = p->next;
+		free(pprev);
+	}
+
+	opt->pathchange = pathchange_old;
+	return 0;
+}
+
 int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base_str, struct diff_options *opt)
 {
 	struct strbuf base;
-- 
1.9.rc0.143.g6fd479e

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Junio C Hamano]

Kirill Smelkov <kirr@navytux.spb.ru> writes:

> The following
> ...
> maybe looks a bit simpler, but calls tree_entry_pathcmp twice more times.
>
> Besides for important nparent=1 case we were not calling
> tree_entry_pathcmp at all and here we'll call it once, which would slow
> execution down a bit, as base_name_compare shows measurable enough in profile.
> To avoid that we'll need to add 'if (i==imin) continue' and this won't
> be so simple then. And for general nparent case, as I've said, we'll be
> calling tree_entry_pathcmp twice more times...
>
> Because of all that I'd suggest to go with my original version.

OK.

> ... After some break on the topic,
> with a fresh eye I see a lot of confusion goes from the notation I've
> chosen initially (because of how I was reasoning about it on paper, when
> it was in flux) - i.e. xi for x[imin] and also using i as looping
> variable. And also because xi was already used for x[imin] I've used
> another letter 'k' denoting all other x'es, which leads to confusion...
>
>
> I propose we do the following renaming to clarify things:
>
>     A/a     ->      T/t     (to match resulting tree t name in the code)
>     X/x     ->      P/p     (to match parents trees tp in the code)
>     i       ->      imin    (so that i would be free for other tasks)
>
> then the above (with a prologue) would look like
>
> ---- 8< ----
>  *       T     P1       Pn
>  *       -     -        -
>  *      |t|   |p1|     |pn|
>  *      |-|   |--| ... |--|      imin = argmin(p1...pn)
>  *      | |   |  |     |  |
>  *      |-|   |--|     |--|
>  *      |.|   |. |     |. |
>  *       .     .        .
>  *       .     .        .
>  *
>  * at any time there could be 3 cases:
>  *
>  *      1)  t < p[imin];
>  *      2)  t > p[imin];
>  *      3)  t = p[imin].
>  *
>  * Schematic deduction of what every case means, and what to do, follows:
>  *
>  * 1)  t < p[imin]  ->  ∀j t ∉ Pj  ->  "+t" ∈ D(T,Pj)  ->  D += "+t";  t↓
>  *
>  * 2)  t > p[imin]
>  *
>  *     2.1) ∃j: pj > p[imin]  ->  "-p[imin]" ∉ D(T,Pj)  ->  D += ø;  ∀ pi=p[imin]  pi↓
>  *     2.2) ∀i  pi = p[imin]  ->  pi ∉ T  ->  "-pi" ∈ D(T,Pi)  ->  D += "-p[imin]";  ∀i pi↓
>  *
>  * 3)  t = p[imin]
>  *
>  *     3.1) ∃j: pj > p[imin]  ->  "+t" ∈ D(T,Pj)  ->  only pi=p[imin] remains to investigate
>  *     3.2) pi = p[imin]  ->  investigate δ(t,pi)
>  *      |
>  *      |
>  *      v
>  *
>  *     3.1+3.2) looking at δ(t,pi) ∀i: pi=p[imin] - if all != ø  ->
>  *
>  *                       ⎧δ(t,pi)  - if pi=p[imin]
>  *              ->  D += ⎨
>  *                       ⎩"+t"     - if pi>p[imin]
>  *
>  *
>  *     in any case t↓  ∀ pi=p[imin]  pi↓
> ...
> now xk is gone and i matches p[i] (= pi) etc so variable names correlate
> to algorithm description better.
>
> Does that maybe clarify things?

That sounds more consistent (modulo perhaps s/argmin/min/ at the
beginning?).

> P.S. Sorry for maybe some crept-in mistakes - I've tried to verify it
> thoroughly, but am too sleepy to be completely sure. On the other hand I
> think and hope the patch should be ok.

Thanks and do not be sorry for "mistakes"---we have the review
process exactly for catching them.

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Junio C Hamano]

Kirill Smelkov <kirr@navytux.spb.ru> writes:

>> > +			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
>> > +				for (i = 0; i < nparent; ++i)
>> > +					if (tp[i].entry.mode & S_IFXMIN_NEQ)
>> > +						goto skip_emit_tp;
>> > +			}
>> > +
>> > +			p = emit_path(p, base, opt, nparent,
>> > +					/*t=*/NULL, tp, imin);
>> > +
>> > +		skip_emit_tp:
>> > +			/* ∀ xk=ximin  xk↓ */
>> > +			update_tp_entries(tp, nparent);
>> 
>> There are parents whose path sort earlier than what is in 't'
>> (i.e. they were lost in the result---we would want to show
>> removal).  What makes us jump to the skip label?
>> 
>>     We are looking at path in 't', and some parents have paths that
>>     sort earlier than that path.  We will not go to skip label if
>>     any one of the parent's entry sorts after some other parent (or
>>     the parent in question has ran out its entries), which means we
>>     show the entry from the parents only when all the parents have
>>     that same path, which is missing from 't'.
>> 
>> I am not sure if I am reading this correctly, though.
>> 
>> For the two-way diff, the above degenerates to "show all parent
>> entries that come before the first entry in 't'", which is correct.
>> For the combined diff, the current intersect_paths() makes sure that
>> each path appears in all the pair-wise diff between t and tp[],
>> which again means that the above logic match the current behaviour.
>
> Yes, correct (modulo we *will* go to skip label if any one of the
> parent's entry sorts after some other parent). By definition of combined
> diff we show a path only if it shows in every diff D(T,Pi), and if 
>
>     2.1) ∃j: pj > p[imin]  ->  "-p[imin]" ∉ D(T,Pj)  ->  D += ø;  ∀ pi=p[imin]  pi↓
>
> some pj sorts after p[imin] that would mean that Pj does not have
> p[imin] and since t > p[imin] (which means T does not have p[imin]
> either) diff D(T,Pj) does not have p[imin]. And because of that we know
> the whole combined-diff will not have p[imin] as, by definition,
> combined diff is sets intersection and one of the sets does not have
> that path.
>
>   ( In usual words p[imin] is not changed between Pj..T - it was
>     e.g. removed in Pj~, so merging parents to T does not bring any new
>     information wrt path p[imin] and that is why we do not want to show
>     p[imin] in combined-diff output - no new change about that path )
>
> So nothing to append to the output, and update minimum tree entries,
> preparing for the next step.

That's all in line with the current and traditional definition of
combined diff.

This is a tangent that is outside the scope of this current topic,
but I wonder if you found it disturbing that we treat the result 't'
that has a path and the result 't' that does not have a path with
respect to a parent that does not have the path in a somewhat
assymmetric way.

With a merge M between commits A and B, where they all have the same
path with different contents, we obviously show that path in the
combined diff format.  A merge N that records exactly the same tree
as M that merges the same commits A and B plus another commit C that
does not have that path still shows the combined diff, with one
extra column to express "everything in the result N has been added
with respect to C which did not have the path at all".

However, a merge O between the same commits A and B, where A and B
have a path and O loses it, shows the path in the combined format.
A merge P among the same A, B and an extra parent C that does not
have that path ceases to show it (this is the assymmetry).

It is a natural extension of "Do not show the path when the result
matches one of the parent" rule, and in this case the result P takes
contents, "the path does not exist", from one parent "C", so it is
internally consistent, and I originally designed it that way on
purpose, but somehow it feels a bit strange.

Re: [PATCH v2 18/19] tree-diff: rework diff_tree() to generate diffs for multiparent cases as well

[Kirill Smelkov]

On Mon, Apr 07, 2014 at 10:29:46AM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@navytux.spb.ru> writes:
> 
> > The following
> > ...
> > maybe looks a bit simpler, but calls tree_entry_pathcmp twice more times.
> >
> > Besides for important nparent=1 case we were not calling
> > tree_entry_pathcmp at all and here we'll call it once, which would slow
> > execution down a bit, as base_name_compare shows measurable enough in profile.
> > To avoid that we'll need to add 'if (i==imin) continue' and this won't
> > be so simple then. And for general nparent case, as I've said, we'll be
> > calling tree_entry_pathcmp twice more times...
> >
> > Because of all that I'd suggest to go with my original version.
> 
> OK.

Thanks.

> > ... After some break on the topic,
> > with a fresh eye I see a lot of confusion goes from the notation I've
> > chosen initially (because of how I was reasoning about it on paper, when
> > it was in flux) - i.e. xi for x[imin] and also using i as looping
> > variable. And also because xi was already used for x[imin] I've used
> > another letter 'k' denoting all other x'es, which leads to confusion...
> >
> >
> > I propose we do the following renaming to clarify things:
> >
> >     A/a     ->      T/t     (to match resulting tree t name in the code)
> >     X/x     ->      P/p     (to match parents trees tp in the code)
> >     i       ->      imin    (so that i would be free for other tasks)
> >
> > then the above (with a prologue) would look like
> >
> > ---- 8< ----
> >  *       T     P1       Pn
> >  *       -     -        -
> >  *      |t|   |p1|     |pn|
> >  *      |-|   |--| ... |--|      imin = argmin(p1...pn)
> >  *      | |   |  |     |  |
> >  *      |-|   |--|     |--|
> >  *      |.|   |. |     |. |
> >  *       .     .        .
> >  *       .     .        .
> >  *
> >  * at any time there could be 3 cases:
> >  *
> >  *      1)  t < p[imin];
> >  *      2)  t > p[imin];
> >  *      3)  t = p[imin].
> >  *
> >  * Schematic deduction of what every case means, and what to do, follows:
> >  *
> >  * 1)  t < p[imin]  ->  ∀j t ∉ Pj  ->  "+t" ∈ D(T,Pj)  ->  D += "+t";  t↓
> >  *
> >  * 2)  t > p[imin]
> >  *
> >  *     2.1) ∃j: pj > p[imin]  ->  "-p[imin]" ∉ D(T,Pj)  ->  D += ø;  ∀ pi=p[imin]  pi↓
> >  *     2.2) ∀i  pi = p[imin]  ->  pi ∉ T  ->  "-pi" ∈ D(T,Pi)  ->  D += "-p[imin]";  ∀i pi↓
> >  *
> >  * 3)  t = p[imin]
> >  *
> >  *     3.1) ∃j: pj > p[imin]  ->  "+t" ∈ D(T,Pj)  ->  only pi=p[imin] remains to investigate
> >  *     3.2) pi = p[imin]  ->  investigate δ(t,pi)
> >  *      |
> >  *      |
> >  *      v
> >  *
> >  *     3.1+3.2) looking at δ(t,pi) ∀i: pi=p[imin] - if all != ø  ->
> >  *
> >  *                       ⎧δ(t,pi)  - if pi=p[imin]
> >  *              ->  D += ⎨
> >  *                       ⎩"+t"     - if pi>p[imin]
> >  *
> >  *
> >  *     in any case t↓  ∀ pi=p[imin]  pi↓
> > ...
> > now xk is gone and i matches p[i] (= pi) etc so variable names correlate
> > to algorithm description better.
> >
> > Does that maybe clarify things?
> 
> That sounds more consistent (modulo perhaps s/argmin/min/ at the
> beginning?).

Thanks. argmin is there on purpose - min(p1...pn) is the minimal p, and
argmin(p1...pn) is imin such that p[imin] is minimal. As we are finding
the index of the minimal element we should use argmin.


> > P.S. Sorry for maybe some crept-in mistakes - I've tried to verify it
> > thoroughly, but am too sleepy to be completely sure. On the other hand I
> > think and hope the patch should be ok.
> 
> Thanks and do not be sorry for "mistakes"---we have the review
> process exactly for catching them.

Thanks, I appreciate that.


On Mon, Apr 07, 2014 at 11:07:47AM -0700, Junio C Hamano wrote:
> Kirill Smelkov <kirr@navytux.spb.ru> writes:
> 
> >> > +			if (!DIFF_OPT_TST(opt, FIND_COPIES_HARDER)) {
> >> > +				for (i = 0; i < nparent; ++i)
> >> > +					if (tp[i].entry.mode & S_IFXMIN_NEQ)
> >> > +						goto skip_emit_tp;
> >> > +			}
> >> > +
> >> > +			p = emit_path(p, base, opt, nparent,
> >> > +					/*t=*/NULL, tp, imin);
> >> > +
> >> > +		skip_emit_tp:
> >> > +			/* ∀ xk=ximin  xk↓ */
> >> > +			update_tp_entries(tp, nparent);
> >> 
> >> There are parents whose path sort earlier than what is in 't'
> >> (i.e. they were lost in the result---we would want to show
> >> removal).  What makes us jump to the skip label?
> >> 
> >>     We are looking at path in 't', and some parents have paths that
> >>     sort earlier than that path.  We will not go to skip label if
> >>     any one of the parent's entry sorts after some other parent (or
> >>     the parent in question has ran out its entries), which means we
> >>     show the entry from the parents only when all the parents have
> >>     that same path, which is missing from 't'.
> >> 
> >> I am not sure if I am reading this correctly, though.
> >> 
> >> For the two-way diff, the above degenerates to "show all parent
> >> entries that come before the first entry in 't'", which is correct.
> >> For the combined diff, the current intersect_paths() makes sure that
> >> each path appears in all the pair-wise diff between t and tp[],
> >> which again means that the above logic match the current behaviour.
> >
> > Yes, correct (modulo we *will* go to skip label if any one of the
> > parent's entry sorts after some other parent). By definition of combined
> > diff we show a path only if it shows in every diff D(T,Pi), and if 
> >
> >     2.1) ∃j: pj > p[imin]  ->  "-p[imin]" ∉ D(T,Pj)  ->  D += ø;  ∀ pi=p[imin]  pi↓
> >
> > some pj sorts after p[imin] that would mean that Pj does not have
> > p[imin] and since t > p[imin] (which means T does not have p[imin]
> > either) diff D(T,Pj) does not have p[imin]. And because of that we know
> > the whole combined-diff will not have p[imin] as, by definition,
> > combined diff is sets intersection and one of the sets does not have
> > that path.
> >
> >   ( In usual words p[imin] is not changed between Pj..T - it was
> >     e.g. removed in Pj~, so merging parents to T does not bring any new
> >     information wrt path p[imin] and that is why we do not want to show
> >     p[imin] in combined-diff output - no new change about that path )
> >
> > So nothing to append to the output, and update minimum tree entries,
> > preparing for the next step.
> 
> That's all in line with the current and traditional definition of
> combined diff.

Ok.


> This is a tangent that is outside the scope of this current topic,
> but I wonder if you found it disturbing that we treat the result 't'
> that has a path and the result 't' that does not have a path with
> respect to a parent that does not have the path in a somewhat
> assymmetric way.
> 
> With a merge M between commits A and B, where they all have the same
> path with different contents, we obviously show that path in the
> combined diff format.  A merge N that records exactly the same tree
> as M that merges the same commits A and B plus another commit C that
> does not have that path still shows the combined diff, with one
> extra column to express "everything in the result N has been added
> with respect to C which did not have the path at all".
> 
> However, a merge O between the same commits A and B, where A and B
> have a path and O loses it, shows the path in the combined format.
> A merge P among the same A, B and an extra parent C that does not
> have that path ceases to show it (this is the assymmetry).

Symmetry properties are very important and if something does not fit into
symmetry - then something somewhere is really wrong for sure, but I
think there is no asymmetry here. In your example

      M∆∙    N∆∙
     / \ . '''
    / .'\  '  '
    A∆   B∙    Cø
    \ ''/. '  '
     \ /   '''
      Oø     Pø

let's say a path can be:

    ø   - empty
    ∆   - triangle
    ∙   - bullet
    ∆∙  - triangle+bullet

then some symmetry operation is

    ∆∙  <-> ø
    ∙   <-> ∙
    ∆   <-> ∆

so you "mirror" ∆∙ to ø (M,N->O,P), and so ø is mirrored back to ∆∙
(O,P->M,N). But then, when we mirror the whole graph, Cø should be
mirrored to C'∆∙ and then that would be correct:

    A∆   B∙    C'∆∙
    \ ''/. '  '
     \ /   '''
      Oø     Pø

P to A,B,C' would show the path as N to A,B,C(without')


In other words a merge P among the same A, B and extra parent C' with
"contains everything" content is symmetrical to merge N to A,B and C
with empty content.


> It is a natural extension of "Do not show the path when the result
> matches one of the parent" rule, and in this case the result P takes
> contents, "the path does not exist", from one parent "C", so it is
> internally consistent, and I originally designed it that way on
> purpose, but somehow it feels a bit strange.

I hope it does not fill strange once the true symmetry is discovered,
and also P does not add a change compared to C, so the combined diff
should be empty as no new information is present in a merge itself.

A bit unusual on the first glance, but not strange, once you are used to
it and consistent.

Thanks,
Kirill

Re: [PATCH 17/19] Portable alloca for Git

[Kirill Smelkov]

On Thu, Mar 27, 2014 at 06:22:50PM +0400, Kirill Smelkov wrote:
> On Mon, Mar 24, 2014 at 02:47:24PM -0700, Junio C Hamano wrote:
> > Kirill Smelkov <kirr@mns.spb.ru> writes:
> > 
> > > On Fri, Feb 28, 2014 at 06:19:58PM +0100, Erik Faye-Lund wrote:
> > >> On Fri, Feb 28, 2014 at 6:00 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
> > >> ...
> > >> > In fact that would be maybe preferred, for maintainers to enable alloca
> > >> > with knowledge and testing, as one person can't have them all at hand.
> > >> 
> > >> Yeah, you're probably right.
> > >
> > > Erik, the patch has been merged into pu today. Would you please
> > > follow-up with tested MINGW change?
> > 
> > Sooo.... I lost track but this discussion seems to have petered out
> > around here.  I think the copy we have had for a while on 'pu' is
> > basically sound, and can easily built on by platform folks by adding
> > or removing the -DHAVE_ALLOCA_H from the Makefile.
> 
> Yes, that is all correct - that version works and we can improve it in
> the future with platform-specific follow-up patches, if needed.

Junio, thanks for merging this and other diff-tree patches to next.  It
so happened that I'm wrestling with MSysGit today, so please also find
alloca-for-mingw patch attached below.

Thanks,
Kirill

---- 8< ----
Subject: [PATCH] mingw: activate alloca

Both MSVC and MINGW have alloca(3) definitions in malloc.h, so by moving
win32-compat alloca.h from compat/vcbuild/include/ to compat/win32/ ,
which is included by both MSVC and MINGW CFLAGS, we can make alloca()
work on both those Windows environments.

In MINGW, malloc.h has explicit check for GNUC and if it is so, defines
alloca to __builtin_alloca, so it looks like we don't need to add any
code to here-shipped alloca.h to get optimum performance.

Compile-tested on Windows in MSysGit.

Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
---
 compat/{vcbuild/include => win32}/alloca.h | 0
 config.mak.uname                           | 1 +
 2 files changed, 1 insertion(+)
 rename compat/{vcbuild/include => win32}/alloca.h (100%)

diff --git a/compat/vcbuild/include/alloca.h b/compat/win32/alloca.h
similarity index 100%
rename from compat/vcbuild/include/alloca.h
rename to compat/win32/alloca.h
diff --git a/config.mak.uname b/config.mak.uname
index 17ef893..67bc054 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -480,6 +480,7 @@ ifeq ($(uname_S),NONSTOP_KERNEL)
 endif
 ifneq (,$(findstring MINGW,$(uname_S)))
 	pathsep = ;
+	HAVE_ALLOCA_H = YesPlease
 	NO_PREAD = YesPlease
 	NEEDS_CRYPTO_WITH_SSL = YesPlease
 	NO_LIBGEN_H = YesPlease
-- 
1.9.0.msysgit.0.31.g74d1b9a.dirty

Re: [PATCH 17/19] Portable alloca for Git

[Erik Faye-Lund]

On Wed, Apr 9, 2014 at 2:48 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
> On Thu, Mar 27, 2014 at 06:22:50PM +0400, Kirill Smelkov wrote:
>> On Mon, Mar 24, 2014 at 02:47:24PM -0700, Junio C Hamano wrote:
>> > Kirill Smelkov <kirr@mns.spb.ru> writes:
>> >
>> > > On Fri, Feb 28, 2014 at 06:19:58PM +0100, Erik Faye-Lund wrote:
>> > >> On Fri, Feb 28, 2014 at 6:00 PM, Kirill Smelkov <kirr@mns.spb.ru> wrote:
>> > >> ...
>> > >> > In fact that would be maybe preferred, for maintainers to enable alloca
>> > >> > with knowledge and testing, as one person can't have them all at hand.
>> > >>
>> > >> Yeah, you're probably right.
>> > >
>> > > Erik, the patch has been merged into pu today. Would you please
>> > > follow-up with tested MINGW change?
>> >
>> > Sooo.... I lost track but this discussion seems to have petered out
>> > around here.  I think the copy we have had for a while on 'pu' is
>> > basically sound, and can easily built on by platform folks by adding
>> > or removing the -DHAVE_ALLOCA_H from the Makefile.
>>
>> Yes, that is all correct - that version works and we can improve it in
>> the future with platform-specific follow-up patches, if needed.
>
> Junio, thanks for merging this and other diff-tree patches to next.  It
> so happened that I'm wrestling with MSysGit today, so please also find
> alloca-for-mingw patch attached below.
>
> Thanks,
> Kirill
>
> ---- 8< ----
> Subject: [PATCH] mingw: activate alloca
>
> Both MSVC and MINGW have alloca(3) definitions in malloc.h, so by moving
> win32-compat alloca.h from compat/vcbuild/include/ to compat/win32/ ,
> which is included by both MSVC and MINGW CFLAGS, we can make alloca()
> work on both those Windows environments.
>
> In MINGW, malloc.h has explicit check for GNUC and if it is so, defines
> alloca to __builtin_alloca, so it looks like we don't need to add any
> code to here-shipped alloca.h to get optimum performance.
>
> Compile-tested on Windows in MSysGit.
>
> Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>

Looks good to me!

Re: [PATCH 17/19] Portable alloca for Git

[Junio C Hamano]

Erik Faye-Lund <kusmabite@gmail.com> writes:

>> Subject: [PATCH] mingw: activate alloca
>>
>> Both MSVC and MINGW have alloca(3) definitions in malloc.h, so by moving
>> win32-compat alloca.h from compat/vcbuild/include/ to compat/win32/ ,
>> which is included by both MSVC and MINGW CFLAGS, we can make alloca()
>> work on both those Windows environments.
>>
>> In MINGW, malloc.h has explicit check for GNUC and if it is so, defines
>> alloca to __builtin_alloca, so it looks like we don't need to add any
>> code to here-shipped alloca.h to get optimum performance.
>>
>> Compile-tested on Windows in MSysGit.
>>
>> Signed-off-by: Kirill Smelkov <kirr@mns.spb.ru>
>
> Looks good to me!

Thanks; queued and pushed out on 'next'.