[PATCH v2 2/4] apply: Allow blank context lines to match beyond EOF

[PATCH v2 2/4] apply: Allow blank context lines to match beyond EOF

[Björn Gustavsson]

"git apply --whitespace=fix" will not always succeed when used
on a series of patches in the following circumstances:

* One patch adds a blank line at the end of a file. (Since
  --whitespace=fix is used, the blank line will *not* be added.)

* The next patch adds non-blanks lines after the blank line
  introduced in the first patch. That patch will not apply
  because the blank line that is expected to be found at end
  of the file is no longer there.

A patch series that starts by deleting lines at the end
will fail in a similar way.

Fix this problem by allowing a blank context line at the beginning
of a hunk to match if parts of it falls beyond end of the file.
We still require that at least one non-blank context line match
before the end of the file.

Signed-off-by: Björn Gustavsson <bgustavsson@gmail.com>
---
 builtin-apply.c |  135 +++++++++++++++++++++++++++++++++++++++++++++---------
 1 files changed, 112 insertions(+), 23 deletions(-)

diff --git a/builtin-apply.c b/builtin-apply.c
index e1f849d..b8b89f7 100644
--- a/builtin-apply.c
+++ b/builtin-apply.c
@@ -1854,33 +1854,81 @@ static int match_fragment(struct image *img,
 {
 	int i;
 	char *fixed_buf, *buf, *orig, *target;
+	int limit;
+	int preimage_limit;
 
-	if (preimage->nr + try_lno > img->nr)
+	if (preimage->nr + try_lno <= img->nr) {
+		/*
+		 * The hunk falls within the boundaries of img.
+		 */
+		limit = img->nr;
+		preimage_limit = preimage->nr;
+	} else if (ws_error_action == correct_ws_error &&
+		   ws_rule & WS_BLANK_AT_EOF && match_end) {
+		/*
+		 * This hunk that matches at the end, extends beyond
+		 * the end of img and we are removing blanks line at
+		 * the end of the file. Set up the limits so that
+		 * tests below will pass and the quick hash test
+		 * will only test the lines up to the last line
+		 * in img.
+		 */
+		limit = try_lno + preimage->nr;
+		preimage_limit = img->nr - try_lno;
+	} else {
+		/*
+		 * The hunk extends beyond the end of the img and
+		 * we are not removing blanks at the end, so we
+		 * should reject the hunk at this position.
+		 */
 		return 0;
+	}
 
 	if (match_beginning && try_lno)
 		return 0;
 
-	if (match_end && preimage->nr + try_lno != img->nr)
+	if (match_end && preimage->nr + try_lno != limit)
 		return 0;
 
 	/* Quick hash check */
-	for (i = 0; i < preimage->nr; i++)
+	for (i = 0; i < preimage_limit; i++)
 		if (preimage->line[i].hash != img->line[try_lno + i].hash)
 			return 0;
 
-	/*
-	 * Do we have an exact match?  If we were told to match
-	 * at the end, size must be exactly at try+fragsize,
-	 * otherwise try+fragsize must be still within the preimage,
-	 * and either case, the old piece should match the preimage
-	 * exactly.
-	 */
-	if ((match_end
-	     ? (try + preimage->len == img->len)
-	     : (try + preimage->len <= img->len)) &&
-	    !memcmp(img->buf + try, preimage->buf, preimage->len))
-		return 1;
+	if (preimage_limit == preimage->nr) {
+		/*
+		 * Do we have an exact match?  If we were told to match
+		 * at the end, size must be exactly at try+fragsize,
+		 * otherwise try+fragsize must be still within the preimage,
+		 * and either case, the old piece should match the preimage
+		 * exactly.
+		 */
+		if ((match_end
+		     ? (try + preimage->len == img->len)
+		     : (try + preimage->len <= img->len)) &&
+		    !memcmp(img->buf + try, preimage->buf, preimage->len))
+			return 1;
+	} else {
+		/*
+		 * The preimage extends beyond the end of img, so
+		 * there cannot be an exact match.
+		 *
+		 * There must be one non-blank context line that match
+		 * a line before the of img.
+		 */
+		char *buf_end;
+
+		buf = preimage->buf; 
+		buf_end = buf;
+		for (i = 0; i < preimage_limit; i++)
+			buf_end += preimage->line[i].len;
+
+		for ( ; buf < buf_end; buf++)
+			if (!isspace(*buf))
+				break;
+		if (buf == buf_end)
+			return 0;
+	}
 
 	/*
 	 * No exact match. If we are ignoring whitespace, run a line-by-line
@@ -1932,12 +1980,16 @@ static int match_fragment(struct image *img,
 	 * it might with whitespace fuzz. We haven't been asked to
 	 * ignore whitespace, we were asked to correct whitespace
 	 * errors, so let's try matching after whitespace correction.
+	 *
+	 * The preimage may extend beyond the end of the file,
+	 * but in this loop we will only handle the part of the
+	 * preimage that falls within the file.
 	 */
 	fixed_buf = xmalloc(preimage->len + 1);
 	buf = fixed_buf;
 	orig = preimage->buf;
 	target = img->buf + try;
-	for (i = 0; i < preimage->nr; i++) {
+	for (i = 0; i < preimage_limit; i++) {
 		size_t fixlen; /* length after fixing the preimage */
 		size_t oldlen = preimage->line[i].len;
 		size_t tgtlen = img->line[try_lno + i].len;
@@ -1977,6 +2029,29 @@ static int match_fragment(struct image *img,
 		target += tgtlen;
 	}
 
+
+	/*
+	 * Now handle the lines in the preimage that falls beyond the
+	 * end of the file (if any). They will only match if they are
+	 * empty or only contain whitespace (if WS_BLANK_AT_EOL is
+	 * false).
+	 */
+	for ( ; i < preimage->nr; i++) {
+		size_t fixlen; /* length after fixing the preimage */
+		size_t oldlen = preimage->line[i].len;
+		int j;
+
+		/* Try fixing the line in the preimage */
+		fixlen = ws_fix_copy(buf, orig, oldlen, ws_rule, NULL);
+
+		for (j = 0; j < fixlen; j++)
+			if (!isspace(buf[j]))
+				goto unmatch_exit;
+
+		orig += oldlen;
+		buf += fixlen;
+	}
+
 	/*
 	 * Yes, the preimage is based on an older version that still
 	 * has whitespace breakages unfixed, and fixing them makes the
@@ -2092,12 +2167,26 @@ static void update_image(struct image *img,
 	int i, nr;
 	size_t remove_count, insert_count, applied_at = 0;
 	char *result;
+	int preimage_limit;
+
+	/*
+	 * If we are removing blank lines at the end of img,
+	 * the preimage may extend beyond the end.
+	 * If that is the case, we must be careful only to
+	 * remove the part of the preimage that falls within
+	 * the boundaries of img. Initialize preimage_limit
+	 * to the number of lines in the preimage that falls
+	 * within the boundaries.
+	 */
+	preimage_limit = preimage->nr;
+	if (preimage_limit > img->nr - applied_pos)
+		preimage_limit = img->nr - applied_pos;
 
 	for (i = 0; i < applied_pos; i++)
 		applied_at += img->line[i].len;
 
 	remove_count = 0;
-	for (i = 0; i < preimage->nr; i++)
+	for (i = 0; i < preimage_limit; i++)
 		remove_count += img->line[applied_pos + i].len;
 	insert_count = postimage->len;
 
@@ -2114,8 +2203,8 @@ static void update_image(struct image *img,
 	result[img->len] = '\0';
 
 	/* Adjust the line table */
-	nr = img->nr + postimage->nr - preimage->nr;
-	if (preimage->nr < postimage->nr) {
+	nr = img->nr + postimage->nr - preimage_limit;
+	if (preimage_limit < postimage->nr) {
 		/*
 		 * NOTE: this knows that we never call remove_first_line()
 		 * on anything other than pre/post image.
@@ -2123,10 +2212,10 @@ static void update_image(struct image *img,
 		img->line = xrealloc(img->line, nr * sizeof(*img->line));
 		img->line_allocated = img->line;
 	}
-	if (preimage->nr != postimage->nr)
+	if (preimage_limit != postimage->nr)
 		memmove(img->line + applied_pos + postimage->nr,
-			img->line + applied_pos + preimage->nr,
-			(img->nr - (applied_pos + preimage->nr)) *
+			img->line + applied_pos + preimage_limit,
+			(img->nr - (applied_pos + preimage_limit)) *
 			sizeof(*img->line));
 	memcpy(img->line + applied_pos,
 	       postimage->line,
@@ -2322,7 +2411,7 @@ static int apply_one_fragment(struct image *img, struct fragment *frag,
 
 	if (applied_pos >= 0) {
 		if (new_blank_lines_at_end &&
-		    preimage.nr + applied_pos == img->nr &&
+		    preimage.nr + applied_pos >= img->nr &&
 		    (ws_rule & WS_BLANK_AT_EOF) &&
 		    ws_error_action != nowarn_ws_error) {
 			record_ws_error(WS_BLANK_AT_EOF, "+", 1, frag->linenr);
-- 
1.7.0

Re: [PATCH v2 2/4] apply: Allow blank context lines to match beyond EOF

[Junio C Hamano]

Björn Gustavsson <bgustavsson@gmail.com> writes:

> "git apply --whitespace=fix" will not always succeed when used
> on a series of patches in the following circumstances:

Very nicely done.

I wanted to explain to myself what "limit" and "preimage_limit" variables
mean.  preimage-limit has a very clear definition: this many lines from
the beginning of preimage must match img, and the remainder of the
preimage must be all blank (the remainder can exist only when we are
trying to match at the end).

On the other hand "limit" does not have such a good definition, other than
as a work around to bypass line-number check when we are trying to match
at the end.  It might be cleaner to read if we move the problematic "line
numbers must match" logic and eliminate this variable, like the attached
patch does on top of this one.

I couldn't figure out how this would interact with the ignore_ws_change
codepath, though.  That one shows a clear sign of being bolted on as an
afterthought (once you fall into that "if()" statement you will not come
back).

 builtin-apply.c |   15 +++++----------
 1 files changed, 5 insertions(+), 10 deletions(-)

diff --git a/builtin-apply.c b/builtin-apply.c
index 4374d33..ae4452c 100644
--- a/builtin-apply.c
+++ b/builtin-apply.c
@@ -1854,26 +1854,24 @@ static int match_fragment(struct image *img,
 {
 	int i;
 	char *fixed_buf, *buf, *orig, *target;
-	int limit;
 	int preimage_limit;
 
 	if (preimage->nr + try_lno <= img->nr) {
 		/*
 		 * The hunk falls within the boundaries of img.
 		 */
-		limit = img->nr;
 		preimage_limit = preimage->nr;
+		if (match_end && (preimage->nr + try_lno != img->nr))
+			return 0;
 	} else if (ws_error_action == correct_ws_error &&
 		   (ws_rule & WS_BLANK_AT_EOF) && match_end) {
 		/*
 		 * This hunk that matches at the end extends beyond
 		 * the end of img, and we are removing blank lines
-		 * at the end of the file. Set up the limits so that
-		 * tests below will pass and the quick hash test
-		 * will only test the lines up to the last line
-		 * in img.
+		 * at the end of the file.  This many lines from the
+		 * beginning of the preimage must match with img, and
+		 * the remainder of the preimage must be blank.
 		 */
-		limit = try_lno + preimage->nr;
 		preimage_limit = img->nr - try_lno;
 	} else {
 		/*
@@ -1887,9 +1885,6 @@ static int match_fragment(struct image *img,
 	if (match_beginning && try_lno)
 		return 0;
 
-	if (match_end && preimage->nr + try_lno != limit)
-		return 0;
-
 	/* Quick hash check */
 	for (i = 0; i < preimage_limit; i++)
 		if (preimage->line[i].hash != img->line[try_lno + i].hash)

Re: [PATCH v2 2/4] apply: Allow blank context lines to match beyond EOF

[Björn Gustavsson]

2010/2/24 Junio C Hamano <gitster@pobox.com>:
> Very nicely done.

Thanks! :)

> On the other hand "limit" does not have such a good definition, other than
> as a work around to bypass line-number check when we are trying to match
> at the end.  It might be cleaner to read if we move the problematic "line
> numbers must match" logic and eliminate this variable, like the attached
> patch does on top of this one.

Yes, your version is better. Having a "limit" variable no longer makes
sense (my original patch used "limit" in two places). Feel free to
squeeze it in.

> I couldn't figure out how this would interact with the ignore_ws_change
> codepath, though.  That one shows a clear sign of being bolted on as an
> afterthought (once you fall into that "if()" statement you will not come
> back).

Yes, it does seem bolted on.

I haven't looked much at that if() statement, because I
sort of assumed that because of the return it couldn't do any
harm.

It is too late in my time zone for me to think clearly, but it does
seem that I was wrong and that I'll need to do some changes in
that "if()" statement, and also write some more tests for
the combination of --whitespace=fix and --ignore-space-change.

I'll be back another day.

Thanks for the review.

-- 
Björn Gustavsson, Erlang/OTP, Ericsson AB