[PATCH 1/2] sha1_file: replace PATH_MAX buffer wirh strbuf in prepare_packed_git_one()

[PATCH 1/2] sha1_file: replace PATH_MAX buffer wirh strbuf in prepare_packed_git_one()

[René Scharfe]

Instead of using strbuf to create a message string in case a path is
too long for our fixed-size buffer, replace that buffer with a strbuf
and thus get rid of the limitation.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 sha1_file.c | 41 +++++++++++++++--------------------------
 1 file changed, 15 insertions(+), 26 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 34d527f..0c3cada 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1177,48 +1177,36 @@ static void report_pack_garbage(struct string_list *list)
 
 static void prepare_packed_git_one(char *objdir, int local)
 {
-	/* Ensure that this buffer is large enough so that we can
-	   append "/pack/" without clobbering the stack even if
-	   strlen(objdir) were PATH_MAX.  */
-	char path[PATH_MAX + 1 + 4 + 1 + 1];
-	int len;
+	struct strbuf path = STRBUF_INIT;
+	size_t dirnamelen;
 	DIR *dir;
 	struct dirent *de;
 	struct string_list garbage = STRING_LIST_INIT_DUP;
 
-	sprintf(path, "%s/pack", objdir);
-	len = strlen(path);
-	dir = opendir(path);
+	strbuf_addstr(&path, objdir);
+	strbuf_addstr(&path, "/pack");
+	dir = opendir(path.buf);
 	if (!dir) {
 		if (errno != ENOENT)
 			error("unable to open object pack directory: %s: %s",
-			      path, strerror(errno));
+			      path.buf, strerror(errno));
 		return;
 	}
-	path[len++] = '/';
+	strbuf_addch(&path, '/');
+	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
-		int namelen = strlen(de->d_name);
 		struct packed_git *p;
 
-		if (len + namelen + 1 > sizeof(path)) {
-			if (report_garbage) {
-				struct strbuf sb = STRBUF_INIT;
-				strbuf_addf(&sb, "%.*s/%s", len - 1, path, de->d_name);
-				report_garbage("path too long", sb.buf);
-				strbuf_release(&sb);
-			}
-			continue;
-		}
-
 		if (is_dot_or_dotdot(de->d_name))
 			continue;
 
-		strcpy(path + len, de->d_name);
+		strbuf_setlen(&path, dirnamelen);
+		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path, p->pack_name, len + namelen - 4))
+				if (!memcmp(path.buf, p->pack_name, path.len - 4))
 					break;
 			}
 			if (p == NULL &&
@@ -1226,7 +1214,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 			     * See if it really is a valid .idx file with
 			     * corresponding .pack file that we can map.
 			     */
-			    (p = add_packed_git(path, len + namelen, local)) != NULL)
+			    (p = add_packed_git(path.buf, path.len, local)) != NULL)
 				install_packed_git(p);
 		}
 
@@ -1237,13 +1225,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		    has_extension(de->d_name, ".pack") ||
 		    has_extension(de->d_name, ".bitmap") ||
 		    has_extension(de->d_name, ".keep"))
-			string_list_append(&garbage, path);
+			string_list_append(&garbage, path.buf);
 		else
-			report_garbage("garbage found", path);
+			report_garbage("garbage found", path.buf);
 	}
 	closedir(dir);
 	report_pack_garbage(&garbage);
 	string_list_clear(&garbage, 0);
+	strbuf_release(&path);
 }
 
 static int sort_pack(const void *a_, const void *b_)
-- 
2.0.0

[PATCH 2/4] sha1_file: use strncmp for string comparison

[René Scharfe]

Avoid overrunning the existing pack name (p->pack_name, a C string) in
the case that the new path is longer by using strncmp instead of memcmp
for comparing.  While at it, replace the magic constant 4 with a
strlen call to document its meaning.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 sha1_file.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/sha1_file.c b/sha1_file.c
index 0c3cada..72b8fcb 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1206,7 +1206,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path.buf, p->pack_name, path.len - 4))
+				if (!strncmp(path.buf, p->pack_name,
+					     path.len - strlen(".idx")))
 					break;
 			}
 			if (p == NULL &&
-- 
2.0.0

Re: [PATCH 1/2] sha1_file: replace PATH_MAX buffer wirh strbuf in prepare_packed_git_one()

[Duy Nguyen]

On Sat, Jun 28, 2014 at 9:47 PM, René Scharfe <l.s.r@web.de> wrote:
> -       sprintf(path, "%s/pack", objdir);
> -       len = strlen(path);
> -       dir = opendir(path);
> +       strbuf_addstr(&path, objdir);
> +       strbuf_addstr(&path, "/pack");
> +       dir = opendir(path.buf);
>         if (!dir) {
>                 if (errno != ENOENT)
>                         error("unable to open object pack directory: %s: %s",
> -                             path, strerror(errno));
> +                             path.buf, strerror(errno));
>                 return;

Memory leak. The rest looks good.
-- 
Duy

[PATCH v2 1/2] sha1_file: replace PATH_MAX buffer wirh strbuf in, prepare_packed_git_one()

[René Scharfe]

Instead of using strbuf to create a message string in case a path is
too long for our fixed-size buffer, replace that buffer with a strbuf
and thus get rid of the limitation.

Helped-by: Duy Nguyen <pclouds@gmail.com>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
@Duy: Thanks for catching the missing strbuf_release in the opendir
error case.

 sha1_file.c | 42 ++++++++++++++++--------------------------
 1 file changed, 16 insertions(+), 26 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 34d527f..394fa45 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1177,48 +1177,37 @@ static void report_pack_garbage(struct string_list *list)
 
 static void prepare_packed_git_one(char *objdir, int local)
 {
-	/* Ensure that this buffer is large enough so that we can
-	   append "/pack/" without clobbering the stack even if
-	   strlen(objdir) were PATH_MAX.  */
-	char path[PATH_MAX + 1 + 4 + 1 + 1];
-	int len;
+	struct strbuf path = STRBUF_INIT;
+	size_t dirnamelen;
 	DIR *dir;
 	struct dirent *de;
 	struct string_list garbage = STRING_LIST_INIT_DUP;
 
-	sprintf(path, "%s/pack", objdir);
-	len = strlen(path);
-	dir = opendir(path);
+	strbuf_addstr(&path, objdir);
+	strbuf_addstr(&path, "/pack");
+	dir = opendir(path.buf);
 	if (!dir) {
 		if (errno != ENOENT)
 			error("unable to open object pack directory: %s: %s",
-			      path, strerror(errno));
+			      path.buf, strerror(errno));
+		strbuf_release(&path);
 		return;
 	}
-	path[len++] = '/';
+	strbuf_addch(&path, '/');
+	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
-		int namelen = strlen(de->d_name);
 		struct packed_git *p;
 
-		if (len + namelen + 1 > sizeof(path)) {
-			if (report_garbage) {
-				struct strbuf sb = STRBUF_INIT;
-				strbuf_addf(&sb, "%.*s/%s", len - 1, path, de->d_name);
-				report_garbage("path too long", sb.buf);
-				strbuf_release(&sb);
-			}
-			continue;
-		}
-
 		if (is_dot_or_dotdot(de->d_name))
 			continue;
 
-		strcpy(path + len, de->d_name);
+		strbuf_setlen(&path, dirnamelen);
+		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path, p->pack_name, len + namelen - 4))
+				if (!memcmp(path.buf, p->pack_name, path.len - 4))
 					break;
 			}
 			if (p == NULL &&
@@ -1226,7 +1215,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 			     * See if it really is a valid .idx file with
 			     * corresponding .pack file that we can map.
 			     */
-			    (p = add_packed_git(path, len + namelen, local)) != NULL)
+			    (p = add_packed_git(path.buf, path.len, local)) != NULL)
 				install_packed_git(p);
 		}
 
@@ -1237,13 +1226,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		    has_extension(de->d_name, ".pack") ||
 		    has_extension(de->d_name, ".bitmap") ||
 		    has_extension(de->d_name, ".keep"))
-			string_list_append(&garbage, path);
+			string_list_append(&garbage, path.buf);
 		else
-			report_garbage("garbage found", path);
+			report_garbage("garbage found", path.buf);
 	}
 	closedir(dir);
 	report_pack_garbage(&garbage);
 	string_list_clear(&garbage, 0);
+	strbuf_release(&path);
 }
 
 static int sort_pack(const void *a_, const void *b_)
-- 
2.0.0

[PATCH v2 2/2] sha1_file: use strncmp for string comparison

[René Scharfe]

Avoid overrunning the existing pack name (p->pack_name, a C string) in
the case that the new path is longer by using strncmp instead of memcmp
for comparing.  While at it, replace the magic constant 4 with a
strlen call to document its meaning.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
No changes from intial round.

 sha1_file.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/sha1_file.c b/sha1_file.c
index 394fa45..8adab14 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path.buf, p->pack_name, path.len - 4))
+				if (!strncmp(path.buf, p->pack_name,
+					     path.len - strlen(".idx")))
 					break;
 			}
 			if (p == NULL &&
-- 
2.0.0

Re: [PATCH v2 1/2] sha1_file: replace PATH_MAX buffer wirh strbuf in, prepare_packed_git_one()

[Jeff King]

On Sun, Jun 29, 2014 at 07:43:17AM +0200, René Scharfe wrote:

> Instead of using strbuf to create a message string in case a path is
> too long for our fixed-size buffer, replace that buffer with a strbuf
> and thus get rid of the limitation.

Yay. Safer, and the end result is much more readable.

-Peff

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

[Jeff King]

On Sun, Jun 29, 2014 at 07:56:25AM +0200, René Scharfe wrote:

> Avoid overrunning the existing pack name (p->pack_name, a C string) in
> the case that the new path is longer by using strncmp instead of memcmp
> for comparing.  While at it, replace the magic constant 4 with a
> strlen call to document its meaning.

An unwritten assumption with this code is that we have "foo.idx" and we
want to make sure that we are matching "foo.pack" from the existing pack
list. Both before and after your patch, I think we would match
"foobar.pack". It's probably not a big deal, as we don't expect random
junk in the pack directory, but I wonder if it would be better to be
explicit, like:

  /*
   * like ends_with, but return the truncated size of str via
   * the "len" parameter.
   */
  int strip_suffix(const char *str, const char *suffix, size_t *len)
  {
	size_t suflen = strlen(suffix);
	*len = strlen(str);

	if (len < suflen)
		return 0;
	else if (strcmp(str + len - suflen, suffix))
		return 0;
	else {
		*len -= suflen;
		return 1;
	}
  }

  int idx_matches_pack(const char *idx, const char *pack)[
  {
	size_t idx_len, pack_len;
	if (!strip_suffix(idx, ".idx", &idx_len) ||
	    !strip_suffix(pack, ".pack", &pack_len))
		return 0;
	if (idx_len != pack_len)
		return 0;
	return !memcmp(idx, pack, idx_len);
  }

You'd perhaps want to split idx_matches_pack across the loop below (our
idx is actually a strbuf, so you can reuse path->len to avoid a strlen,
and you do not have to verify idx_len each time through the loop).

I think strip_suffix would have other uses, too. It's a more featureful
version of ends_with, as skip_prefix is to starts_with. E.g.,
builtin/remote.c:config_read_branches could use it to avoid some magic
numbers.

> diff --git a/sha1_file.c b/sha1_file.c
> index 394fa45..8adab14 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
>  		if (has_extension(de->d_name, ".idx")) {
>  			/* Don't reopen a pack we already have. */

If we don't follow my suggestion above, we still have this
has_extension. This is a reimplementation of ends_with, isn't it? We can
probably drop it and just use ends_with.

-Peff

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

[Duy Nguyen]

On Mon, Jun 30, 2014 at 8:43 PM, Jeff King <peff@peff.net> wrote:
>> diff --git a/sha1_file.c b/sha1_file.c
>> index 394fa45..8adab14 100644
>> --- a/sha1_file.c
>> +++ b/sha1_file.c
>> @@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
>>               if (has_extension(de->d_name, ".idx")) {
>>                       /* Don't reopen a pack we already have. */
>
> If we don't follow my suggestion above, we still have this
> has_extension. This is a reimplementation of ends_with, isn't it? We can
> probably drop it and just use ends_with.

This calls for another patch if we just want to kill has_extension()
in favor of ends_with(). There are 12 call sites of it.
-- 
Duy

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

[Jeff King]

On Mon, Jun 30, 2014 at 08:59:53PM +0700, Duy Nguyen wrote:

> On Mon, Jun 30, 2014 at 8:43 PM, Jeff King <peff@peff.net> wrote:
> >> diff --git a/sha1_file.c b/sha1_file.c
> >> index 394fa45..8adab14 100644
> >> --- a/sha1_file.c
> >> +++ b/sha1_file.c
> >> @@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
> >>               if (has_extension(de->d_name, ".idx")) {
> >>                       /* Don't reopen a pack we already have. */
> >
> > If we don't follow my suggestion above, we still have this
> > has_extension. This is a reimplementation of ends_with, isn't it? We can
> > probably drop it and just use ends_with.
> 
> This calls for another patch if we just want to kill has_extension()
> in favor of ends_with(). There are 12 call sites of it.

Yes. Some of those would want to become ends_with, and some would
actually want to become strip_suffix. I'm working up a series now.

-Peff

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

[René Scharfe]

Am 30.06.2014 15:43, schrieb Jeff King:
> On Sun, Jun 29, 2014 at 07:56:25AM +0200, René Scharfe wrote:
> 
>> Avoid overrunning the existing pack name (p->pack_name, a C string) in
>> the case that the new path is longer by using strncmp instead of memcmp
>> for comparing.  While at it, replace the magic constant 4 with a
>> strlen call to document its meaning.
> 
> An unwritten assumption with this code is that we have "foo.idx" and we
> want to make sure that we are matching "foo.pack" from the existing pack
> list. Both before and after your patch, I think we would match
> "foobar.pack".

Yes, indeed.

> It's probably not a big deal, as we don't expect random
> junk in the pack directory, but I wonder if it would be better to be
> explicit, like:

<snip>

Here's a simpler approach:

-- >8 --
Subject: [PATCH v2 3/2] sha1_file: more precise packname matching

Consider the full length of the already loaded pack names when checking
for duplicates.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 sha1_file.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index b7ad6c1..a13fbbd 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1206,10 +1206,12 @@ static void prepare_packed_git_one(char *objdir, int local)
 		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
+			size_t len = path.len - strlen(".idx");
+
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!strncmp(path.buf, p->pack_name,
-					     path.len - strlen(".idx")))
+				if (!strncmp(p->pack_name, path.buf, len) &&
+				    !strcmp(p->pack_name + len, ".pack"))
 					break;
 			}
 			if (p == NULL &&
-- 
2.0.0

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

[René Scharfe]

Am 30.06.2014 16:22, schrieb Jeff King:
> On Mon, Jun 30, 2014 at 08:59:53PM +0700, Duy Nguyen wrote:
>
>> On Mon, Jun 30, 2014 at 8:43 PM, Jeff King <peff@peff.net> wrote:
>>>> diff --git a/sha1_file.c b/sha1_file.c
>>>> index 394fa45..8adab14 100644
>>>> --- a/sha1_file.c
>>>> +++ b/sha1_file.c
>>>> @@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
>>>>                if (has_extension(de->d_name, ".idx")) {
>>>>                        /* Don't reopen a pack we already have. */
>>>
>>> If we don't follow my suggestion above, we still have this
>>> has_extension. This is a reimplementation of ends_with, isn't it? We can
>>> probably drop it and just use ends_with.
>>
>> This calls for another patch if we just want to kill has_extension()
>> in favor of ends_with(). There are 12 call sites of it.
>
> Yes. Some of those would want to become ends_with, and some would
> actually want to become strip_suffix. I'm working up a series now.

NB: has_extension is almost the same as ends_with, but it also checks if 
the string is longer than just the extension:

	ends_with("x", "x") => 1
	has_extension("x", "x") => 0

René

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

[Jeff King]

On Mon, Jun 30, 2014 at 06:43:35PM +0200, René Scharfe wrote:

> Am 30.06.2014 16:22, schrieb Jeff King:
> >On Mon, Jun 30, 2014 at 08:59:53PM +0700, Duy Nguyen wrote:
> >
> >>On Mon, Jun 30, 2014 at 8:43 PM, Jeff King <peff@peff.net> wrote:
> >>>>diff --git a/sha1_file.c b/sha1_file.c
> >>>>index 394fa45..8adab14 100644
> >>>>--- a/sha1_file.c
> >>>>+++ b/sha1_file.c
> >>>>@@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
> >>>>               if (has_extension(de->d_name, ".idx")) {
> >>>>                       /* Don't reopen a pack we already have. */
> >>>
> >>>If we don't follow my suggestion above, we still have this
> >>>has_extension. This is a reimplementation of ends_with, isn't it? We can
> >>>probably drop it and just use ends_with.
> >>
> >>This calls for another patch if we just want to kill has_extension()
> >>in favor of ends_with(). There are 12 call sites of it.
> >
> >Yes. Some of those would want to become ends_with, and some would
> >actually want to become strip_suffix. I'm working up a series now.
> 
> NB: has_extension is almost the same as ends_with, but it also checks if the
> string is longer than just the extension:
> 
> 	ends_with("x", "x") => 1
> 	has_extension("x", "x") => 0

Thanks, I didn't notice that. I don't think the distinction is
important in any callers, but I'll double check.

-Peff

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

[Jeff King]

On Mon, Jun 30, 2014 at 06:35:50PM +0200, René Scharfe wrote:

> > It's probably not a big deal, as we don't expect random
> > junk in the pack directory, but I wonder if it would be better to be
> > explicit, like:
> 
> <snip>
> 
> Here's a simpler approach:

I agree that solves the problem. However, I'm about to post an
alternative series that also replaces has_extension with strip_suffix,
which I think ends up a bit nicer.

-Peff

[PATCH 0/9] add strip_suffix as an alternative to ends_with

[Jeff King]

Here's a series to do for ends_with what the recent skip_prefix series
did for starts_with. Namely: drop some magic numbers and repeated string
literals, and hopefully make things more readable.

The first patch is René's patch 1/2, with the leak fix from Duy and typo
fixes in the commit message from me. The rest are new, and replace the
changes to prepare_packed_git_one done elsewhere in the thread.

  [1/9]: sha1_file: replace PATH_MAX buffer with strbuf in prepare_packed_git_one()
  [2/9]: add strip_suffix function
  [3/9]: implement ends_with via strip_suffix
  [4/9]: replace has_extension with ends_with
  [5/9]: use strip_suffix instead of ends_with in simple cases
  [6/9]: index-pack: use strip_suffix to avoid magic numbers
  [7/9]: strbuf: implement strbuf_strip_suffix
  [8/9]: verify-pack: use strbuf_strip_suffix
  [9/9]: prepare_packed_git_one: refactor duplicate-pack check

-Peff

[PATCH 1/9] sha1_file: replace PATH_MAX buffer with strbuf in prepare_packed_git_one()

[Jeff King]

From: René Scharfe <l.s.r@web.de>

Instead of using strbuf to create a message string in case a path is
too long for our fixed-size buffer, replace that buffer with a strbuf
and thus get rid of the limitation.

Helped-by: Duy Nguyen <pclouds@gmail.com>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Jeff King <peff@peff.net>
---
 sha1_file.c | 42 ++++++++++++++++--------------------------
 1 file changed, 16 insertions(+), 26 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 34d527f..394fa45 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1177,48 +1177,37 @@ static void report_pack_garbage(struct string_list *list)
 
 static void prepare_packed_git_one(char *objdir, int local)
 {
-	/* Ensure that this buffer is large enough so that we can
-	   append "/pack/" without clobbering the stack even if
-	   strlen(objdir) were PATH_MAX.  */
-	char path[PATH_MAX + 1 + 4 + 1 + 1];
-	int len;
+	struct strbuf path = STRBUF_INIT;
+	size_t dirnamelen;
 	DIR *dir;
 	struct dirent *de;
 	struct string_list garbage = STRING_LIST_INIT_DUP;
 
-	sprintf(path, "%s/pack", objdir);
-	len = strlen(path);
-	dir = opendir(path);
+	strbuf_addstr(&path, objdir);
+	strbuf_addstr(&path, "/pack");
+	dir = opendir(path.buf);
 	if (!dir) {
 		if (errno != ENOENT)
 			error("unable to open object pack directory: %s: %s",
-			      path, strerror(errno));
+			      path.buf, strerror(errno));
+		strbuf_release(&path);
 		return;
 	}
-	path[len++] = '/';
+	strbuf_addch(&path, '/');
+	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
-		int namelen = strlen(de->d_name);
 		struct packed_git *p;
 
-		if (len + namelen + 1 > sizeof(path)) {
-			if (report_garbage) {
-				struct strbuf sb = STRBUF_INIT;
-				strbuf_addf(&sb, "%.*s/%s", len - 1, path, de->d_name);
-				report_garbage("path too long", sb.buf);
-				strbuf_release(&sb);
-			}
-			continue;
-		}
-
 		if (is_dot_or_dotdot(de->d_name))
 			continue;
 
-		strcpy(path + len, de->d_name);
+		strbuf_setlen(&path, dirnamelen);
+		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path, p->pack_name, len + namelen - 4))
+				if (!memcmp(path.buf, p->pack_name, path.len - 4))
 					break;
 			}
 			if (p == NULL &&
@@ -1226,7 +1215,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 			     * See if it really is a valid .idx file with
 			     * corresponding .pack file that we can map.
 			     */
-			    (p = add_packed_git(path, len + namelen, local)) != NULL)
+			    (p = add_packed_git(path.buf, path.len, local)) != NULL)
 				install_packed_git(p);
 		}
 
@@ -1237,13 +1226,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		    has_extension(de->d_name, ".pack") ||
 		    has_extension(de->d_name, ".bitmap") ||
 		    has_extension(de->d_name, ".keep"))
-			string_list_append(&garbage, path);
+			string_list_append(&garbage, path.buf);
 		else
-			report_garbage("garbage found", path);
+			report_garbage("garbage found", path.buf);
 	}
 	closedir(dir);
 	report_pack_garbage(&garbage);
 	string_list_clear(&garbage, 0);
+	strbuf_release(&path);
 }
 
 static int sort_pack(const void *a_, const void *b_)
-- 
2.0.0.566.gfe3e6b2

[PATCH 2/9] add strip_suffix function

[Jeff King]

Many callers of ends_with want to not only find out whether
a string has a suffix, but want to also strip it off. Doing
that separately has two minor problems:

  1. We often run over the string twice (once to find
     the suffix, and then once more to find its length to
     subtract the suffix length).

  2. We have to specify the suffix length again, which means
     either a magic number, or repeating ourselves with
     strlen("suffix").

Just as we have skip_prefix to avoid these cases with
starts_with, we can add a strip_suffix to avoid them with
ends_with.

Note that we add two forms of strip_suffix here: one that
takes a string, with the resulting length as an
out-parameter; and one that takes a pointer/length pair, and
reuses the length as an out-parameter. The latter is more
efficient when the caller already has the length (e.g., when
using strbufs), but it can be easy to confuse the two, as
they take the same number and types of parameters.

For that reason, the "mem" form puts its length parameter
next to the buffer (since they are a pair), and the string
form puts it at the end (since it is an out-parameter). The
compiler can notice when you get the order wrong, which
should help prevent writing one when you meant the other.

Signed-off-by: Jeff King <peff@peff.net>
---
I hope the word "strip" is OK, as it does not actually NUL-terminate
(doing so would make it unusable for many cases). Between the comment
below and the "const" in the parameter, I think it should be pretty
clear that it does not touch the string. And I could not think of a
better word.

 git-compat-util.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/git-compat-util.h b/git-compat-util.h
index b6f03b3..d044c42 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -358,6 +358,33 @@ static inline const char *skip_prefix(const char *str, const char *prefix)
 	return NULL;
 }
 
+/*
+ * If buf ends with suffix, return 1 and subtract the length of the suffix
+ * from *len. Otherwise, return 0 and leave *len untouched.
+ */
+static inline int strip_suffix_mem(const char *buf, size_t *len,
+				   const char *suffix)
+{
+	size_t suflen = strlen(suffix);
+	if (*len < suflen || memcmp(buf + (*len - suflen), suffix, suflen))
+		return 0;
+	*len -= suflen;
+	return 1;
+}
+
+/*
+ * If str ends with suffix, return 1 and set *len to the size of the string
+ * without the suffix. Otherwise, return 0 and set *len to the size of the
+ * string.
+ *
+ * Note that we do _not_ NUL-terminate str to the new length.
+ */
+static inline int strip_suffix(const char *str, const char *suffix, size_t *len)
+{
+	*len = strlen(str);
+	return strip_suffix_mem(str, len, suffix);
+}
+
 #if defined(NO_MMAP) || defined(USE_WIN32_MMAP)
 
 #ifndef PROT_READ
-- 
2.0.0.566.gfe3e6b2

[PATCH 3/9] implement ends_with via strip_suffix

[Jeff King]

The ends_with function is essentially a simplified version
of strip_suffix, in which we throw away the stripped length.
Implementing it as an inline on top of strip_suffix has two
advantages:

  1. We save a bit of duplicated code.

  2. The suffix is typically a string literal, and we call
     strlen on it. By making the function inline, many
     compilers can replace the strlen call with a constant.

Signed-off-by: Jeff King <peff@peff.net>
---
 git-compat-util.h | 7 ++++++-
 strbuf.c          | 9 ---------
 2 files changed, 6 insertions(+), 10 deletions(-)

diff --git a/git-compat-util.h b/git-compat-util.h
index d044c42..4cfde49 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -347,7 +347,6 @@ extern void set_error_routine(void (*routine)(const char *err, va_list params));
 extern void set_die_is_recursing_routine(int (*routine)(void));
 
 extern int starts_with(const char *str, const char *prefix);
-extern int ends_with(const char *str, const char *suffix);
 
 static inline const char *skip_prefix(const char *str, const char *prefix)
 {
@@ -385,6 +384,12 @@ static inline int strip_suffix(const char *str, const char *suffix, size_t *len)
 	return strip_suffix_mem(str, len, suffix);
 }
 
+static inline int ends_with(const char *str, const char *suffix)
+{
+	size_t len;
+	return strip_suffix(str, suffix, &len);
+}
+
 #if defined(NO_MMAP) || defined(USE_WIN32_MMAP)
 
 #ifndef PROT_READ
diff --git a/strbuf.c b/strbuf.c
index ac62982..99dbeba 100644
--- a/strbuf.c
+++ b/strbuf.c
@@ -11,15 +11,6 @@ int starts_with(const char *str, const char *prefix)
 			return 0;
 }
 
-int ends_with(const char *str, const char *suffix)
-{
-	int len = strlen(str), suflen = strlen(suffix);
-	if (len < suflen)
-		return 0;
-	else
-		return !strcmp(str + len - suflen, suffix);
-}
-
 /*
  * Used as the default ->buf value, so that people can always assume
  * buf is non NULL and ->buf is NUL terminated even for a freshly
-- 
2.0.0.566.gfe3e6b2

[PATCH 4/9] replace has_extension with ends_with

[Jeff King]

These two are almost the same function, with the exception
that has_extension only matches if there is content before
the suffix. So ends_with(".exe", ".exe") is true, but
has_extension would not be.

This distinction does not matter to any of the callers,
though, and we can just replace uses of has_extension with
ends_with. We prefer the "ends_with" name because it is more
generic, and there is nothing about the function that
requires it to be used for file extensions.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/index-pack.c  |  4 ++--
 builtin/verify-pack.c |  4 ++--
 git-compat-util.h     |  7 -------
 help.c                |  2 +-
 refs.c                |  4 ++--
 sha1_file.c           | 10 +++++-----
 6 files changed, 12 insertions(+), 19 deletions(-)

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 18f57de..46376b6 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1603,7 +1603,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 		die(_("--fix-thin cannot be used without --stdin"));
 	if (!index_name && pack_name) {
 		int len = strlen(pack_name);
-		if (!has_extension(pack_name, ".pack"))
+		if (!ends_with(pack_name, ".pack"))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
 		index_name_buf = xmalloc(len);
@@ -1613,7 +1613,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	}
 	if (keep_msg && !keep_name && pack_name) {
 		int len = strlen(pack_name);
-		if (!has_extension(pack_name, ".pack"))
+		if (!ends_with(pack_name, ".pack"))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
 		keep_name_buf = xmalloc(len);
diff --git a/builtin/verify-pack.c b/builtin/verify-pack.c
index 66cd2df..2fd29ce 100644
--- a/builtin/verify-pack.c
+++ b/builtin/verify-pack.c
@@ -27,9 +27,9 @@ static int verify_one_pack(const char *path, unsigned int flags)
 	 * normalize these forms to "foo.pack" for "index-pack --verify".
 	 */
 	strbuf_addstr(&arg, path);
-	if (has_extension(arg.buf, ".idx"))
+	if (ends_with(arg.buf, ".idx"))
 		strbuf_splice(&arg, arg.len - 3, 3, "pack", 4);
-	else if (!has_extension(arg.buf, ".pack"))
+	else if (!ends_with(arg.buf, ".pack"))
 		strbuf_add(&arg, ".pack", 5);
 	argv[2] = arg.buf;
 
diff --git a/git-compat-util.h b/git-compat-util.h
index 4cfde49..8cf79ae 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -594,13 +594,6 @@ static inline size_t xsize_t(off_t len)
 	return (size_t)len;
 }
 
-static inline int has_extension(const char *filename, const char *ext)
-{
-	size_t len = strlen(filename);
-	size_t extlen = strlen(ext);
-	return len > extlen && !memcmp(filename + len - extlen, ext, extlen);
-}
-
 /* in ctype.c, for kwset users */
 extern const char tolower_trans_tbl[256];
 
diff --git a/help.c b/help.c
index b266b09..372728f 100644
--- a/help.c
+++ b/help.c
@@ -156,7 +156,7 @@ static void list_commands_in_dir(struct cmdnames *cmds,
 			continue;
 
 		entlen = strlen(de->d_name) - prefix_len;
-		if (has_extension(de->d_name, ".exe"))
+		if (ends_with(de->d_name, ".exe"))
 			entlen -= 4;
 
 		add_cmdname(cmds, de->d_name + prefix_len, entlen);
diff --git a/refs.c b/refs.c
index dc45774..e31adfd 100644
--- a/refs.c
+++ b/refs.c
@@ -1162,7 +1162,7 @@ static void read_loose_refs(const char *dirname, struct ref_dir *dir)
 
 		if (de->d_name[0] == '.')
 			continue;
-		if (has_extension(de->d_name, ".lock"))
+		if (ends_with(de->d_name, ".lock"))
 			continue;
 		strbuf_addstr(&refname, de->d_name);
 		refdir = *refs->name
@@ -3233,7 +3233,7 @@ static int do_for_each_reflog(struct strbuf *name, each_ref_fn fn, void *cb_data
 
 		if (de->d_name[0] == '.')
 			continue;
-		if (has_extension(de->d_name, ".lock"))
+		if (ends_with(de->d_name, ".lock"))
 			continue;
 		strbuf_addstr(name, de->d_name);
 		if (stat(git_path("logs/%s", name->buf), &st) < 0) {
diff --git a/sha1_file.c b/sha1_file.c
index 394fa45..93b794f 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1204,7 +1204,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 		strbuf_setlen(&path, dirnamelen);
 		strbuf_addstr(&path, de->d_name);
 
-		if (has_extension(de->d_name, ".idx")) {
+		if (ends_with(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
 				if (!memcmp(path.buf, p->pack_name, path.len - 4))
@@ -1222,10 +1222,10 @@ static void prepare_packed_git_one(char *objdir, int local)
 		if (!report_garbage)
 			continue;
 
-		if (has_extension(de->d_name, ".idx") ||
-		    has_extension(de->d_name, ".pack") ||
-		    has_extension(de->d_name, ".bitmap") ||
-		    has_extension(de->d_name, ".keep"))
+		if (ends_with(de->d_name, ".idx") ||
+		    ends_with(de->d_name, ".pack") ||
+		    ends_with(de->d_name, ".bitmap") ||
+		    ends_with(de->d_name, ".keep"))
 			string_list_append(&garbage, path.buf);
 		else
 			report_garbage("garbage found", path.buf);
-- 
2.0.0.566.gfe3e6b2

[PATCH 5/9] use strip_suffix instead of ends_with in simple cases

[Jeff King]

When stripping a suffix like:

  if (ends_with(str, "foo"))
	buf = xmemdupz(str, strlen(str) - 3);

we can instead use strip_suffix to avoid the constant 3,
which must match the literal "foo" (we sometimes use
strlen("foo") instead, but that means we are repeating
ourselves). The example above becomes:

  if (strip_suffix(str, "foo", &len))
	buf = xmemdupz(str, len);

This also saves a strlen(), since we calculate the string
length when detecting the suffix.

Note that in some cases we also switch from xstrndup to
xmemdupz, which saves a further strlen call.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/remote.c | 13 +++++++------
 builtin/repack.c |  5 ++---
 connected.c      |  6 +++---
 help.c           |  5 ++---
 4 files changed, 14 insertions(+), 15 deletions(-)

diff --git a/builtin/remote.c b/builtin/remote.c
index c9102e8..7c2999e 100644
--- a/builtin/remote.c
+++ b/builtin/remote.c
@@ -265,16 +265,17 @@ static int config_read_branches(const char *key, const char *value, void *cb)
 		struct string_list_item *item;
 		struct branch_info *info;
 		enum { REMOTE, MERGE, REBASE } type;
+		size_t key_len;
 
 		key += 7;
-		if (ends_with(key, ".remote")) {
-			name = xstrndup(key, strlen(key) - 7);
+		if (strip_suffix(key, ".remote", &key_len)) {
+			name = xmemdupz(key, key_len);
 			type = REMOTE;
-		} else if (ends_with(key, ".merge")) {
-			name = xstrndup(key, strlen(key) - 6);
+		} else if (strip_suffix(key, ".merge", &key_len)) {
+			name = xmemdupz(key, key_len);
 			type = MERGE;
-		} else if (ends_with(key, ".rebase")) {
-			name = xstrndup(key, strlen(key) - 7);
+		} else if (strip_suffix(key, ".rebase", &key_len)) {
+			name = xmemdupz(key, key_len);
 			type = REBASE;
 		} else
 			return 0;
diff --git a/builtin/repack.c b/builtin/repack.c
index ff2216a..a77e743 100644
--- a/builtin/repack.c
+++ b/builtin/repack.c
@@ -83,16 +83,15 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list)
 	DIR *dir;
 	struct dirent *e;
 	char *fname;
-	size_t len;
 
 	if (!(dir = opendir(packdir)))
 		return;
 
 	while ((e = readdir(dir)) != NULL) {
-		if (!ends_with(e->d_name, ".pack"))
+		size_t len;
+		if (!strip_suffix(e->d_name, ".pack", &len))
 			continue;
 
-		len = strlen(e->d_name) - strlen(".pack");
 		fname = xmemdupz(e->d_name, len);
 
 		if (!file_exists(mkpath("%s/%s.keep", packdir, fname)))
diff --git a/connected.c b/connected.c
index be0253e..dae9c99 100644
--- a/connected.c
+++ b/connected.c
@@ -31,6 +31,7 @@ static int check_everything_connected_real(sha1_iterate_fn fn,
 	unsigned char sha1[20];
 	int err = 0, ac = 0;
 	struct packed_git *new_pack = NULL;
+	size_t base_len;
 
 	if (fn(cb_data, sha1))
 		return err;
@@ -38,10 +39,9 @@ static int check_everything_connected_real(sha1_iterate_fn fn,
 	if (transport && transport->smart_options &&
 	    transport->smart_options->self_contained_and_connected &&
 	    transport->pack_lockfile &&
-	    ends_with(transport->pack_lockfile, ".keep")) {
+	    strip_suffix(transport->pack_lockfile, ".keep", &base_len)) {
 		struct strbuf idx_file = STRBUF_INIT;
-		strbuf_addstr(&idx_file, transport->pack_lockfile);
-		strbuf_setlen(&idx_file, idx_file.len - 5); /* ".keep" */
+		strbuf_add(&idx_file, transport->pack_lockfile, base_len);
 		strbuf_addstr(&idx_file, ".idx");
 		new_pack = add_packed_git(idx_file.buf, idx_file.len, 1);
 		strbuf_release(&idx_file);
diff --git a/help.c b/help.c
index 372728f..97567c4 100644
--- a/help.c
+++ b/help.c
@@ -145,7 +145,7 @@ static void list_commands_in_dir(struct cmdnames *cmds,
 	len = buf.len;
 
 	while ((de = readdir(dir)) != NULL) {
-		int entlen;
+		size_t entlen;
 
 		if (!starts_with(de->d_name, prefix))
 			continue;
@@ -156,8 +156,7 @@ static void list_commands_in_dir(struct cmdnames *cmds,
 			continue;
 
 		entlen = strlen(de->d_name) - prefix_len;
-		if (ends_with(de->d_name, ".exe"))
-			entlen -= 4;
+		strip_suffix(de->d_name, ".exe", &entlen);
 
 		add_cmdname(cmds, de->d_name + prefix_len, entlen);
 	}
-- 
2.0.0.566.gfe3e6b2

[PATCH 6/9] index-pack: use strip_suffix to avoid magic numbers

[Jeff King]

We also switch to using strbufs, which lets us avoid the
potentially dangerous combination of a manual malloc
followed by a strcpy.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/index-pack.c | 29 ++++++++++++++---------------
 1 file changed, 14 insertions(+), 15 deletions(-)

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 46376b6..d4b77fd 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1505,7 +1505,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	const char *curr_index;
 	const char *index_name = NULL, *pack_name = NULL;
 	const char *keep_name = NULL, *keep_msg = NULL;
-	char *index_name_buf = NULL, *keep_name_buf = NULL;
+	struct strbuf index_name_buf = STRBUF_INIT,
+		      keep_name_buf = STRBUF_INIT;
 	struct pack_idx_entry **idx_objects;
 	struct pack_idx_option opts;
 	unsigned char pack_sha1[20];
@@ -1602,24 +1603,22 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	if (fix_thin_pack && !from_stdin)
 		die(_("--fix-thin cannot be used without --stdin"));
 	if (!index_name && pack_name) {
-		int len = strlen(pack_name);
-		if (!ends_with(pack_name, ".pack"))
+		size_t len;
+		if (!strip_suffix(pack_name, ".pack", &len))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
-		index_name_buf = xmalloc(len);
-		memcpy(index_name_buf, pack_name, len - 5);
-		strcpy(index_name_buf + len - 5, ".idx");
-		index_name = index_name_buf;
+		strbuf_add(&index_name_buf, pack_name, len);
+		strbuf_addstr(&index_name_buf, ".idx");
+		index_name = index_name_buf.buf;
 	}
 	if (keep_msg && !keep_name && pack_name) {
-		int len = strlen(pack_name);
-		if (!ends_with(pack_name, ".pack"))
+		size_t len;
+		if (!strip_suffix(pack_name, ".pack", &len))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
-		keep_name_buf = xmalloc(len);
-		memcpy(keep_name_buf, pack_name, len - 5);
-		strcpy(keep_name_buf + len - 5, ".keep");
-		keep_name = keep_name_buf;
+		strbuf_add(&keep_name_buf, pack_name, len);
+		strbuf_addstr(&keep_name_buf, ".idx");
+		keep_name = keep_name_buf.buf;
 	}
 	if (verify) {
 		if (!index_name)
@@ -1667,8 +1666,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	else
 		close(input_fd);
 	free(objects);
-	free(index_name_buf);
-	free(keep_name_buf);
+	strbuf_release(&index_name_buf);
+	strbuf_release(&keep_name_buf);
 	if (pack_name == NULL)
 		free((void *) curr_pack);
 	if (index_name == NULL)
-- 
2.0.0.566.gfe3e6b2

[PATCH 7/9] strbuf: implement strbuf_strip_suffix

[Jeff King]

You can almost get away with just calling "strip_suffix_mem"
on a strbuf's buf and len fields. But we also need to move
the NUL-terminator to satisfy strbuf's invariants. Let's
provide a convenience wrapper that handles this.

Signed-off-by: Jeff King <peff@peff.net>
---
I called strbuf_setlen here because it seemed sensible to use that as an
opaque building block, but we are violating the invariant here for a
moment by setting sb->len for a moment. I think that is fine, as this is
a strbuf function, and can violate the invariant for a moment if it
wants to.

But if we care, we can keep a separate "size_t len" variable, and
strbuf_setlen to that.

Or we can make it less opaque, and replace the setlen with
"sb->buf[sb->len] = 0".

 strbuf.h | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/strbuf.h b/strbuf.h
index e9ad03e..ec11742 100644
--- a/strbuf.h
+++ b/strbuf.h
@@ -49,6 +49,15 @@ extern int strbuf_reencode(struct strbuf *sb, const char *from, const char *to);
 extern void strbuf_tolower(struct strbuf *sb);
 extern int strbuf_cmp(const struct strbuf *, const struct strbuf *);
 
+static inline int strbuf_strip_suffix(struct strbuf *sb, const char *suffix)
+{
+	if (strip_suffix_mem(sb->buf, &sb->len, suffix)) {
+		strbuf_setlen(sb, sb->len);
+		return 1;
+	} else
+		return 0;
+}
+
 /*
  * Split str (of length slen) at the specified terminator character.
  * Return a null-terminated array of pointers to strbuf objects
-- 
2.0.0.566.gfe3e6b2

[PATCH 8/9] verify-pack: use strbuf_strip_suffix

[Jeff King]

In this code, we try to convert both "foo.idx" and "foo"
into "foo.pack". By stripping the suffix, we can avoid a
confusing use of strbuf_splice, and make it clear that both
cases are adding ".pack" to the end.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/verify-pack.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/builtin/verify-pack.c b/builtin/verify-pack.c
index 2fd29ce..972579f 100644
--- a/builtin/verify-pack.c
+++ b/builtin/verify-pack.c
@@ -27,10 +27,9 @@ static int verify_one_pack(const char *path, unsigned int flags)
 	 * normalize these forms to "foo.pack" for "index-pack --verify".
 	 */
 	strbuf_addstr(&arg, path);
-	if (ends_with(arg.buf, ".idx"))
-		strbuf_splice(&arg, arg.len - 3, 3, "pack", 4);
-	else if (!ends_with(arg.buf, ".pack"))
-		strbuf_add(&arg, ".pack", 5);
+	if (strbuf_strip_suffix(&arg, ".idx") ||
+	    !ends_with(arg.buf, ".pack"))
+		strbuf_addstr(&arg, ".pack");
 	argv[2] = arg.buf;
 
 	memset(&index_pack, 0, sizeof(index_pack));
-- 
2.0.0.566.gfe3e6b2

[PATCH 9/9] prepare_packed_git_one: refactor duplicate-pack check

[Jeff King]

When we are reloading the list of packs, we check whether a
particular pack has been loaded. This is slightly tricky,
because we load packs based on the presence of their ".idx"
files, but record the name of the matching ".pack" file.
Therefore we want to compare their bases.

The existing code stripped off ".idx" from a file we found,
then compared that whole base length to strings containing
the ".pack" version. This meant we could end up comparing
bytes past what the ".pack" string contained, if the ".idx"
file name was much longer.

In practice, it worked OK because memcmp would end up seeing
a difference in the two strings and would return early
before hitting the full length. However, memcmp may
sometimes read extra bytes past a difference (e.g., because
it is comparing 64-bit words), or is even free to compare in
reverse order.

Furthermore, our memcmp made no guarantees that we matched
the whole pack name, up to ".pack". So "foo.idx" would match
"foo-bar.pack", which is wrong (but does not typically
happen, because our pack names have a fixed size).

We can fix both issues, avoid magic numbers, and document
that we expect to compare against a string with ".pack" by
using strip_suffix.

Signed-off-by: Jeff King <peff@peff.net>
---
This is a teeny bit less efficient than it could be, because we are
verifying our assumption at run-time that each pack name ends in
".pack". I'd venture to say if we cared about efficiency here, the low
hanging fruit would be to avoid the O(n^2) loop to find duplicate pack
names in the first place.

 sha1_file.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 93b794f..129a4c5 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1197,6 +1197,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
 		struct packed_git *p;
+		size_t base_len;
 
 		if (is_dot_or_dotdot(de->d_name))
 			continue;
@@ -1204,10 +1205,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		strbuf_setlen(&path, dirnamelen);
 		strbuf_addstr(&path, de->d_name);
 
-		if (ends_with(de->d_name, ".idx")) {
+		base_len = path.len;
+		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path.buf, p->pack_name, path.len - 4))
+				size_t len;
+				if (strip_suffix(p->pack_name, ".pack", &len) &&
+				    len == base_len &&
+				    !memcmp(p->pack_name, path.buf, len))
 					break;
 			}
 			if (p == NULL &&
-- 
2.0.0.566.gfe3e6b2

Re: [PATCH 0/9] add strip_suffix as an alternative to ends_with

[Junio C Hamano]

On Mon, Jun 30, 2014 at 9:55 AM, Jeff King <peff@peff.net> wrote:
> Here's a series to do for ends_with what the recent skip_prefix series
> did for starts_with. Namely: drop some magic numbers and repeated string
> literals, and hopefully make things more readable.
>
> The first patch is René's patch 1/2, with the leak fix from Duy and typo
> fixes in the commit message from me. The rest are new, and replace the
> changes to prepare_packed_git_one done elsewhere in the thread.
>
>   [1/9]: sha1_file: replace PATH_MAX buffer with strbuf in prepare_packed_git_one()
>   [2/9]: add strip_suffix function
>   [3/9]: implement ends_with via strip_suffix
>   [4/9]: replace has_extension with ends_with
>   [5/9]: use strip_suffix instead of ends_with in simple cases
>   [6/9]: index-pack: use strip_suffix to avoid magic numbers
>   [7/9]: strbuf: implement strbuf_strip_suffix
>   [8/9]: verify-pack: use strbuf_strip_suffix
>   [9/9]: prepare_packed_git_one: refactor duplicate-pack check

Looked very sensible. Thanks all ;-)

Re: [PATCH 2/9] add strip_suffix function

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> For that reason, the "mem" form puts its length parameter
> next to the buffer (since they are a pair), and the string
> form puts it at the end (since it is an out-parameter). The
> compiler can notice when you get the order wrong, which
> should help prevent writing one when you meant the other.

Very sensible consideration.  I like commits that careful thinking
behind them shows through them.

> Signed-off-by: Jeff King <peff@peff.net>
> ---
> I hope the word "strip" is OK, as it does not actually NUL-terminate
> (doing so would make it unusable for many cases). Between the comment
> below and the "const" in the parameter, I think it should be pretty
> clear that it does not touch the string. And I could not think of a
> better word.

All other words I can think of offhand, trim, chomp, etc., hint
shortening of the input string, and by definition shortening of a
string implies NUL-termination.

The "mem" variant deals with a counted string, however, so its
shortening implies NUL-termination a lot less [*1*] and should be
fine.

If we want to avoid implying NUL-termination, the only way to do so
would be to use wording that does not hint shortening.  At least for
the C-string variant, which is measuring the length of the basename
part (i.e. `basename $str $suffix`) without touching anything else,
e.g. basename_length("hello.c", ".c", &len), but at the same time
you want to make it a boolean to signal if the string ends with the
suffix, so perhaps has_suffix("hello.c", ".c", &len)?

[Footnote]

 *1* ... but not entirely, because we often NUL-terminate even
     counted strings (the buffer returned from read_sha1_file() and
     the payload of strbuf are two examples).

>  git-compat-util.h | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
>
> diff --git a/git-compat-util.h b/git-compat-util.h
> index b6f03b3..d044c42 100644
> --- a/git-compat-util.h
> +++ b/git-compat-util.h
> @@ -358,6 +358,33 @@ static inline const char *skip_prefix(const char *str, const char *prefix)
>  	return NULL;
>  }
>  
> +/*
> + * If buf ends with suffix, return 1 and subtract the length of the suffix
> + * from *len. Otherwise, return 0 and leave *len untouched.
> + */
> +static inline int strip_suffix_mem(const char *buf, size_t *len,
> +				   const char *suffix)
> +{
> +	size_t suflen = strlen(suffix);
> +	if (*len < suflen || memcmp(buf + (*len - suflen), suffix, suflen))
> +		return 0;
> +	*len -= suflen;
> +	return 1;
> +}
> +
> +/*
> + * If str ends with suffix, return 1 and set *len to the size of the string
> + * without the suffix. Otherwise, return 0 and set *len to the size of the
> + * string.
> + *
> + * Note that we do _not_ NUL-terminate str to the new length.
> + */
> +static inline int strip_suffix(const char *str, const char *suffix, size_t *len)
> +{
> +	*len = strlen(str);
> +	return strip_suffix_mem(str, len, suffix);
> +}
> +
>  #if defined(NO_MMAP) || defined(USE_WIN32_MMAP)
>  
>  #ifndef PROT_READ

Re: [PATCH 2/9] add strip_suffix function

[Jeff King]

On Wed, Jul 02, 2014 at 08:54:44AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > For that reason, the "mem" form puts its length parameter
> > next to the buffer (since they are a pair), and the string
> > form puts it at the end (since it is an out-parameter). The
> > compiler can notice when you get the order wrong, which
> > should help prevent writing one when you meant the other.
> 
> Very sensible consideration.  I like commits that careful thinking
> behind them shows through them.

I would like to take credit for advanced thinking, but I actually did
what felt natural, and only noticed the "compiler will tell you when you
are wrong" effect when I got it wrong while writing a later patch in the
series. :)

> If we want to avoid implying NUL-termination, the only way to do so
> would be to use wording that does not hint shortening.  At least for
> the C-string variant, which is measuring the length of the basename
> part (i.e. `basename $str $suffix`) without touching anything else,
> e.g. basename_length("hello.c", ".c", &len), but at the same time
> you want to make it a boolean to signal if the string ends with the
> suffix, so perhaps has_suffix("hello.c", ".c", &len)?

I think that invites some confusion with "ends_with", which is the same
thing (but just does not take the "len" parameter). We could just add
this feature to ends_with, and ask callers who do not care to pass NULL,
but that makes those call sites uglier.

Having had a day to mull it over, and having read your email, I think I
still prefer strip_suffix.

-Peff

Re: [PATCH 2/9] add strip_suffix function

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Having had a day to mull it over, and having read your email, I think I
> still prefer strip_suffix.

That's OK.  I was only trying to help you explore different avenues,
without having strong preference myself either way.