Bug?: git archive exclude pathspec and gitattributes export-ignore

Bug?: git archive exclude pathspec and gitattributes export-ignore

[David Adam]

Hi all,

I think I have a bug in git (tested 2.11.0 on Debian 8, 2.14.1 on OS X and 
2.14.1.145.gb3622a4 on OS X).

Given a repository with an export-ignore directive for a subdirectory in 
.gitattributes, `git archive` with a pathspec that excludes a different 
subdirectory produces no output file and git exits with -1 as the return 
status.

As shown:

   > git init foo && cd foo
   Initialized empty Git repository in /Users/david/src/foo/.git/
   > mkdir a b
   > touch {a,b}/somefile
   > echo "/a export-ignore" >> .gitattributes
   > git add .
   > git commit -m "Initial commit"
   [master (root-commit) 53527a7] Initial commit
    3 files changed, 1 insertion(+)
    create mode 100644 .gitattributes
    create mode 100644 a/somefile
    create mode 100644 b/somefile
   > git archive --verbose master ':(top)' ':(exclude)b*'
   .gitattributes
   > echo $?
   255

If this is intended behaviour, is there any way of achieving the goal of 
excluding a subdirectory not listed as export-ignore? Using the exclude 
pathspec ":(exclude)b" produces an empty subdirectory b in the output, 
which I would like to avoid.

This is a reduced testcase; my goal is to end up with two archives, one 
containing directory b only, and one containing everything except for 
directory b - so I can't just add 'b export-ignore' to gitattributes.

Thanks

David Adam
zanchey@ucc.gu.uwa.edu.au

Re: Bug?: git archive exclude pathspec and gitattributes export-ignore

[René Scharfe]

Am 13.08.2017 um 06:53 schrieb David Adam:
> Hi all,
> 
> I think I have a bug in git (tested 2.11.0 on Debian 8, 2.14.1 on OS X and
> 2.14.1.145.gb3622a4 on OS X).
> 
> Given a repository with an export-ignore directive for a subdirectory in
> .gitattributes, `git archive` with a pathspec that excludes a different
> subdirectory produces no output file and git exits with -1 as the return
> status.
> 
> As shown:
> 
>     > git init foo && cd foo
>     Initialized empty Git repository in /Users/david/src/foo/.git/
>     > mkdir a b
>     > touch {a,b}/somefile
>     > echo "/a export-ignore" >> .gitattributes
>     > git add .
>     > git commit -m "Initial commit"
>     [master (root-commit) 53527a7] Initial commit
>      3 files changed, 1 insertion(+)
>      create mode 100644 .gitattributes
>      create mode 100644 a/somefile
>      create mode 100644 b/somefile
>     > git archive --verbose master ':(top)' ':(exclude)b*'
>     .gitattributes
>     > echo $?
>     255
> 
> If this is intended behaviour, is there any way of achieving the goal of
> excluding a subdirectory not listed as export-ignore? Using the exclude
> pathspec ":(exclude)b" produces an empty subdirectory b in the output,
> which I would like to avoid.
> 
> This is a reduced testcase; my goal is to end up with two archives, one
> containing directory b only, and one containing everything except for
> directory b - so I can't just add 'b export-ignore' to gitattributes.

Thanks for the thoughtful bug report!

The problem seems to be that archive.c::write_archive_entry() returns 0
instead of READ_TREE_RECURSIVE for directories with the attribute
"export-ignore", and archive.c::write_directory() gets caught by
surprise by that and returns -1, which ends up causing git archive to
exit with return code 255 without actually writing anything.

This should only happen if you use wildcards like "*", i.e. git archive
should behave as expected if you spell out the full name of the
directory.  Can you confirm that?

The real solution is probably to teach tree-walk.c::do_match() how to
handle attributes and then inject ":(attr:-export-ignore)" as a default
internal pathspec in archive.c::parse_pathspec_arg() instead of handling
it in archive.c::write_archive_entry().

@Duy: What do you think?

Thanks,
René

Re: Bug?: git archive exclude pathspec and gitattributes export-ignore

[David Adam]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1676 bytes --]

On Mon, 14 Aug 2017, René Scharfe wrote:
> Am 13.08.2017 um 06:53 schrieb David Adam:
> > I think I have a bug in git (tested 2.11.0 on Debian 8, 2.14.1 on OS X and
> > 2.14.1.145.gb3622a4 on OS X).
> > 
> > Given a repository with an export-ignore directive for a subdirectory in
> > .gitattributes, `git archive` with a pathspec that excludes a different
> > subdirectory produces no output file and git exits with -1 as the return
> > status.
> 
> Thanks for the thoughtful bug report!
> 
> The problem seems to be that archive.c::write_archive_entry() returns 0
> instead of READ_TREE_RECURSIVE for directories with the attribute
> "export-ignore", and archive.c::write_directory() gets caught by
> surprise by that and returns -1, which ends up causing git archive to
> exit with return code 255 without actually writing anything.
> 
> This should only happen if you use wildcards like "*", i.e. git archive
> should behave as expected if you spell out the full name of the
> directory.  Can you confirm that?

Yes - that's definitely the case.

The reason I am trying to use the wildcard is that using an ":(exclude)b" 
pathspec excludes the contents of, but not the actual b directory 
itself:

    > git archive HEAD ':(top)' ':(exclude)b' | tar t
    .gitattributes
    b/

Whereas I would like to export the archive without the b directory 
entirely.

> The real solution is probably to teach tree-walk.c::do_match() how to
> handle attributes and then inject ":(attr:-export-ignore)" as a default
> internal pathspec in archive.c::parse_pathspec_arg() instead of handling
> it in archive.c::write_archive_entry().

Many thanks

David Adam
zanchey@ucc.gu.uwa.edu.au

Re: Bug?: git archive exclude pathspec and gitattributes export-ignore

[René Scharfe]

Am 14.08.2017 um 18:43 schrieb René Scharfe:
> The real solution is probably to teach tree-walk.c::do_match() how to
> handle attributes and then inject ":(attr:-export-ignore)" as a default
> internal pathspec in archive.c::parse_pathspec_arg() instead of handling
> it in archive.c::write_archive_entry().

That's complicated and I'm not sure anymore if it's even a good idea.
Let's solve this in git archive for now.

  t5001: add tests for export-ignore attributes and exclude pathspecs
  archive: factor out helper functions for handling attributes
  archive: don't queue excluded directories
  archive: queue directories for all types of pathspecs

 archive.c               | 49 +++++++++++++++++++++++++++++++++++++++++--------
 t/t5001-archive-attr.sh | 47 ++++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 85 insertions(+), 11 deletions(-)

-- 
2.14.1

[PATCH 1/4] t5001: add tests for export-ignore attributes and exclude pathspecs

[René Scharfe]

Demonstrate mishandling of the attribute export-ignore by git archive
when used together with pathspecs.  Wildcard pathspecs can even cause it
to abort.  And a directory excluded without a wildcard is still included
as an empty folder in the archive.

Test-case-by: David Adam <zanchey@ucc.gu.uwa.edu.au>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 t/t5001-archive-attr.sh | 47 ++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 44 insertions(+), 3 deletions(-)

diff --git a/t/t5001-archive-attr.sh b/t/t5001-archive-attr.sh
index b04d955bfa..063622bc71 100755
--- a/t/t5001-archive-attr.sh
+++ b/t/t5001-archive-attr.sh
@@ -7,11 +7,15 @@ test_description='git archive attribute tests'
 SUBSTFORMAT='%H (%h)%n'
 
 test_expect_exists() {
-	test_expect_success " $1 exists" "test -e $1"
+	test_expect_${2:-success} " $1 exists" "test -e $1"
 }
 
 test_expect_missing() {
-	test_expect_success " $1 does not exist" "test ! -e $1"
+	test_expect_${2:-success} " $1 does not exist" "test ! -e $1"
+}
+
+extract_tar_to_dir () {
+	(mkdir "$1" && cd "$1" && "$TAR" xf -) <"$1.tar"
 }
 
 test_expect_success 'setup' '
@@ -21,12 +25,19 @@ test_expect_success 'setup' '
 
 	echo ignored by tree >ignored-by-tree &&
 	echo ignored-by-tree export-ignore >.gitattributes &&
-	git add ignored-by-tree .gitattributes &&
+	mkdir ignored-by-tree.d &&
+	>ignored-by-tree.d/file &&
+	echo ignored-by-tree.d export-ignore >>.gitattributes &&
+	git add ignored-by-tree ignored-by-tree.d .gitattributes &&
 
 	echo ignored by worktree >ignored-by-worktree &&
 	echo ignored-by-worktree export-ignore >.gitattributes &&
 	git add ignored-by-worktree &&
 
+	mkdir excluded-by-pathspec.d &&
+	>excluded-by-pathspec.d/file &&
+	git add excluded-by-pathspec.d &&
+
 	printf "A\$Format:%s\$O" "$SUBSTFORMAT" >nosubstfile &&
 	printf "A\$Format:%s\$O" "$SUBSTFORMAT" >substfile1 &&
 	printf "A not substituted O" >substfile2 &&
@@ -46,7 +57,37 @@ test_expect_success 'git archive' '
 
 test_expect_missing	archive/ignored
 test_expect_missing	archive/ignored-by-tree
+test_expect_missing	archive/ignored-by-tree.d
+test_expect_missing	archive/ignored-by-tree.d/file
 test_expect_exists	archive/ignored-by-worktree
+test_expect_exists	archive/excluded-by-pathspec.d
+test_expect_exists	archive/excluded-by-pathspec.d/file
+
+test_expect_success 'git archive with pathspec' '
+	git archive HEAD ":!excluded-by-pathspec.d" >archive-pathspec.tar &&
+	extract_tar_to_dir archive-pathspec
+'
+
+test_expect_missing	archive-pathspec/ignored
+test_expect_missing	archive-pathspec/ignored-by-tree
+test_expect_missing	archive-pathspec/ignored-by-tree.d
+test_expect_missing	archive-pathspec/ignored-by-tree.d/file
+test_expect_exists	archive-pathspec/ignored-by-worktree
+test_expect_missing	archive-pathspec/excluded-by-pathspec.d failure
+test_expect_missing	archive-pathspec/excluded-by-pathspec.d/file
+
+test_expect_failure 'git archive with wildcard pathspec' '
+	git archive HEAD ":!excluded-by-p*" >archive-pathspec-wildcard.tar &&
+	extract_tar_to_dir archive-pathspec-wildcard
+'
+
+test_expect_missing	archive-pathspec-wildcard/ignored
+test_expect_missing	archive-pathspec-wildcard/ignored-by-tree
+test_expect_missing	archive-pathspec-wildcard/ignored-by-tree.d
+test_expect_missing	archive-pathspec-wildcard/ignored-by-tree.d/file
+test_expect_exists	archive-pathspec-wildcard/ignored-by-worktree failure
+test_expect_missing	archive-pathspec-wildcard/excluded-by-pathspec.d
+test_expect_missing	archive-pathspec-wildcard/excluded-by-pathspec.d/file
 
 test_expect_success 'git archive with worktree attributes' '
 	git archive --worktree-attributes HEAD >worktree.tar &&
-- 
2.14.1

[PATCH 2/4] archive: factor out helper functions for handling attributes

[René Scharfe]

Add helpers for accessing attributes that encapsulate the details of how
to retrieve their values.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 archive.c | 31 +++++++++++++++++++++++--------
 1 file changed, 23 insertions(+), 8 deletions(-)

diff --git a/archive.c b/archive.c
index 557dd2db85..8e5f632912 100644
--- a/archive.c
+++ b/archive.c
@@ -103,12 +103,30 @@ struct archiver_context {
 	struct directory *bottom;
 };
 
+static const struct attr_check *get_archive_attrs(const char *path)
+{
+	static struct attr_check *check;
+	if (!check)
+		check = attr_check_initl("export-ignore", "export-subst", NULL);
+	return git_check_attr(path, check) ? NULL : check;
+}
+
+static int check_attr_export_ignore(const struct attr_check *check)
+{
+	return check && ATTR_TRUE(check->items[0].value);
+}
+
+static int check_attr_export_subst(const struct attr_check *check)
+{
+	return check && ATTR_TRUE(check->items[1].value);
+}
+
 static int write_archive_entry(const unsigned char *sha1, const char *base,
 		int baselen, const char *filename, unsigned mode, int stage,
 		void *context)
 {
 	static struct strbuf path = STRBUF_INIT;
-	static struct attr_check *check;
+	const struct attr_check *check;
 	struct archiver_context *c = context;
 	struct archiver_args *args = c->args;
 	write_archive_entry_fn_t write_entry = c->write_entry;
@@ -125,13 +143,10 @@ static int write_archive_entry(const unsigned char *sha1, const char *base,
 		strbuf_addch(&path, '/');
 	path_without_prefix = path.buf + args->baselen;
 
-	if (!check)
-		check = attr_check_initl("export-ignore", "export-subst", NULL);
-	if (!git_check_attr(path_without_prefix, check)) {
-		if (ATTR_TRUE(check->items[0].value))
-			return 0;
-		args->convert = ATTR_TRUE(check->items[1].value);
-	}
+	check = get_archive_attrs(path_without_prefix);
+	if (check_attr_export_ignore(check))
+		return 0;
+	args->convert = check_attr_export_subst(check);
 
 	if (S_ISDIR(mode) || S_ISGITLINK(mode)) {
 		if (args->verbose)
-- 
2.14.1

[PATCH 3/4] archive: don't queue excluded directories

[René Scharfe]

Reject directories with the attribute export-ignore already while
queuing them.  This prevents read_tree_recursive() from descending into
them and this avoids write_archive_entry() rejecting them later on,
which queue_or_write_archive_entry() is not prepared for.

Borrow the existing strbuf to build the full path to avoid string
copies and extra allocations; just make sure we restore the original
value before moving on.

Keep checking any other attributes in write_archive_entry() as before,
but avoid checking them twice.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 archive.c               | 32 +++++++++++++++++++++++++-------
 t/t5001-archive-attr.sh |  4 ++--
 2 files changed, 27 insertions(+), 9 deletions(-)

diff --git a/archive.c b/archive.c
index 8e5f632912..1ab8d3a1d7 100644
--- a/archive.c
+++ b/archive.c
@@ -121,17 +121,21 @@ static int check_attr_export_subst(const struct attr_check *check)
 	return check && ATTR_TRUE(check->items[1].value);
 }
 
+static int should_queue_directories(const struct archiver_args *args)
+{
+	return args->pathspec.has_wildcard;
+}
+
 static int write_archive_entry(const unsigned char *sha1, const char *base,
 		int baselen, const char *filename, unsigned mode, int stage,
 		void *context)
 {
 	static struct strbuf path = STRBUF_INIT;
-	const struct attr_check *check;
 	struct archiver_context *c = context;
 	struct archiver_args *args = c->args;
 	write_archive_entry_fn_t write_entry = c->write_entry;
-	const char *path_without_prefix;
 	int err;
+	const char *path_without_prefix;
 
 	args->convert = 0;
 	strbuf_reset(&path);
@@ -143,10 +147,13 @@ static int write_archive_entry(const unsigned char *sha1, const char *base,
 		strbuf_addch(&path, '/');
 	path_without_prefix = path.buf + args->baselen;
 
-	check = get_archive_attrs(path_without_prefix);
-	if (check_attr_export_ignore(check))
-		return 0;
-	args->convert = check_attr_export_subst(check);
+	if (!S_ISDIR(mode) || !should_queue_directories(args)) {
+		const struct attr_check *check;
+		check = get_archive_attrs(path_without_prefix);
+		if (check_attr_export_ignore(check))
+			return 0;
+		args->convert = check_attr_export_subst(check);
+	}
 
 	if (S_ISDIR(mode) || S_ISGITLINK(mode)) {
 		if (args->verbose)
@@ -219,6 +226,17 @@ static int queue_or_write_archive_entry(const unsigned char *sha1,
 	}
 
 	if (S_ISDIR(mode)) {
+		size_t baselen = base->len;
+		const struct attr_check *check;
+
+		/* Borrow base, but restore its original value when done. */
+		strbuf_addstr(base, filename);
+		strbuf_addch(base, '/');
+		check = get_archive_attrs(base->buf);
+		strbuf_setlen(base, baselen);
+
+		if (check_attr_export_ignore(check))
+			return 0;
 		queue_directory(sha1, base, filename,
 				mode, stage, c);
 		return READ_TREE_RECURSIVE;
@@ -272,7 +290,7 @@ int write_archive_entries(struct archiver_args *args,
 	}
 
 	err = read_tree_recursive(args->tree, "", 0, 0, &args->pathspec,
-				  args->pathspec.has_wildcard ?
+				  should_queue_directories(args) ?
 				  queue_or_write_archive_entry :
 				  write_archive_entry_buf,
 				  &context);
diff --git a/t/t5001-archive-attr.sh b/t/t5001-archive-attr.sh
index 063622bc71..897f6f06d5 100755
--- a/t/t5001-archive-attr.sh
+++ b/t/t5001-archive-attr.sh
@@ -76,7 +76,7 @@ test_expect_exists	archive-pathspec/ignored-by-worktree
 test_expect_missing	archive-pathspec/excluded-by-pathspec.d failure
 test_expect_missing	archive-pathspec/excluded-by-pathspec.d/file
 
-test_expect_failure 'git archive with wildcard pathspec' '
+test_expect_success 'git archive with wildcard pathspec' '
 	git archive HEAD ":!excluded-by-p*" >archive-pathspec-wildcard.tar &&
 	extract_tar_to_dir archive-pathspec-wildcard
 '
@@ -85,7 +85,7 @@ test_expect_missing	archive-pathspec-wildcard/ignored
 test_expect_missing	archive-pathspec-wildcard/ignored-by-tree
 test_expect_missing	archive-pathspec-wildcard/ignored-by-tree.d
 test_expect_missing	archive-pathspec-wildcard/ignored-by-tree.d/file
-test_expect_exists	archive-pathspec-wildcard/ignored-by-worktree failure
+test_expect_exists	archive-pathspec-wildcard/ignored-by-worktree
 test_expect_missing	archive-pathspec-wildcard/excluded-by-pathspec.d
 test_expect_missing	archive-pathspec-wildcard/excluded-by-pathspec.d/file
 
-- 
2.14.1

[PATCH 4/4] archive: queue directories for all types of pathspecs

[René Scharfe]

When read_tree_recursive() encounters a directory excluded by a pathspec
then it enters it anyway because it might contain included entries.  It
calls the callback function before it is able to decide if the directory
is actually needed.

For that reason git archive adds directories to a queue and writes
entries for them only when it encounters the first child item -- but
only if pathspecs with wildcards are used.  Do the same for literal
pathspecs as well, as the reasoning above applies to them, too.  This
prevents git archive from writing entries for excluded directories.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 archive.c               | 2 +-
 t/t5001-archive-attr.sh | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/archive.c b/archive.c
index 1ab8d3a1d7..174c0555b6 100644
--- a/archive.c
+++ b/archive.c
@@ -123,7 +123,7 @@ static int check_attr_export_subst(const struct attr_check *check)
 
 static int should_queue_directories(const struct archiver_args *args)
 {
-	return args->pathspec.has_wildcard;
+	return args->pathspec.nr;
 }
 
 static int write_archive_entry(const unsigned char *sha1, const char *base,
diff --git a/t/t5001-archive-attr.sh b/t/t5001-archive-attr.sh
index 897f6f06d5..e9aa97117a 100755
--- a/t/t5001-archive-attr.sh
+++ b/t/t5001-archive-attr.sh
@@ -73,7 +73,7 @@ test_expect_missing	archive-pathspec/ignored-by-tree
 test_expect_missing	archive-pathspec/ignored-by-tree.d
 test_expect_missing	archive-pathspec/ignored-by-tree.d/file
 test_expect_exists	archive-pathspec/ignored-by-worktree
-test_expect_missing	archive-pathspec/excluded-by-pathspec.d failure
+test_expect_missing	archive-pathspec/excluded-by-pathspec.d
 test_expect_missing	archive-pathspec/excluded-by-pathspec.d/file
 
 test_expect_success 'git archive with wildcard pathspec' '
-- 
2.14.1

Re: [PATCH 4/4] archive: queue directories for all types of pathspecs

[René Scharfe]

Am 19.08.2017 um 07:33 schrieb René Scharfe:
> When read_tree_recursive() encounters a directory excluded by a pathspec
> then it enters it anyway because it might contain included entries.  It
> calls the callback function before it is able to decide if the directory
> is actually needed.
> 
> For that reason git archive adds directories to a queue and writes
> entries for them only when it encounters the first child item -- but
> only if pathspecs with wildcards are used.  Do the same for literal
> pathspecs as well, as the reasoning above applies to them, too.  This
> prevents git archive from writing entries for excluded directories.

This breaks the test "archive empty subtree with no pathspec" in t5004 by
omitting the empty directory from the archive.  Sorry for missing that!

This is kind of a bonus patch, so please discard it for now; the first
three are OK IMHO.

A better version of this patch would at least update t5004 as well, but
there might be a better way.

> 
> Signed-off-by: Rene Scharfe <l.s.r@web.de>
> ---
>   archive.c               | 2 +-
>   t/t5001-archive-attr.sh | 2 +-
>   2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/archive.c b/archive.c
> index 1ab8d3a1d7..174c0555b6 100644
> --- a/archive.c
> +++ b/archive.c
> @@ -123,7 +123,7 @@ static int check_attr_export_subst(const struct attr_check *check)
>   
>   static int should_queue_directories(const struct archiver_args *args)
>   {
> -	return args->pathspec.has_wildcard;
> +	return args->pathspec.nr;
>   }
>   
>   static int write_archive_entry(const unsigned char *sha1, const char *base,
> diff --git a/t/t5001-archive-attr.sh b/t/t5001-archive-attr.sh
> index 897f6f06d5..e9aa97117a 100755
> --- a/t/t5001-archive-attr.sh
> +++ b/t/t5001-archive-attr.sh
> @@ -73,7 +73,7 @@ test_expect_missing	archive-pathspec/ignored-by-tree
>   test_expect_missing	archive-pathspec/ignored-by-tree.d
>   test_expect_missing	archive-pathspec/ignored-by-tree.d/file
>   test_expect_exists	archive-pathspec/ignored-by-worktree
> -test_expect_missing	archive-pathspec/excluded-by-pathspec.d failure
> +test_expect_missing	archive-pathspec/excluded-by-pathspec.d
>   test_expect_missing	archive-pathspec/excluded-by-pathspec.d/file
>   
>   test_expect_success 'git archive with wildcard pathspec' '
> 

Re: [PATCH 4/4] archive: queue directories for all types of pathspecs

[René Scharfe]

Am 19.08.2017 um 18:53 schrieb René Scharfe:
> Am 19.08.2017 um 07:33 schrieb René Scharfe:
>> When read_tree_recursive() encounters a directory excluded by a pathspec
>> then it enters it anyway because it might contain included entries.  It
>> calls the callback function before it is able to decide if the directory
>> is actually needed.
>>
>> For that reason git archive adds directories to a queue and writes
>> entries for them only when it encounters the first child item -- but
>> only if pathspecs with wildcards are used.  Do the same for literal
>> pathspecs as well, as the reasoning above applies to them, too.  This
>> prevents git archive from writing entries for excluded directories.
> 
> This breaks the test "archive empty subtree with no pathspec" in t5004 by

No, it's "archive empty subtree by direct pathspec" that's broken.  Gah!

> omitting the empty directory from the archive.  Sorry for missing that!
> 
> This is kind of a bonus patch, so please discard it for now; the first
> three are OK IMHO.
> 
> A better version of this patch would at least update t5004 as well, but
> there might be a better way.
> 
>>
>> Signed-off-by: Rene Scharfe <l.s.r@web.de>
>> ---
>>    archive.c               | 2 +-
>>    t/t5001-archive-attr.sh | 2 +-
>>    2 files changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/archive.c b/archive.c
>> index 1ab8d3a1d7..174c0555b6 100644
>> --- a/archive.c
>> +++ b/archive.c
>> @@ -123,7 +123,7 @@ static int check_attr_export_subst(const struct attr_check *check)
>>    
>>    static int should_queue_directories(const struct archiver_args *args)
>>    {
>> -	return args->pathspec.has_wildcard;
>> +	return args->pathspec.nr;
>>    }
>>    
>>    static int write_archive_entry(const unsigned char *sha1, const char *base,
>> diff --git a/t/t5001-archive-attr.sh b/t/t5001-archive-attr.sh
>> index 897f6f06d5..e9aa97117a 100755
>> --- a/t/t5001-archive-attr.sh
>> +++ b/t/t5001-archive-attr.sh
>> @@ -73,7 +73,7 @@ test_expect_missing	archive-pathspec/ignored-by-tree
>>    test_expect_missing	archive-pathspec/ignored-by-tree.d
>>    test_expect_missing	archive-pathspec/ignored-by-tree.d/file
>>    test_expect_exists	archive-pathspec/ignored-by-worktree
>> -test_expect_missing	archive-pathspec/excluded-by-pathspec.d failure
>> +test_expect_missing	archive-pathspec/excluded-by-pathspec.d
>>    test_expect_missing	archive-pathspec/excluded-by-pathspec.d/file
>>    
>>    test_expect_success 'git archive with wildcard pathspec' '
>>

Re: [PATCH 4/4] archive: queue directories for all types of pathspecs

[Junio C Hamano]

René Scharfe <l.s.r@web.de> writes:

> No, it's "archive empty subtree by direct pathspec" that's broken.  Gah!
>
>> omitting the empty directory from the archive.  Sorry for missing that!
>> 
>> This is kind of a bonus patch, so please discard it for now; the first
>> three are OK IMHO.

Ah, our mails crossed.  Thanks; will drop the last one for now.

Re: [PATCH 4/4] archive: queue directories for all types of pathspecs

[Jeff King]

On Sat, Aug 19, 2017 at 06:53:26PM +0200, René Scharfe wrote:

> Am 19.08.2017 um 07:33 schrieb René Scharfe:
> > When read_tree_recursive() encounters a directory excluded by a pathspec
> > then it enters it anyway because it might contain included entries.  It
> > calls the callback function before it is able to decide if the directory
> > is actually needed.
> > 
> > For that reason git archive adds directories to a queue and writes
> > entries for them only when it encounters the first child item -- but
> > only if pathspecs with wildcards are used.  Do the same for literal
> > pathspecs as well, as the reasoning above applies to them, too.  This
> > prevents git archive from writing entries for excluded directories.
> 
> This breaks the test "archive empty subtree with no pathspec" in t5004 by
> omitting the empty directory from the archive.  Sorry for missing that!
> 
> This is kind of a bonus patch, so please discard it for now; the first
> three are OK IMHO.
> 
> A better version of this patch would at least update t5004 as well, but
> there might be a better way.

I actually think it would be reasonable to omit the empty directory in
t5004. The main thing we care about there is that we produce an archive
with no files rather than barfing. Checking that the empty directory is
actually there was mostly "this is what it happens to produce" rather
than any conscious decision, I think.

If the new rule is "we omit empty directories", then it would make sense
for us to follow that, regardless of whether it happened by pathspec
limiting or if the tree was empty in the first place (and such an
empty tree is insane anyway; Git tries hard not to create them).

-Peff

Re: [PATCH 4/4] archive: queue directories for all types of pathspecs

[René Scharfe]

Am 20.08.2017 um 11:06 schrieb Jeff King:
> I actually think it would be reasonable to omit the empty directory in
> t5004. The main thing we care about there is that we produce an archive
> with no files rather than barfing. Checking that the empty directory is
> actually there was mostly "this is what it happens to produce" rather
> than any conscious decision, I think.
> 
> If the new rule is "we omit empty directories", then it would make sense
> for us to follow that, regardless of whether it happened by pathspec
> limiting or if the tree was empty in the first place (and such an
> empty tree is insane anyway; Git tries hard not to create them).

Right.

-- >8 --
Subject: [PATCH] archive: don't add empty directories to archives

While git doesn't track empty directories, git archive can be tricked
into putting some into archives.  One way is to construct an empty tree
object, as t5004 does.  While that is supported by the object database,
it can't be represented in the index and thus it's unlikely to occur in
the wild.

Another way is using the literal name of a directory in an exclude
pathspec -- its contents are are excluded, but the directory stub is
included.  That's inconsistent: exclude pathspecs containing wildcards
don't leave empty directories in the archive.

Yet another way is have a few levels of nested subdirectories (e.g.
d1/d2/d3/file1) and ignoring the entries at the leaved (e.g. file1).
The directories with the ignored content are ignored as well (e.g. d3),
but their empty parents are included (e.g. d2).

As empty directories are not supported by git, they should also not be
written into archives.  If an empty directory is really needed then it
can be tracked and archived by placing an empty .gitignore file in it.

There already is a mechanism in place for suppressing empty directories.
When read_tree_recursive() encounters a directory excluded by a pathspec
then it enters it anyway because it might contain included entries.  It
calls the callback function before it is able to decide if the directory
is actually needed.  For that reason git archive adds directories to a
queue and writes entries for them only when it encounters the first
child item -- but currently only if pathspecs with wildcards are used.

Queue *all* directories, no matter if there even are pathspecs present.
This prevents git archive from writing entries for empty directories in
all cases.

Suggested-by: Jeff King <peff@peff.net>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 archive.c                       | 19 ++-----------------
 t/t5001-archive-attr.sh         |  2 +-
 t/t5002-archive-attr-pattern.sh |  2 +-
 t/t5004-archive-corner-cases.sh |  4 ++--
 4 files changed, 6 insertions(+), 21 deletions(-)

diff --git a/archive.c b/archive.c
index 1ab8d3a1d7..1e41f4bbeb 100644
--- a/archive.c
+++ b/archive.c
@@ -121,11 +121,6 @@ static int check_attr_export_subst(const struct attr_check *check)
 	return check && ATTR_TRUE(check->items[1].value);
 }
 
-static int should_queue_directories(const struct archiver_args *args)
-{
-	return args->pathspec.has_wildcard;
-}
-
 static int write_archive_entry(const unsigned char *sha1, const char *base,
 		int baselen, const char *filename, unsigned mode, int stage,
 		void *context)
@@ -147,7 +142,7 @@ static int write_archive_entry(const unsigned char *sha1, const char *base,
 		strbuf_addch(&path, '/');
 	path_without_prefix = path.buf + args->baselen;
 
-	if (!S_ISDIR(mode) || !should_queue_directories(args)) {
+	if (!S_ISDIR(mode)) {
 		const struct attr_check *check;
 		check = get_archive_attrs(path_without_prefix);
 		if (check_attr_export_ignore(check))
@@ -169,14 +164,6 @@ static int write_archive_entry(const unsigned char *sha1, const char *base,
 	return write_entry(args, sha1, path.buf, path.len, mode);
 }
 
-static int write_archive_entry_buf(const unsigned char *sha1, struct strbuf *base,
-		const char *filename, unsigned mode, int stage,
-		void *context)
-{
-	return write_archive_entry(sha1, base->buf, base->len,
-				     filename, mode, stage, context);
-}
-
 static void queue_directory(const unsigned char *sha1,
 		struct strbuf *base, const char *filename,
 		unsigned mode, int stage, struct archiver_context *c)
@@ -290,9 +277,7 @@ int write_archive_entries(struct archiver_args *args,
 	}
 
 	err = read_tree_recursive(args->tree, "", 0, 0, &args->pathspec,
-				  should_queue_directories(args) ?
-				  queue_or_write_archive_entry :
-				  write_archive_entry_buf,
+				  queue_or_write_archive_entry,
 				  &context);
 	if (err == READ_TREE_RECURSIVE)
 		err = 0;
diff --git a/t/t5001-archive-attr.sh b/t/t5001-archive-attr.sh
index 897f6f06d5..e9aa97117a 100755
--- a/t/t5001-archive-attr.sh
+++ b/t/t5001-archive-attr.sh
@@ -73,7 +73,7 @@ test_expect_missing	archive-pathspec/ignored-by-tree
 test_expect_missing	archive-pathspec/ignored-by-tree.d
 test_expect_missing	archive-pathspec/ignored-by-tree.d/file
 test_expect_exists	archive-pathspec/ignored-by-worktree
-test_expect_missing	archive-pathspec/excluded-by-pathspec.d failure
+test_expect_missing	archive-pathspec/excluded-by-pathspec.d
 test_expect_missing	archive-pathspec/excluded-by-pathspec.d/file
 
 test_expect_success 'git archive with wildcard pathspec' '
diff --git a/t/t5002-archive-attr-pattern.sh b/t/t5002-archive-attr-pattern.sh
index 6667d159ab..bda6d7d7e9 100755
--- a/t/t5002-archive-attr-pattern.sh
+++ b/t/t5002-archive-attr-pattern.sh
@@ -76,7 +76,7 @@ test_expect_missing	archive/deep/and/slashless/ &&
 test_expect_missing	archive/deep/and/slashless/foo &&
 test_expect_missing	archive/deep/with/wildcard/ &&
 test_expect_missing	archive/deep/with/wildcard/foo &&
-test_expect_exists	archive/one-level-lower/
+test_expect_missing	archive/one-level-lower/
 test_expect_missing	archive/one-level-lower/two-levels-lower/ignored-only-if-dir/
 test_expect_missing	archive/one-level-lower/two-levels-lower/ignored-ony-if-dir/ignored-by-ignored-dir
 
diff --git a/t/t5004-archive-corner-cases.sh b/t/t5004-archive-corner-cases.sh
index f6207f42b5..ced44355ca 100755
--- a/t/t5004-archive-corner-cases.sh
+++ b/t/t5004-archive-corner-cases.sh
@@ -108,14 +108,14 @@ test_expect_success 'archive empty subtree with no pathspec' '
 	git archive --format=tar $root_tree >subtree-all.tar &&
 	make_dir extract &&
 	"$TAR" xf subtree-all.tar -C extract &&
-	check_dir extract sub
+	check_dir extract
 '
 
 test_expect_success 'archive empty subtree by direct pathspec' '
 	git archive --format=tar $root_tree -- sub >subtree-path.tar &&
 	make_dir extract &&
 	"$TAR" xf subtree-path.tar -C extract &&
-	check_dir extract sub
+	check_dir extract
 '
 
 ZIPINFO=zipinfo
-- 
2.14.1

Re: [PATCH 4/4] archive: queue directories for all types of pathspecs

[Jeff King]

On Wed, Sep 13, 2017 at 12:43:57AM +0200, René Scharfe wrote:

> -- >8 --
> Subject: [PATCH] archive: don't add empty directories to archives
> 
> While git doesn't track empty directories, git archive can be tricked
> into putting some into archives.  One way is to construct an empty tree
> object, as t5004 does.  While that is supported by the object database,
> it can't be represented in the index and thus it's unlikely to occur in
> the wild.
> 
> Another way is using the literal name of a directory in an exclude
> pathspec -- its contents are are excluded, but the directory stub is
> included.  That's inconsistent: exclude pathspecs containing wildcards
> don't leave empty directories in the archive.
> 
> Yet another way is have a few levels of nested subdirectories (e.g.
> d1/d2/d3/file1) and ignoring the entries at the leaved (e.g. file1).

s/leaved/leaves/ ?

> The directories with the ignored content are ignored as well (e.g. d3),
> but their empty parents are included (e.g. d2).
> 
> As empty directories are not supported by git, they should also not be
> written into archives.  If an empty directory is really needed then it
> can be tracked and archived by placing an empty .gitignore file in it.
> 
> There already is a mechanism in place for suppressing empty directories.
> When read_tree_recursive() encounters a directory excluded by a pathspec
> then it enters it anyway because it might contain included entries.  It
> calls the callback function before it is able to decide if the directory
> is actually needed.  For that reason git archive adds directories to a
> queue and writes entries for them only when it encounters the first
> child item -- but currently only if pathspecs with wildcards are used.
> 
> Queue *all* directories, no matter if there even are pathspecs present.
> This prevents git archive from writing entries for empty directories in
> all cases.

Nicely explained, and this seems like the right level to be handling it.
Simple, and it will catch the cases we know about _and_ and any new ones
which pop up.

> ---
>  archive.c                       | 19 ++-----------------
>  t/t5001-archive-attr.sh         |  2 +-
>  t/t5002-archive-attr-pattern.sh |  2 +-
>  t/t5004-archive-corner-cases.sh |  4 ++--
>  4 files changed, 6 insertions(+), 21 deletions(-)

I'm not too familiar with this part of the archive code, but it seemed
pretty easy to follow. The patch looks good to me.

-Peff

Re: [PATCH 4/4] archive: queue directories for all types of pathspecs

[René Scharfe]

Am 13.09.2017 um 14:53 schrieb Jeff King:
> On Wed, Sep 13, 2017 at 12:43:57AM +0200, René Scharfe wrote:
>> Yet another way is have a few levels of nested subdirectories (e.g.
>> d1/d2/d3/file1) and ignoring the entries at the leaved (e.g. file1).
> 
> s/leaved/leaves/ ?

Indeed.

René

Re: [PATCH 4/4] archive: queue directories for all types of pathspecs

[Stefan Beller]

On Sat, Aug 19, 2017 at 9:53 AM, René Scharfe <l.s.r@web.de> wrote:
> Am 19.08.2017 um 07:33 schrieb René Scharfe:
>> When read_tree_recursive() encounters a directory excluded by a pathspec
>> then it enters it anyway because it might contain included entries.  It
>> calls the callback function before it is able to decide if the directory
>> is actually needed.
>>
>> For that reason git archive adds directories to a queue and writes
>> entries for them only when it encounters the first child item -- but
>> only if pathspecs with wildcards are used.  Do the same for literal
>> pathspecs as well, as the reasoning above applies to them, too.  This
>> prevents git archive from writing entries for excluded directories.
>
> This breaks the test "archive empty subtree with no pathspec" in t5004 by
> omitting the empty directory from the archive.  Sorry for missing that!
>
> This is kind of a bonus patch, so please discard it for now; the first
> three are OK IMHO.

I thought so up to reading this patch. I found the naming and
content of should_queue_directories not obvious at first read.
This patch confused me even more for that name to be chosen.
I have no alternative to propose, though.

Thanks,
Stefan

Re: [PATCH 4/4] archive: queue directories for all types of pathspecs

[Junio C Hamano]

René Scharfe <l.s.r@web.de> writes:

> When read_tree_recursive() encounters a directory excluded by a pathspec
> then it enters it anyway because it might contain included entries.  It
> calls the callback function before it is able to decide if the directory
> is actually needed.
>
> For that reason git archive adds directories to a queue and writes
> entries for them only when it encounters the first child item -- but
> only if pathspecs with wildcards are used.  Do the same for literal
> pathspecs as well, as the reasoning above applies to them, too.  This
> prevents git archive from writing entries for excluded directories.
>
> Signed-off-by: Rene Scharfe <l.s.r@web.de>
> ---
>  archive.c               | 2 +-
>  t/t5001-archive-attr.sh | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)

This seems to break t/t5004-archive-corner-cases.sh though...

    expecting success:
            git archive --format=tar $root_tree -- sub >subtree-path.tar &&
            make_dir extract &&
            "$TAR" xf subtree-path.tar -C extract &&
            check_dir extract sub

    --- expect      2017-08-19 16:56:49.761513537 +0000
    +++ actual      2017-08-19 16:56:49.769513535 +0000
    @@ -1,2 +1 @@
     extract
    -extract/sub
    not ok 10 - archive empty subtree by direct pathspec

> diff --git a/archive.c b/archive.c
> index 1ab8d3a1d7..174c0555b6 100644
> --- a/archive.c
> +++ b/archive.c
> @@ -123,7 +123,7 @@ static int check_attr_export_subst(const struct attr_check *check)
>  
>  static int should_queue_directories(const struct archiver_args *args)
>  {
> -	return args->pathspec.has_wildcard;
> +	return args->pathspec.nr;
>  }
>  
>  static int write_archive_entry(const unsigned char *sha1, const char *base,
> diff --git a/t/t5001-archive-attr.sh b/t/t5001-archive-attr.sh
> index 897f6f06d5..e9aa97117a 100755
> --- a/t/t5001-archive-attr.sh
> +++ b/t/t5001-archive-attr.sh
> @@ -73,7 +73,7 @@ test_expect_missing	archive-pathspec/ignored-by-tree
>  test_expect_missing	archive-pathspec/ignored-by-tree.d
>  test_expect_missing	archive-pathspec/ignored-by-tree.d/file
>  test_expect_exists	archive-pathspec/ignored-by-worktree
> -test_expect_missing	archive-pathspec/excluded-by-pathspec.d failure
> +test_expect_missing	archive-pathspec/excluded-by-pathspec.d
>  test_expect_missing	archive-pathspec/excluded-by-pathspec.d/file
>  
>  test_expect_success 'git archive with wildcard pathspec' '