Commit ID in exported Tar Ball

Commit ID in exported Tar Ball

[Thomas Glanzmann]

Hello,
Michae, a friend of mine, is in the phase of migrating from git to cvs.
He releases tar balls of his software project using gitweb. He would
love to have a way to have the commit-id of HEAD of the export contained
in one of the files he exported that way. Is there infrastructure in git
that makes that already possible or does he need to some kind of
gerneration tool by himself? Maybe it would be helpful if the
git-tar-tree would generate a file .commitid or something like that in
the generated tar tree.

        Thomas

Re: Commit ID in exported Tar Ball

[Johan Herland]

On Thursday 17 May 2007, Thomas Glanzmann wrote:
> Hello,
> Michae, a friend of mine, is in the phase of migrating from git to cvs.
                                                         ^^^^^^^^^^^^^^^
Man... You are _definitely_ on the wrong list. ;)

> He releases tar balls of his software project using gitweb. He would
> love to have a way to have the commit-id of HEAD of the export contained
> in one of the files he exported that way. Is there infrastructure in git
> that makes that already possible or does he need to some kind of
> gerneration tool by himself? Maybe it would be helpful if the
> git-tar-tree would generate a file .commitid or something like that in
> the generated tar tree.

Hmm, doesn't seem like git-tar-tree (or git-archive for that matter) 
supports this out of the box. Maybe it's possible to achieve in combination 
with the $Id$ construct?

I guess it depends on whether git-tar-tree/git-archive actually does a 
checkout from the repo from which the archive is made. If so, it should 
be possible to store "$Id$" in .commitid, and check it in, and it should 
automagically appear with the correct commit-id in your archive.

Of course, it all depends on whether the $Id$ conversion is triggered by 
git-archive...


Have fun!

...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: Commit ID in exported Tar Ball

[Kristian Høgsberg]

On 5/17/07, Thomas Glanzmann <thomas@glanzmann.de> wrote:
> Hello,
> Michae, a friend of mine, is in the phase of migrating from git to cvs.
> He releases tar balls of his software project using gitweb. He would
> love to have a way to have the commit-id of HEAD of the export contained
> in one of the files he exported that way. Is there infrastructure in git
> that makes that already possible or does he need to some kind of
> gerneration tool by himself? Maybe it would be helpful if the
> git-tar-tree would generate a file .commitid or something like that in
> the generated tar tree.

Use git-get-tar-commit-id:

  $ gzip -cd snapshot.tar.gz | git-get-tar-commit-id

cheers,
Kristian

Re: Commit ID in exported Tar Ball

[Frank Lichtenheld]

On Thu, May 17, 2007 at 06:57:22PM +0200, Johan Herland wrote:
> Of course, it all depends on whether the $Id$ conversion is triggered by 
> git-archive...

Another possibility might be to add a commandline switch to git-archive
so you can decide whether the commit id should be added as a header to
the tar file (which it already supports) or as a ordinary file (which
should be reasonable trivial to implement). The question if whether
it would be worth to add that feature. Don't know if there are many
other users out there that need it.

Gruesse,
-- 
Frank Lichtenheld <frank@lichtenheld.de>
www: http://www.djpig.de/

Re: Commit ID in exported Tar Ball

[Thomas Glanzmann]

Hello,

>  $ gzip -cd snapshot.tar.gz | git-get-tar-commit-id

not good enough. What he wants is:

        - user loads tar tree down from gitweb via the 'snapshot'
          button.

        - user extracts the tarball and types make

        - The output of the produced binary contains a unique
          identifier.

        Thomas

Re: Commit ID in exported Tar Ball

[Thomas Glanzmann]

Hello,

> Another possibility might be to add a commandline switch to
> git-archive so you can decide whether the commit id should be added as
> a header to the tar file (which it already supports) or as a ordinary
> file (which should be reasonable trivial to implement). The question
> if whether it would be worth to add that feature. Don't know if there
> are many other users out there that need it.

that would be very good aproach I guess. At least from my point of view.
Because it doesn't break diffs, it concentrates on what matters and if
you don't like it you don't use it.

        Thomas

Re: Commit ID in exported Tar Ball

[Johan Herland]

On Thursday 17 May 2007, Frank Lichtenheld wrote:
> On Thu, May 17, 2007 at 06:57:22PM +0200, Johan Herland wrote:
> > Of course, it all depends on whether the $Id$ conversion is triggered by 
> > git-archive...
> 
> Another possibility might be to add a commandline switch to git-archive
> so you can decide whether the commit id should be added as a header to
> the tar file (which it already supports) or as a ordinary file (which
> should be reasonable trivial to implement). The question if whether
> it would be worth to add that feature. Don't know if there are many
> other users out there that need it.

Although this efficiently solves Michael's problem, I still think the ideal 
solution would be for git-archive to do the same conversions/filters as a 
regular checkout would. Otherwise, we'll easily get into situations where 
a git-archive tree is different enough from a "regular" working tree to 
cause annoying differences in behaviour.


Have fun!

...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: Commit ID in exported Tar Ball

[Frank Lichtenheld]

On Thu, May 17, 2007 at 06:57:22PM +0200, Johan Herland wrote:
> Hmm, doesn't seem like git-tar-tree (or git-archive for that matter) 
> supports this out of the box. Maybe it's possible to achieve in combination 
> with the $Id$ construct?

$Id$ contains the blob id, not the commit id.

Gruesse,
-- 
Frank Lichtenheld <frank@lichtenheld.de>
www: http://www.djpig.de/

Re: Commit ID in exported Tar Ball

[Johan Herland]

On Thursday 17 May 2007, Frank Lichtenheld wrote:
> On Thu, May 17, 2007 at 06:57:22PM +0200, Johan Herland wrote:
> > Hmm, doesn't seem like git-tar-tree (or git-archive for that matter) 
> > supports this out of the box. Maybe it's possible to achieve in combination 
> > with the $Id$ construct?
> 
> $Id$ contains the blob id, not the commit id.

Oops. My bad. In that case, your solution (to add another commandline switch 
to git-archive) is definitely what Michael needs.


...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

[PATCH] git-archive: convert archive entries like checkouts do

[René Scharfe]

As noted by Johan Herland, git-archive is a kind of checkout and needs
to apply any checkout filters that might be configured.

This patch adds the convenience function convert_sha1_file which returns
a buffer containing the object's contents, after converting, if necessary
(i.e. it's a combination of read_sha1_file and convert_to_working_tree).
Direct calls to read_sha1_file in git-archive are then replaced by calls
to convert_sha1_file.

Since convert_sha1_file expects its path argument to be NUL-terminated --
a convention it inherits from convert_to_working_tree -- the patch also
changes the path handling in archive-tar.c to always NUL-terminate the
string.  It used to solely rely on the len field of struct strbuf before.

archive-zip.c already NUL-terminates the path and thus needs no such
change.

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>

---
 archive-tar.c |   12 +++++++-----
 archive-zip.c |    2 +-
 cache.h       |    1 +
 convert.c     |   15 +++++++++++++++
 4 files changed, 24 insertions(+), 6 deletions(-)

diff --git a/archive-tar.c b/archive-tar.c
index 56ff356..eb0abc7 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -83,11 +83,12 @@ static void strbuf_append_string(struct strbuf *sb, const char *s)
 	int slen = strlen(s);
 	int total = sb->len + slen;
 	if (total > sb->alloc) {
-		sb->buf = xrealloc(sb->buf, total);
-		sb->alloc = total;
+		sb->buf = xrealloc(sb->buf, total + 1);
+		sb->alloc = total + 1;
 	}
 	memcpy(sb->buf + sb->len, s, slen);
 	sb->len = total;
+	sb->buf[total] = '\0';
 }
 
 /*
@@ -272,18 +273,19 @@ static int write_tar_entry(const unsigned char *sha1,
 	}
 	if (path.alloc < baselen + filenamelen) {
 		free(path.buf);
-		path.buf = xmalloc(baselen + filenamelen);
-		path.alloc = baselen + filenamelen;
+		path.buf = xmalloc(baselen + filenamelen + 1);
+		path.alloc = baselen + filenamelen + 1;
 	}
 	memcpy(path.buf, base, baselen);
 	memcpy(path.buf + baselen, filename, filenamelen);
 	path.len = baselen + filenamelen;
+	path.buf[path.len] = '\0';
 	if (S_ISDIR(mode) || S_ISDIRLNK(mode)) {
 		strbuf_append_string(&path, "/");
 		buffer = NULL;
 		size = 0;
 	} else {
-		buffer = read_sha1_file(sha1, &type, &size);
+		buffer = convert_sha1_file(path.buf, sha1, mode, &type, &size);
 		if (!buffer)
 			die("cannot read %s", sha1_to_hex(sha1));
 	}
diff --git a/archive-zip.c b/archive-zip.c
index 1eaf262..3cbf6bb 100644
--- a/archive-zip.c
+++ b/archive-zip.c
@@ -195,7 +195,7 @@ static int write_zip_entry(const unsigned char *sha1,
 		if (S_ISREG(mode) && zlib_compression_level != 0)
 			method = 8;
 		result = 0;
-		buffer = read_sha1_file(sha1, &type, &size);
+		buffer = convert_sha1_file(path, sha1, mode, &type, &size);
 		if (!buffer)
 			die("cannot read %s", sha1_to_hex(sha1));
 		crc = crc32(crc, buffer, size);
diff --git a/cache.h b/cache.h
index aaeb04a..4204bc1 100644
--- a/cache.h
+++ b/cache.h
@@ -548,6 +548,7 @@ extern void trace_argv_printf(const char **argv, int count, const char *format,
 /* convert.c */
 extern char *convert_to_git(const char *path, const char *src, unsigned long *sizep);
 extern char *convert_to_working_tree(const char *path, const char *src, unsigned long *sizep);
+extern void *convert_sha1_file(const char *path, const unsigned char *sha1, unsigned int mode, enum object_type *type, unsigned long *size);
 
 /* match-trees.c */
 void shift_tree(const unsigned char *, const unsigned char *, unsigned char *, int);
diff --git a/convert.c b/convert.c
index 12abdaf..c64880b 100644
--- a/convert.c
+++ b/convert.c
@@ -652,3 +652,18 @@ char *convert_to_working_tree(const char *path, const char *src, unsigned long *
 
 	return buf;
 }
+
+void *convert_sha1_file(const char *path, const unsigned char *sha1,
+                        unsigned int mode, enum object_type *type,
+                        unsigned long *size)
+{
+	void *buffer = read_sha1_file(sha1, type, size);
+	if (S_ISREG(mode) && buffer) {
+		void *converted = convert_to_working_tree(path, buffer, size);
+		if (converted) {
+			free(buffer);
+			buffer = converted;
+		}
+	}
+	return buffer;
+}

Re: [PATCH] git-archive: convert archive entries like checkouts do

[Daniel Barkalow]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1051 bytes --]

On Sat, 19 May 2007, René Scharfe wrote:

> diff --git a/archive-tar.c b/archive-tar.c
> index 56ff356..eb0abc7 100644
> --- a/archive-tar.c
> +++ b/archive-tar.c
> @@ -83,11 +83,12 @@ static void strbuf_append_string(struct strbuf *sb, const char *s)
>  	int slen = strlen(s);
>  	int total = sb->len + slen;
>  	if (total > sb->alloc) {
> -		sb->buf = xrealloc(sb->buf, total);
> -		sb->alloc = total;
> +		sb->buf = xrealloc(sb->buf, total + 1);
> +		sb->alloc = total + 1;

Conditional needs a "+ 1", too.

>  	}
>  	memcpy(sb->buf + sb->len, s, slen);
>  	sb->len = total;
> +	sb->buf[total] = '\0';
>  }
>  
>  /*
> @@ -272,18 +273,19 @@ static int write_tar_entry(const unsigned char *sha1,
>  	}
>  	if (path.alloc < baselen + filenamelen) {
>  		free(path.buf);
> -		path.buf = xmalloc(baselen + filenamelen);
> -		path.alloc = baselen + filenamelen;
> +		path.buf = xmalloc(baselen + filenamelen + 1);
> +		path.alloc = baselen + filenamelen + 1;

Same here.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH] git-archive: convert archive entries like checkouts do

[René Scharfe]

Daniel Barkalow schrieb:
> Conditional needs a "+ 1", too.
[...]
> Same here.

Thank you for spotting this.  Fix-up patch below.

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>

---
Embarrassing.  I'm off to go to sleep now.

 archive-tar.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/archive-tar.c b/archive-tar.c
index eb0abc7..33e7657 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -82,7 +82,7 @@ static void strbuf_append_string(struct strbuf *sb, const char *s)
 {
 	int slen = strlen(s);
 	int total = sb->len + slen;
-	if (total > sb->alloc) {
+	if (total + 1 > sb->alloc) {
 		sb->buf = xrealloc(sb->buf, total + 1);
 		sb->alloc = total + 1;
 	}
@@ -271,7 +271,7 @@ static int write_tar_entry(const unsigned char *sha1,
 		path.alloc = PATH_MAX;
 		path.len = path.eof = 0;
 	}
-	if (path.alloc < baselen + filenamelen) {
+	if (path.alloc < baselen + filenamelen + 1) {
 		free(path.buf);
 		path.buf = xmalloc(baselen + filenamelen + 1);
 		path.alloc = baselen + filenamelen + 1;

Re: Commit ID in exported Tar Ball

[René Scharfe]

Frank Lichtenheld schrieb:
> On Thu, May 17, 2007 at 06:57:22PM +0200, Johan Herland wrote:
>> Of course, it all depends on whether the $Id$ conversion is triggered by 
>> git-archive...
> 
> Another possibility might be to add a commandline switch to git-archive
> so you can decide whether the commit id should be added as a header to
> the tar file (which it already supports) or as a ordinary file (which
> should be reasonable trivial to implement). The question if whether
> it would be worth to add that feature. Don't know if there are many
> other users out there that need it.

Something like the following patch?  Since we're already embedding the
commit ID in a comment, we might as well offer creating a synthetic file
for it, too, if that solves a user's problem that might be difficult to
work around otherwise.

René


 Documentation/git-archive.txt |    4 ++++
 archive-tar.c                 |    7 +++++++
 archive-zip.c                 |    7 +++++++
 archive.h                     |    1 +
 builtin-archive.c             |   11 +++++++++++
 5 files changed, 30 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt
index 721e035..ac31aac 100644
--- a/Documentation/git-archive.txt
+++ b/Documentation/git-archive.txt
@@ -43,6 +43,10 @@ OPTIONS
 --prefix=<prefix>/::
 	Prepend <prefix>/ to each filename in the archive.
 
+--commit-id-file=<filename>::
+	Adds a file to the archive containing the commit ID.  This option
+	is ignored if <tree-ish> references a tree instead of a commit.
+
 <extra>::
 	This can be any options that the archiver backend understand.
 	See next section.
diff --git a/archive-tar.c b/archive-tar.c
index 33e7657..555850a 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -319,6 +319,13 @@ int write_tar_archive(struct archiver_args *args)
 	}
 	read_tree_recursive(args->tree, args->base, plen, 0,
 			    args->pathspec, write_tar_entry);
+	if (args->commit_sha1 && args->commit_sha1_file) {
+		unsigned char fake_sha1[20];
+		pretend_sha1_file(sha1_to_hex(args->commit_sha1), 40,
+		                  OBJ_BLOB, fake_sha1);
+		write_tar_entry(fake_sha1, args->base, plen,
+		                args->commit_sha1_file, 0100666, 0);
+	}
 	write_trailer();
 
 	return 0;
diff --git a/archive-zip.c b/archive-zip.c
index 3cbf6bb..88c5dfa 100644
--- a/archive-zip.c
+++ b/archive-zip.c
@@ -328,6 +328,13 @@ int write_zip_archive(struct archiver_args *args)
 	}
 	read_tree_recursive(args->tree, args->base, plen, 0,
 			    args->pathspec, write_zip_entry);
+	if (args->commit_sha1 && args->commit_sha1_file) {
+		unsigned char fake_sha1[20];
+		pretend_sha1_file(sha1_to_hex(args->commit_sha1), 40,
+		                  OBJ_BLOB, fake_sha1);
+		write_zip_entry(fake_sha1, args->base, plen,
+		                args->commit_sha1_file, 0100666, 0);
+	}
 	write_zip_trailer(args->commit_sha1);
 
 	free(zip_dir);
diff --git a/archive.h b/archive.h
index 6838dc7..020f82f 100644
--- a/archive.h
+++ b/archive.h
@@ -8,6 +8,7 @@ struct archiver_args {
 	const char *base;
 	struct tree *tree;
 	const unsigned char *commit_sha1;
+	const char *commit_sha1_file;
 	time_t time;
 	const char **pathspec;
 	unsigned int verbose : 1;
diff --git a/builtin-archive.c b/builtin-archive.c
index 7f4e409..e58ea16 100644
--- a/builtin-archive.c
+++ b/builtin-archive.c
@@ -151,6 +151,7 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 	int extra_argc = 0;
 	const char *format = "tar";
 	const char *base = "";
+	const char *commit_sha1_file = NULL;
 	int verbose = 0;
 	int i;
 
@@ -174,6 +175,10 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 			base = arg + 9;
 			continue;
 		}
+		if (!prefixcmp(arg, "--commit-id-file=")) {
+			commit_sha1_file = arg + 17;
+			continue;
+		}
 		if (!strcmp(arg, "--")) {
 			i++;
 			break;
@@ -192,6 +197,11 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 		usage(archive_usage);
 	if (init_archiver(format, ar) < 0)
 		die("Unknown archive format '%s'", format);
+	if (commit_sha1_file) {
+		size_t namelen = strlen(commit_sha1_file);
+		if (namelen == 0 || commit_sha1_file[namelen - 1] == '/')
+			die("Invalid commit ID file name: %s", commit_sha1_file);
+	}
 
 	if (extra_argc) {
 		if (!ar->parse_extra)
@@ -201,6 +211,7 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 	}
 	ar->args.verbose = verbose;
 	ar->args.base = base;
+	ar->args.commit_sha1_file = commit_sha1_file;
 
 	return i;
 }

Re: Commit ID in exported Tar Ball

[Junio C Hamano]

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:

> Something like the following patch?  Since we're already embedding the
> commit ID in a comment, we might as well offer creating a synthetic file
> for it, too, if that solves a user's problem that might be difficult to
> work around otherwise.

Are you detecting and erroring out if the named file already
exists in the tree being archived?

Should we?  Maybe we should just replace with warning?

Also should we silently ignore the request if the tree-ish is
not a commit-ish, or error out?

Re: Commit ID in exported Tar Ball

[A Large Angry SCM]

Junio C Hamano wrote:
> René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
> 
>> Something like the following patch?  Since we're already embedding the
>> commit ID in a comment, we might as well offer creating a synthetic file
>> for it, too, if that solves a user's problem that might be difficult to
>> work around otherwise.
> 
> Are you detecting and erroring out if the named file already
> exists in the tree being archived?
> 
> Should we?  Maybe we should just replace with warning?
> 
> Also should we silently ignore the request if the tree-ish is
> not a commit-ish, or error out?

Silently ignoring conflicting command arguments is (almost always) a 
bug; either in the implementation or the design.

Re: Commit ID in exported Tar Ball

[René Scharfe]

Junio C Hamano schrieb:
> René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
> 
>> Something like the following patch?  Since we're already embedding
>> the commit ID in a comment, we might as well offer creating a
>> synthetic file for it, too, if that solves a user's problem that
>> might be difficult to work around otherwise.
> 
> Are you detecting and erroring out if the named file already exists
> in the tree being archived?
> 
> Should we?  Maybe we should just replace with warning?

Currently the commit ID file is appended to the archive, so if there is
a name conflict, we keep both files.  unzip offers a choice between
renaming, overwriting and keeping the first extracted version when the
second one is about to be extracted.  tar has a -k option: with -k you
get the first version, without -k you get the second one.

To avoid confusion, we should disallow a name that already comes from
the tree.  Only I can't see an easy way to implement this.  Perhaps a
check using get_tree_entry is enough -- it doesn't take pathspecs into
account, though.  That means we would disallow all names in the tree,
even if a pathspec excludes the chosen commit ID file name.

Before I roll my own path existence checker with pathspec support, is
there something like that already implemented?  I suspect it's hiding in
the diff code, but I don't dare go near it. ;-)

> Also should we silently ignore the request if the tree-ish is not a
> commit-ish, or error out?

As Angry said, erroring out is better.  It's also easy to do.

 Documentation/git-archive.txt |    4 ++++
 archive-tar.c                 |    7 +++++++
 archive-zip.c                 |    7 +++++++
 archive.h                     |    1 +
 builtin-archive.c             |   32 ++++++++++++++++++++++++++++++++
 5 files changed, 51 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt
index 721e035..7016d1e 100644
--- a/Documentation/git-archive.txt
+++ b/Documentation/git-archive.txt
@@ -43,6 +43,10 @@ OPTIONS
 --prefix=<prefix>/::
 	Prepend <prefix>/ to each filename in the archive.
 
+--commit-id-file=<filename>::
+	Adds a file to the archive containing the commit ID.  This option
+	is can only be used if <tree-ish> references a commit or tag.
+
 <extra>::
 	This can be any options that the archiver backend understand.
 	See next section.
diff --git a/archive-tar.c b/archive-tar.c
index 33e7657..555850a 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -319,6 +319,13 @@ int write_tar_archive(struct archiver_args *args)
 	}
 	read_tree_recursive(args->tree, args->base, plen, 0,
 			    args->pathspec, write_tar_entry);
+	if (args->commit_sha1 && args->commit_sha1_file) {
+		unsigned char fake_sha1[20];
+		pretend_sha1_file(sha1_to_hex(args->commit_sha1), 40,
+		                  OBJ_BLOB, fake_sha1);
+		write_tar_entry(fake_sha1, args->base, plen,
+		                args->commit_sha1_file, 0100666, 0);
+	}
 	write_trailer();
 
 	return 0;
diff --git a/archive-zip.c b/archive-zip.c
index 3cbf6bb..88c5dfa 100644
--- a/archive-zip.c
+++ b/archive-zip.c
@@ -328,6 +328,13 @@ int write_zip_archive(struct archiver_args *args)
 	}
 	read_tree_recursive(args->tree, args->base, plen, 0,
 			    args->pathspec, write_zip_entry);
+	if (args->commit_sha1 && args->commit_sha1_file) {
+		unsigned char fake_sha1[20];
+		pretend_sha1_file(sha1_to_hex(args->commit_sha1), 40,
+		                  OBJ_BLOB, fake_sha1);
+		write_zip_entry(fake_sha1, args->base, plen,
+		                args->commit_sha1_file, 0100666, 0);
+	}
 	write_zip_trailer(args->commit_sha1);
 
 	free(zip_dir);
diff --git a/archive.h b/archive.h
index 6838dc7..020f82f 100644
--- a/archive.h
+++ b/archive.h
@@ -8,6 +8,7 @@ struct archiver_args {
 	const char *base;
 	struct tree *tree;
 	const unsigned char *commit_sha1;
+	const char *commit_sha1_file;
 	time_t time;
 	const char **pathspec;
 	unsigned int verbose : 1;
diff --git a/builtin-archive.c b/builtin-archive.c
index 7f4e409..6bb0781 100644
--- a/builtin-archive.c
+++ b/builtin-archive.c
@@ -151,6 +151,7 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 	int extra_argc = 0;
 	const char *format = "tar";
 	const char *base = "";
+	const char *commit_sha1_file = NULL;
 	int verbose = 0;
 	int i;
 
@@ -174,6 +175,10 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 			base = arg + 9;
 			continue;
 		}
+		if (!prefixcmp(arg, "--commit-id-file=")) {
+			commit_sha1_file = arg + 17;
+			continue;
+		}
 		if (!strcmp(arg, "--")) {
 			i++;
 			break;
@@ -192,6 +197,11 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 		usage(archive_usage);
 	if (init_archiver(format, ar) < 0)
 		die("Unknown archive format '%s'", format);
+	if (commit_sha1_file) {
+		size_t namelen = strlen(commit_sha1_file);
+		if (namelen == 0 || commit_sha1_file[namelen - 1] == '/')
+			die("Invalid commit ID file name: %s", commit_sha1_file);
+	}
 
 	if (extra_argc) {
 		if (!ar->parse_extra)
@@ -201,6 +211,7 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 	}
 	ar->args.verbose = verbose;
 	ar->args.base = base;
+	ar->args.commit_sha1_file = commit_sha1_file;
 
 	return i;
 }
@@ -236,6 +247,20 @@ static const char *extract_remote_arg(int *ac, const char **av)
 	return remote;
 }
 
+static int is_path_in_spec(struct tree *tree, const char **pathspec,
+                           const char *path)
+{
+	unsigned char sha1[20];
+	unsigned int mode;
+
+	if (get_tree_entry(tree->object.sha1, path, sha1, &mode))
+		return 0;
+	if (!pathspec)
+		return 1;
+	/* TODO: the actual pathspec check */
+	return 1;
+}
+
 int cmd_archive(int argc, const char **argv, const char *prefix)
 {
 	struct archiver ar;
@@ -257,5 +282,12 @@ int cmd_archive(int argc, const char **argv, const char *prefix)
 	parse_treeish_arg(argv, &ar.args, prefix);
 	parse_pathspec_arg(argv + 1, &ar.args);
 
+	if (ar.args.commit_sha1_file) {
+		if (is_path_in_spec(ar.args.tree, ar.args.pathspec, ar.args.commit_sha1_file))
+			die("Commit ID file name already exists in archive.");
+		if (!ar.args.commit_sha1)
+			die("Need a commit to use --commit-id-file, and not a tree.");
+	}
+
 	return ar.write_archive(&ar.args);
 }

Re: Commit ID in exported Tar Ball

[Shawn O. Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
> 
> > Something like the following patch?  Since we're already embedding the
> > commit ID in a comment, we might as well offer creating a synthetic file
> > for it, too, if that solves a user's problem that might be difficult to
> > work around otherwise.

What about being able to get the output of git-describe embedded
into an archive file?  Doesn't git.git do that in its Makefile?  ;-)

git-describe is more human-friendly than a SHA-1...

-- 
Shawn.

Re: Commit ID in exported Tar Ball

[René Scharfe]

Shawn O. Pearce schrieb:
> Junio C Hamano <junkio@cox.net> wrote:
>> René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
>>
>>> Something like the following patch?  Since we're already embedding the
>>> commit ID in a comment, we might as well offer creating a synthetic file
>>> for it, too, if that solves a user's problem that might be difficult to
>>> work around otherwise.
> 
> What about being able to get the output of git-describe embedded
> into an archive file?  Doesn't git.git do that in its Makefile?  ;-)
> 
> git-describe is more human-friendly than a SHA-1...

Yes, and the Makefile does even more than that: it adds a version file,
a spec file and another version file for git-gui.

The first two are probably useful for most projects that actually do
versioned releases.  We could have a simple parser that reads a
template, replaces @@VERSION@@ with a git-describe output string and
adds the result as a synthetic file to the archive.  It's not exactly
trivial -- e.g., how to specify git-describe options, template file and
synthetic name, all in one command line parameter? -- but it's doable.

I'm not sure how the git-gui version file fits in.  I guess it's just a
special case and doesn't need git-archive support?

René

Re: Commit ID in exported Tar Ball

[René Scharfe]

Turns out a matcher for the kind of pathspecs used in git-archive
is much easier to write than I thought. :-)

This is just a progress note and not for inclusion, yet -- Shawn has
a good point suggesting git-describe output to be used instead of
bare commit IDs.

 Documentation/git-archive.txt |    4 +++
 archive-tar.c                 |    7 ++++++
 archive-zip.c                 |    7 ++++++
 archive.h                     |    1 +
 builtin-archive.c             |   44 +++++++++++++++++++++++++++++++++++++++++
 5 files changed, 63 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt
index 721e035..7016d1e 100644
--- a/Documentation/git-archive.txt
+++ b/Documentation/git-archive.txt
@@ -43,6 +43,10 @@ OPTIONS
 --prefix=<prefix>/::
 	Prepend <prefix>/ to each filename in the archive.
 
+--commit-id-file=<filename>::
+	Adds a file to the archive containing the commit ID.  This option
+	is can only be used if <tree-ish> references a commit or tag.
+
 <extra>::
 	This can be any options that the archiver backend understand.
 	See next section.
diff --git a/archive-tar.c b/archive-tar.c
index 33e7657..555850a 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -319,6 +319,13 @@ int write_tar_archive(struct archiver_args *args)
 	}
 	read_tree_recursive(args->tree, args->base, plen, 0,
 			    args->pathspec, write_tar_entry);
+	if (args->commit_sha1 && args->commit_sha1_file) {
+		unsigned char fake_sha1[20];
+		pretend_sha1_file(sha1_to_hex(args->commit_sha1), 40,
+		                  OBJ_BLOB, fake_sha1);
+		write_tar_entry(fake_sha1, args->base, plen,
+		                args->commit_sha1_file, 0100666, 0);
+	}
 	write_trailer();
 
 	return 0;
diff --git a/archive-zip.c b/archive-zip.c
index 3cbf6bb..88c5dfa 100644
--- a/archive-zip.c
+++ b/archive-zip.c
@@ -328,6 +328,13 @@ int write_zip_archive(struct archiver_args *args)
 	}
 	read_tree_recursive(args->tree, args->base, plen, 0,
 			    args->pathspec, write_zip_entry);
+	if (args->commit_sha1 && args->commit_sha1_file) {
+		unsigned char fake_sha1[20];
+		pretend_sha1_file(sha1_to_hex(args->commit_sha1), 40,
+		                  OBJ_BLOB, fake_sha1);
+		write_zip_entry(fake_sha1, args->base, plen,
+		                args->commit_sha1_file, 0100666, 0);
+	}
 	write_zip_trailer(args->commit_sha1);
 
 	free(zip_dir);
diff --git a/archive.h b/archive.h
index 6838dc7..020f82f 100644
--- a/archive.h
+++ b/archive.h
@@ -8,6 +8,7 @@ struct archiver_args {
 	const char *base;
 	struct tree *tree;
 	const unsigned char *commit_sha1;
+	const char *commit_sha1_file;
 	time_t time;
 	const char **pathspec;
 	unsigned int verbose : 1;
diff --git a/builtin-archive.c b/builtin-archive.c
index 7f4e409..1fe4d47 100644
--- a/builtin-archive.c
+++ b/builtin-archive.c
@@ -151,6 +151,7 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 	int extra_argc = 0;
 	const char *format = "tar";
 	const char *base = "";
+	const char *commit_sha1_file = NULL;
 	int verbose = 0;
 	int i;
 
@@ -174,6 +175,10 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 			base = arg + 9;
 			continue;
 		}
+		if (!prefixcmp(arg, "--commit-id-file=")) {
+			commit_sha1_file = arg + 17;
+			continue;
+		}
 		if (!strcmp(arg, "--")) {
 			i++;
 			break;
@@ -192,6 +197,11 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 		usage(archive_usage);
 	if (init_archiver(format, ar) < 0)
 		die("Unknown archive format '%s'", format);
+	if (commit_sha1_file) {
+		size_t namelen = strlen(commit_sha1_file);
+		if (namelen == 0 || commit_sha1_file[namelen - 1] == '/')
+			die("Invalid commit ID file name: %s", commit_sha1_file);
+	}
 
 	if (extra_argc) {
 		if (!ar->parse_extra)
@@ -201,6 +211,7 @@ int parse_archive_args(int argc, const char **argv, struct archiver *ar)
 	}
 	ar->args.verbose = verbose;
 	ar->args.base = base;
+	ar->args.commit_sha1_file = commit_sha1_file;
 
 	return i;
 }
@@ -236,6 +247,32 @@ static const char *extract_remote_arg(int *ac, const char **av)
 	return remote;
 }
 
+static int is_path_in_spec(const struct archiver_args *args, const char *path)
+{
+	unsigned char sha1[20];
+	unsigned int mode;
+	const char *match;
+	const char **pathspec = args->pathspec;
+
+	if (get_tree_entry(args->tree->object.sha1, path, sha1, &mode))
+		return 0;
+	if (!pathspec)
+		return 1;
+	while ((match = *pathspec++) != NULL) {
+		size_t matchlen = strlen(match);
+		if (matchlen == 0)
+			return 1;
+		if (match[matchlen - 1] == '/') {
+			if (!prefixcmp(path, match))
+				return 1;
+		} else {
+			if (!strcmp(path, match))
+				return 1;
+		}
+	}
+	return 0;
+}
+
 int cmd_archive(int argc, const char **argv, const char *prefix)
 {
 	struct archiver ar;
@@ -257,5 +294,12 @@ int cmd_archive(int argc, const char **argv, const char *prefix)
 	parse_treeish_arg(argv, &ar.args, prefix);
 	parse_pathspec_arg(argv + 1, &ar.args);
 
+	if (ar.args.commit_sha1_file) {
+		if (is_path_in_spec(&ar.args, ar.args.commit_sha1_file))
+			die("Commit ID file name already exists in archive.");
+		if (!ar.args.commit_sha1)
+			die("Need a commit to use --commit-id-file, and not a tree.");
+	}
+
 	return ar.write_archive(&ar.args);
 }

Re: Commit ID in exported Tar Ball

[René Scharfe]

Shawn O. Pearce schrieb:
> Junio C Hamano <junkio@cox.net> wrote:
>> René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
>>
>>> Something like the following patch?  Since we're already embedding the
>>> commit ID in a comment, we might as well offer creating a synthetic file
>>> for it, too, if that solves a user's problem that might be difficult to
>>> work around otherwise.
> 
> What about being able to get the output of git-describe embedded
> into an archive file?  Doesn't git.git do that in its Makefile?  ;-)
> 
> git-describe is more human-friendly than a SHA-1...

Yes, and the Makefile does even more than that: it adds a version file,
a spec file and another version file for git-gui.

The first two are probably useful for most projects that actually do
versioned releases.  We could have a simple parser that reads a
template, replaces @@VERSION@@ with a git-describe output string and
adds the result as a synthetic file to the archive.  It's not exactly
trivial -- e.g., how to specify git-describe options, template file and
synthetic name, all in one command line parameter? -- but it's doable.

I'm not sure how the git-gui version file fits in.  I guess it's just a
special case and doesn't need git-archive support?

René

Re: Commit ID in exported Tar Ball

[Thomas Glanzmann]

Hello,

> git-describe is more human-friendly than a SHA-1...

For me git-describe does the following:

        (thinkpad) [~/work/slides] git-describe --all HEAD
        heads/master

I don't see how Michael is able to tell from this output what version
his users is using. Do I miss something? I really do like the patch Rene
has submitted. And it is exactly what is missing. A unique identifier
which tells the exact version of the tree the user is using.

        Thomas

Re: Commit ID in exported Tar Ball

[Brian Gernhardt]

On May 20, 2007, at 12:10 PM, Thomas Glanzmann wrote:

> Hello,
>
>> git-describe is more human-friendly than a SHA-1...
>
> For me git-describe does the following:
>
>         (thinkpad) [~/work/slides] git-describe --all HEAD
>         heads/master
>
> I don't see how Michael is able to tell from this output what version
> his users is using. Do I miss something? I really do like the patch  
> Rene
> has submitted. And it is exactly what is missing. A unique identifier
> which tells the exact version of the tree the user is using.

For version information it is far more useful to use --tags or no  
options (annotated tags only) instead of --all.

# On git.git's master this morning:
$ git describe HEAD
v1.5.2
$ git describe HEAD^^
v1.5.2-rc3-97-g03f6db0

~~ Brian

Re: Commit ID in exported Tar Ball

[Thomas Glanzmann]

Hello,

> For version information it is far more useful to use --tags or no options 
> (annotated tags only) instead of --all.

so this output is useless if you don't have tagged the commit which
isn't the case. But thanks for the awareness.

        Thomas

Re: Commit ID in exported Tar Ball

[Shawn O. Pearce]

Ren?? Scharfe <rene.scharfe@lsrfire.ath.cx> wrote:
> Shawn O. Pearce schrieb:
> > 
> > git-describe is more human-friendly than a SHA-1...
> 
> Yes, and the Makefile does even more than that: it adds a version file,
> a spec file and another version file for git-gui.
> 
> The first two are probably useful for most projects that actually do
> versioned releases.  We could have a simple parser that reads a
> template, replaces @@VERSION@@ with a git-describe output string and
> adds the result as a synthetic file to the archive.  It's not exactly
> trivial -- e.g., how to specify git-describe options, template file and
> synthetic name, all in one command line parameter? -- but it's doable.

Maybe something just as simple as allowing the user to specify a
shell script in-tree that we unpack and run for them?  That script
prints to stdout the content of the file to include.
 
> I'm not sure how the git-gui version file fits in.  I guess it's just a
> special case and doesn't need git-archive support?

Well, if you look at git-gui's own version script it really just
wants to do `git-describe` like git.git's script, but it cannot as
when its hosted in git.git `git-describe` gives us back Git's version
number, not git-gui's version number.  So we get cute and look for
the merge commit, and take the second parent, and describe that.
That's (by convention of how Junio works) always a true git-gui
commit.

In other words, git-gui.git gets a little whacky when Junio
distributes it in git.git.  git-gui really needs to become
a subproject.  When that happens its git-describe will become
much easier.

So now we're also really talking about, what should git-archive
do for a subproject?  Sometimes you really do want to repackage
and redistribute the subproject as part of the superproject's
tarball. Sometimes you don't.  I think in the case of git.git and
git-gui.git we want to include the subproject.  ;-)

-- 
Shawn.

Re: Commit ID in exported Tar Ball

[Peter Baumann]

On Sun, May 20, 2007 at 06:30:26PM +0200, Thomas Glanzmann wrote:
> Hello,
> 
> > For version information it is far more useful to use --tags or no options 
> > (annotated tags only) instead of --all.
> 
> so this output is useless if you don't have tagged the commit which
> isn't the case. But thanks for the awareness.
> 

No. Just _ONE_ tagged commit should be enough. In a project of mine I
tagged the root commit and as I am the only committer and this project
is so simple that I don't have to do a lot of branching/merging, I get a
single line of history

	t - o1 - o2 - o3 - o4 <- master
	^
	|- tagged root commit

This gives me very nicely enumerated commits like tag-1-g<sha1-abbrev> for
the commit o1 and tag-2-g<sha1-abbrev> for o2 ... you get the idea.

-Peter

Re: Commit ID in exported Tar Ball

[Thomas Glanzmann]

Hello,

> This gives me very nicely enumerated commits like tag-1-g<sha1-abbrev> for
> the commit o1 and tag-2-g<sha1-abbrev> for o2 ... you get the idea.

I get the idea but I don't like it very much. From my point of view the
commit id makes much more sense. The output is cut&pasted anyway.

        Thomas

Re: Commit ID in exported Tar Ball

[Shawn O. Pearce]

Thomas Glanzmann <thomas@glanzmann.de> wrote:
> so this output is useless if you don't have tagged the commit which
> isn't the case. But thanks for the awareness.

Thanks for not quoting Brian's reply.  Because I had to go and
quote it manually, so I can say its *NOT* useless...

> Brian Gernhardt <benji@silverinsanity.com> wrote:
> > For version information it is far more useful to use --tags or no
> > options (annotated tags only) instead of --all.
> > 
> > # On git.git's master this morning:
> > $ git describe HEAD
> > v1.5.2

Here whatever HEAD's commit is is exactly the commit that the tag
v1.5.2 points at.  This commit is definately v1.5.2.

> > $ git describe HEAD^^
> > v1.5.2-rc3-97-g03f6db0

Here whatever commit is 2 commits earlier than HEAD is 97 commits
*after* v1.5.2-rc3 was tagged.  That's a good deal of information
right there.  I know its v1.5.2-rc3 plus a bunch of additional
commits (97 to be exact).  Add another commit and that 97 will
go to 98.  Wow, look, an automatic version counter!  No user
intervention required!

Sometimes I don't even bother tagging git-gui fixes, for exactly
that reason.  The output of git-describe is giving me a count along
my maint branch, or my master branch.

Now that g03f6db0 suffix is also very useful, it means its the
commit whose SHA- starts with 03f6db0.  That abbreviated SHA-1
is unique at the time that git-describe ran.  At 8 hex digits it
will probably also stay unique for quite some time, even in large
projects like the kernel.

And even if that isn't unique later on, I doubt there will be another
commit with the same leading hex digits that is also 97 commits
after v1.5.2, as counted by `git-rev-list v1.5.2..$it | wc -l`.
So even in the case of a later duplicate, we can get back a full
SHA-1.

And did you know that Git knows how to parse those, and can checkout
that commit?

  $ git checkout v1.5.2-rc3-97-g03f6db0
  Note: moving to "v1.5.2-rc3-97-g03f6db0" which isn't a local branch
  If you want to create a new branch from this checkout, you may do so
  (now or later) by using -b with the checkout command again. Example:
    git checkout -b <new_branch_name>
  HEAD is now at 03f6db0... Merge branch 'maint' to synchronize with 1.5.1.6

Wow.  Magic!  Not useless!

-- 
Shawn.

Re: Commit ID in exported Tar Ball

[Thomas Glanzmann]

Hello,
so maybe it isn't that useless as I thought it initial is. But my point
still stands. I want the commit id of the HEAD in a _file within_ the
tarball and I definitively don't want to tag my project before I get a
unique identifier.

        Thomas

Re: Commit ID in exported Tar Ball

[Shawn O. Pearce]

Thomas Glanzmann <thomas@glanzmann.de> wrote:
> so maybe it isn't that useless as I thought it initial is. But my point
> still stands. I want the commit id of the HEAD in a _file within_ the
> tarball and I definitively don't want to tag my project before I get a
> unique identifier.

So what, a magic flag to git-describe like:

	git-describe --untagged HEAD
	1a8213...

Where --untagged just means "give me back the raw commit SHA-1 if
the input ref(s) aren't tagged"?

-- 
Shawn.

Re: Commit ID in exported Tar Ball

[Brian Gernhardt]

On May 21, 2007, at 2:37 AM, Thomas Glanzmann wrote:

> so maybe it isn't that useless as I thought it initial is. But my  
> point
> still stands. I want the commit id of the HEAD in a _file within_ the
> tarball and I definitively don't want to tag my project before I get a
> unique identifier.

If you haven't tagged anything, then git can't give you anything  
better than the SHA-1.  Giving a reference relative to a branch isn't  
useful, as the branch can change.  If the branch doesn't change, it's  
really a tag and you should mark it as such.

Hopefully the patch for the "in a file" flag for git-archive will go  
in soon (or something like it).  I don't think we should make it any  
more complex than just the hash, myself.  If you want something more  
complex a Makefile rule like

dist:
	git archive HEAD > dist.tar
	git describe HEAD > version-file
	tar rf dist.tar version-file
	gzip dist.tar

should do the trick.  (And you can replace "git describe" with "git  
parse-rev" for a bare hash if you'd like.)

~~ Brian

Re: Commit ID in exported Tar Ball

[Thomas Glanzmann]

Hello,

> So what, a magic flag to git-describe like:

> 	git-describe --untagged HEAD
> 	1a8213...

> Where --untagged just means "give me back the raw commit SHA-1 if
> the input ref(s) aren't tagged"?

that is something I would love to live with. But it should also get the
primary objective done which is: Get the commit-id in the tar archive.

        Thomas

Re: Commit ID in exported Tar Ball

[Thomas Glanzmann]

Hello Brian,

> dist:
> 	git archive HEAD > dist.tar
> 	git describe HEAD > version-file
> 	tar rf dist.tar version-file
> 	gzip dist.tar

to be precise that was my first idea when I started the whole
discussion. But I like the patch from Rene much more than that. Because
I want to use it with gitweb and it is definitevly easier to use realize
that with Renes patch.

        Thomas

Re: Commit ID in exported Tar Ball

[Petr Baudis]

On Mon, May 21, 2007 at 08:02:32AM CEST, Shawn O. Pearce wrote:
> Ren?? Scharfe <rene.scharfe@lsrfire.ath.cx> wrote:
> > Shawn O. Pearce schrieb:
> > > 
> > > git-describe is more human-friendly than a SHA-1...
> > 
> > Yes, and the Makefile does even more than that: it adds a version file,
> > a spec file and another version file for git-gui.
> > 
> > The first two are probably useful for most projects that actually do
> > versioned releases.  We could have a simple parser that reads a
> > template, replaces @@VERSION@@ with a git-describe output string and
> > adds the result as a synthetic file to the archive.  It's not exactly
> > trivial -- e.g., how to specify git-describe options, template file and
> > synthetic name, all in one command line parameter? -- but it's doable.
> 
> Maybe something just as simple as allowing the user to specify a
> shell script in-tree that we unpack and run for them?  That script
> prints to stdout the content of the file to include.

Specify how? At the point of git-archive execution? At that point you
usually can append the file to the archive as well.

And if you make it somehow a "project default", that becomes a huge
security risk, since anyone who clones the project and runs git-archive
will execute aribtrary code on his account.

Besides, the original motivation for this were snapshots from gitweb.
Gitweb frequently does not run with the uid of the project owner, so
this becomes a security problem as well.

Maybe some kind of format-string in .git/config...

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
		-- Samuel Beckett

Re: Commit ID in exported Tar Ball

[René Scharfe]

Shawn O. Pearce schrieb:
> Ren?? Scharfe <rene.scharfe@lsrfire.ath.cx> wrote:
>> Shawn O. Pearce schrieb:
>>> git-describe is more human-friendly than a SHA-1...
>> Yes, and the Makefile does even more than that: it adds a version file,
>> a spec file and another version file for git-gui.
>>
>> The first two are probably useful for most projects that actually do
>> versioned releases.  We could have a simple parser that reads a
>> template, replaces @@VERSION@@ with a git-describe output string and
>> adds the result as a synthetic file to the archive.  It's not exactly
>> trivial -- e.g., how to specify git-describe options, template file and
>> synthetic name, all in one command line parameter? -- but it's doable.
> 
> Maybe something just as simple as allowing the user to specify a
> shell script in-tree that we unpack and run for them?  That script
> prints to stdout the content of the file to include.

I doubt executing a shell script is simple. :-D  You'd possibly get
different results on different platforms (dare I mention Windows?).

The template system I mentioned would be a kind of scripting language
itself, but in this case we define its syntax and can guarantee
consistency everywhere git runs.  And since it would only have four
types of tokens (@@VERSION@@, @@COMMITID@@, @@@@ and string literals) it
could be fast and simple.

We could implement it as a checkout converter, preferably one that is
only applied by git-archive.  Then we'd rename git.spec.in to git.spec,
assign the "specfile" attribute to it and let git-archive replace the
string @@VERSION@@ with git-describe's output.  git-checkout would not
expand the special strings, so you can simply edit and version the file
as you can do with git.spec.in now.  Michael would have a file
containing only @@COMMITID@@ to solve his original problem.  Make sense?

> So now we're also really talking about, what should git-archive
> do for a subproject?  Sometimes you really do want to repackage
> and redistribute the subproject as part of the superproject's
> tarball. Sometimes you don't.  I think in the case of git.git and
> git-gui.git we want to include the subproject.  ;-)

Oh, yes, subprojects.  git-archive currently exports them as empty
directories.  Using tar's append command you could simply build the
project+subproject archive in the Makefile.  That wouldn't work well
with gitweb, though.  Perhaps a --include-subproject=<path> option is
needed?

René

Re: Commit ID in exported Tar Ball

[René Scharfe]

[I'm quoting myself in full because I somehow sent my reply to everyone
but Shawn.  A patch can be found at the end.]

René Scharfe schrieb:
> Shawn O. Pearce schrieb:
>> Ren?? Scharfe <rene.scharfe@lsrfire.ath.cx> wrote:
>>> Shawn O. Pearce schrieb:
>>>> git-describe is more human-friendly than a SHA-1...
>>> Yes, and the Makefile does even more than that: it adds a version
>>> file, a spec file and another version file for git-gui.
>>> 
>>> The first two are probably useful for most projects that actually
>>> do versioned releases.  We could have a simple parser that reads
>>> a template, replaces @@VERSION@@ with a git-describe output
>>> string and adds the result as a synthetic file to the archive.
>>> It's not exactly trivial -- e.g., how to specify git-describe
>>> options, template file and synthetic name, all in one command
>>> line parameter? -- but it's doable.
>> Maybe something just as simple as allowing the user to specify a 
>> shell script in-tree that we unpack and run for them?  That script 
>> prints to stdout the content of the file to include.
> 
> I doubt executing a shell script is simple. :-D  You'd possibly get 
> different results on different platforms (dare I mention Windows?).
> 
> The template system I mentioned would be a kind of scripting language
>  itself, but in this case we define its syntax and can guarantee 
> consistency everywhere git runs.  And since it would only have four 
> types of tokens (@@VERSION@@, @@COMMITID@@, @@@@ and string literals)
> it could be fast and simple.
> 
> We could implement it as a checkout converter, preferably one that is
> only applied by git-archive.  Then we'd rename git.spec.in to
> git.spec, assign the "specfile" attribute to it and let git-archive
> replace the string @@VERSION@@ with git-describe's output.
> git-checkout would not expand the special strings, so you can simply
> edit and version the file as you can do with git.spec.in now.
> Michael would have a file containing only @@COMMITID@@ to solve his
> original problem.  Make sense?
> 
>> So now we're also really talking about, what should git-archive do
>> for a subproject?  Sometimes you really do want to repackage and
>> redistribute the subproject as part of the superproject's tarball.
>> Sometimes you don't.  I think in the case of git.git and 
>> git-gui.git we want to include the subproject.  ;-)
> 
> Oh, yes, subprojects.  git-archive currently exports them as empty 
> directories.  Using tar's append command you could simply build the 
> project+subproject archive in the Makefile.  That wouldn't work well 
> with gitweb, though.  Perhaps a --include-subproject=<path> option is
>  needed?

OK, so here's a first shot at the mentioned parser.  It only understands
@@COMMITID@@ and @@@@, but it's easily extendible.  The internals of
git-describe would need to be converted to library functions, preferably
offering every piece of version info separately (see thread "[PATCH]
Make sure an autogenerated version has at least four parts" for why).

Before doing that, we should determine if this is the way to, though.

René


 Documentation/gitattributes.txt |   19 ++++++-
 archive-tar.c                   |    5 ++-
 archive-zip.c                   |    5 ++-
 cache.h                         |    1 +
 convert.c                       |  112 +++++++++++++++++++++++++++++++++++++++
 5 files changed, 139 insertions(+), 3 deletions(-)

diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index d3ac9c7..84c414c 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -72,7 +72,7 @@ EFFECTS
 -------
 
 Certain operations by git can be influenced by assigning
-particular attributes to a path.  Currently, three operations
+particular attributes to a path.  Currently, four operations
 are attributes-aware.
 
 Checking-out and checking-in
@@ -374,6 +374,23 @@ frotz	unspecified
 ----------------------------------------------------------------
 
 
+Creating an archive
+~~~~~~~~~~~~~~~~~~~
+
+
+`specfile`
+^^^^^^^^^^
+
+If the attribute `specfile` is set for a file then git will expand
+several placeholders when adding this file to an archive.  The
+expansion depends on the availability of a commit ID, i.e. if
+`git-archive` has been given a tree instead of a commit or a tag
+then no replacement will be done.
+
+`@@COMMITID@@`:: is replaced by the commit hash.
+`@@@@`:: is replaced by `@@`.
+
+
 GIT
 ---
 Part of the gitlink:git[7] suite
diff --git a/archive-tar.c b/archive-tar.c
index 66fe3e3..eba24cb 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -17,6 +17,7 @@ static unsigned long offset;
 static time_t archive_time;
 static int tar_umask = 002;
 static int verbose;
+static const unsigned char *commit_sha1;
 
 /* writes out the whole block, but only if it is full */
 static void write_if_needed(void)
@@ -285,7 +286,8 @@ static int write_tar_entry(const unsigned char *sha1,
 		buffer = NULL;
 		size = 0;
 	} else {
-		buffer = convert_sha1_file(path.buf, sha1, mode, &type, &size);
+		buffer = sha1_file_to_archive(path.buf, sha1, mode, &type,
+		                              &size, commit_sha1);
 		if (!buffer)
 			die("cannot read %s", sha1_to_hex(sha1));
 	}
@@ -304,6 +306,7 @@ int write_tar_archive(struct archiver_args *args)
 
 	archive_time = args->time;
 	verbose = args->verbose;
+	commit_sha1 = args->commit_sha1;
 
 	if (args->commit_sha1)
 		write_global_extended_header(args->commit_sha1);
diff --git a/archive-zip.c b/archive-zip.c
index 444e162..93a5ab3 100644
--- a/archive-zip.c
+++ b/archive-zip.c
@@ -12,6 +12,7 @@
 static int verbose;
 static int zip_date;
 static int zip_time;
+static const unsigned char *commit_sha1;
 
 static unsigned char *zip_dir;
 static unsigned int zip_dir_size;
@@ -195,7 +196,8 @@ static int write_zip_entry(const unsigned char *sha1,
 		if (S_ISREG(mode) && zlib_compression_level != 0)
 			method = 8;
 		result = 0;
-		buffer = convert_sha1_file(path, sha1, mode, &type, &size);
+		buffer = sha1_file_to_archive(path, sha1, mode, &type, &size,
+		                              commit_sha1);
 		if (!buffer)
 			die("cannot read %s", sha1_to_hex(sha1));
 		crc = crc32(crc, buffer, size);
@@ -316,6 +318,7 @@ int write_zip_archive(struct archiver_args *args)
 	zip_dir = xmalloc(ZIP_DIRECTORY_MIN_SIZE);
 	zip_dir_size = ZIP_DIRECTORY_MIN_SIZE;
 	verbose = args->verbose;
+	commit_sha1 = args->commit_sha1;
 
 	if (args->base && plen > 0 && args->base[plen - 1] == '/') {
 		char *base = xstrdup(args->base);
diff --git a/cache.h b/cache.h
index cd875bc..0484904 100644
--- a/cache.h
+++ b/cache.h
@@ -550,6 +550,7 @@ extern void trace_argv_printf(const char **argv, int count, const char *format,
 extern char *convert_to_git(const char *path, const char *src, unsigned long *sizep);
 extern char *convert_to_working_tree(const char *path, const char *src, unsigned long *sizep);
 extern void *convert_sha1_file(const char *path, const unsigned char *sha1, unsigned int mode, enum object_type *type, unsigned long *size);
+extern void *sha1_file_to_archive(const char *path, const unsigned char *sha1, unsigned int mode, enum object_type *type, unsigned long *size, const unsigned char *commit_sha1);
 
 /* match-trees.c */
 void shift_tree(const unsigned char *, const unsigned char *, unsigned char *, int);
diff --git a/convert.c b/convert.c
index 4b26b1a..1cdaec5 100644
--- a/convert.c
+++ b/convert.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "attr.h"
 #include "run-command.h"
+#include "strbuf.h"
 
 /*
  * convert.c - convert a file when checking it out and checking it in.
@@ -667,3 +668,114 @@ void *convert_sha1_file(const char *path, const unsigned char *sha1,
 	}
 	return buffer;
 }
+
+static void strbuf_append(struct strbuf *sb, const void *s, size_t len)
+{
+	if (sb->alloc < sb->len + len) {
+		sb->alloc = (sb->len + len) * 3 / 2 + 16;
+		sb->buf = xrealloc(sb->buf, sb->alloc);
+	}
+	memcpy(sb->buf + sb->len, s, len);
+	sb->len += len;
+}
+
+static unsigned int match_keyword(const char *data, unsigned int datalen,
+                                  const char *keyword)
+{
+	unsigned int keylen = strlen(keyword);
+	if (keylen > datalen)
+		return 0;
+	if (memcmp(data, keyword, keylen))
+		return 0;
+	return keylen;
+}
+
+static void *convert_to_archive(const char *path, const void *src,
+                                unsigned long *sizep,
+                                const unsigned char *commit_sha1)
+{
+	static struct git_attr *attr_specfile;
+	struct git_attr_check check[1];
+	const char *p = src;
+	unsigned long srcsize = *sizep;
+	int at_signs = 0;
+	struct strbuf dst;
+	unsigned int match;
+	int replaced_something = 0;
+
+	if (!commit_sha1)
+		return NULL;
+
+        if (!attr_specfile)
+                attr_specfile = git_attr("specfile", 8);
+
+	check[0].attr = attr_specfile;
+	if (git_checkattr(path, ARRAY_SIZE(check), check))
+		return NULL;
+	if (!ATTR_TRUE(check[0].value))
+		return NULL;
+
+	dst.alloc = srcsize + 128;
+	dst.buf = xmalloc(dst.alloc);
+	dst.len = dst.eof = 0;
+
+	while (srcsize > 0) {
+		if ((at_signs == 0 || at_signs == 1) && *p == '@') {
+			at_signs++;
+			p++;
+			srcsize--;
+			continue;
+		}
+		if (at_signs == 1) {
+			at_signs = 0;
+			strbuf_append(&dst, "@", 1);
+		}
+		if (at_signs == 0) {
+			strbuf_append(&dst, p, 1);
+			p++;
+			srcsize--;
+			continue;
+		}
+
+		if ((match = match_keyword(p, srcsize, "@@")))
+			strbuf_append(&dst, "@@", 2);
+		else if ((match = match_keyword(p, srcsize, "COMMITID@@")))
+			strbuf_append(&dst, sha1_to_hex(commit_sha1), 40);
+		else
+			strbuf_append(&dst, "@@", 2);
+		at_signs = 0;
+		p += match;
+		srcsize -= match;
+		replaced_something = 1;
+	}
+
+	if (!replaced_something) {
+		free(dst.buf);
+		return NULL;
+	}
+
+	*sizep = dst.len;
+	return dst.buf;
+}
+
+void *sha1_file_to_archive(const char *path, const unsigned char *sha1,
+                           unsigned int mode, enum object_type *type,
+                           unsigned long *size,
+                           const unsigned char *commit_sha1)
+{
+	void *buffer = read_sha1_file(sha1, type, size);
+	if (S_ISREG(mode) && buffer) {
+		void *converted = convert_to_working_tree(path, buffer, size);
+		if (converted) {
+			free(buffer);
+			buffer = converted;
+		}
+
+		converted = convert_to_archive(path, buffer, size, commit_sha1);
+		if (converted) {
+			free(buffer);
+			buffer = converted;
+		}
+	}
+	return buffer;
+}

Re: Commit ID in exported Tar Ball

[Junio C Hamano]

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:

> OK, so here's a first shot at the mentioned parser.  It only understands
> @@COMMITID@@ and @@@@, but it's easily extendible.  The internals of
> git-describe would need to be converted to library functions, preferably
> offering every piece of version info separately (see thread "[PATCH]
> Make sure an autogenerated version has at least four parts" for why).
>
> Before doing that, we should determine if this is the way to, though.
>
> René

Hmmm.  I am torn.

It almost feels as if we'd better bite the bullet and do more
insane things in ident substitution, instead of introducing this
apparent syntax inconsistency between "$id$" and "@@COMMITID@@".

That is, we could (I am not seriously proposing to do this, as I
expect this will lead to a lot of insanity at the end):

 (1) introduce "const unsigned char commit_in_focus[20]",
     globally available to git suite, and clear it at the
     beginning of main();

 (2) teach ident substitution to expand "$commit$" to
     sprintf("$commit: %40s $", sha1_to_hex(commit_in_focus[])),
     and unexpand "$commit: .* $".

 (3) have git-archive set commit_in_focus[] before letting the
     convert_to_working_tree do its work.

 (4) later, we _might_ teach a single tree read-tree to also set
     up commit_in_focus[], so that:

	$ rm -f .git/index
        $ git checkout -f HEAD

     would expand "$commit$" in blobs.

This obviously have a lot of problems once we start adding the
commit_in_focus[] to more random programs.  Even two-tree
read-tree case would behave in an unexpected way for an
uninitiated person, if you do something like:

	$ git checkout master
        $ git checkout next

I am CC'ing Linus because he would literally hate me suggesting
the above.

Re: Commit ID in exported Tar Ball

[René Scharfe]

Junio C Hamano schrieb:
> René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
> 
>> OK, so here's a first shot at the mentioned parser.  It only understands
>> @@COMMITID@@ and @@@@, but it's easily extendible.  The internals of
>> git-describe would need to be converted to library functions, preferably
>> offering every piece of version info separately (see thread "[PATCH]
>> Make sure an autogenerated version has at least four parts" for why).
>>
>> Before doing that, we should determine if this is the way to, though.
>>
>> René
> 
> Hmmm.  I am torn.
> 
> It almost feels as if we'd better bite the bullet and do more
> insane things in ident substitution, instead of introducing this
> apparent syntax inconsistency between "$id$" and "@@COMMITID@@".

$Id$ (and $commit$) is reversible, @@COMMITID@@ is not.  That means you
can create a synthetic file byte for byte with @@COMMITID@@ (and its not
yet implemented brethren), but you can't do that with $Id$ -- it's
impossible to get rid of the dollar signs.

I'm not attached to any particular syntax.  It all started with
@@VERSION@@ from git.spec.in, which should not be implemented 1:1 anyway
(we'd need to be able to use arbitrary separators between version parts
to support different ways of ordering version numbers).

We could use $ to indicate reversible substitutions as before and @
(instead of @@) for one-way substitutions.  I can't think of any other
use than in archives, though.  It sure would be very confusing to have
such a conversion happen on checkout -- you'd need to use git-cat-file
to see the real file contents.

> That is, we could (I am not seriously proposing to do this, as I
> expect this will lead to a lot of insanity at the end):
> 
>  (1) introduce "const unsigned char commit_in_focus[20]",
>      globally available to git suite, and clear it at the
>      beginning of main();

Ugh.  Requiring another global variable doesn't smell like good design.

By the way, we already have a similar, but very different syntax: the
one format_commit_message in commit.c.  It's a one-way conversion, too.
 Maybe we should copy the relevant pieces like %H and %h from there..

Do we want git-archive specific one-way conversions that are capable of
creating files like git.spec?  Or is this just a shiny toy hypnotizing
me? 8-)

René

Re: Commit ID in exported Tar Ball

[Shawn O. Pearce]

Ren?? Scharfe <rene.scharfe@lsrfire.ath.cx> wrote:
> $Id$ (and $commit$) is reversible, @@COMMITID@@ is not.  That means you
> can create a synthetic file byte for byte with @@COMMITID@@ (and its not
> yet implemented brethren), but you can't do that with $Id$ -- it's
> impossible to get rid of the dollar signs.

Yes, and that's one of the big problems with the $Id$ syntax so
commonly used by versioning systems.  Most files you want to insert
that automatic id into want a clean id string, not something that
starts with $Id: and ends with $...

Since we are apparently supporting $Foo: ...$ to collapse back to
$Foo$ reusing that syntax for git-archive is actually probably a
bad idea.  We should support the checkout filters in git-archive (as
much as possible anyway) but what this thread has been going on is
something quite different...  so we probably want a different syntax.
Which is why I'm also in favor of the @@COMMITID@@ syntax...
 
> >  (1) introduce "const unsigned char commit_in_focus[20]",
> >      globally available to git suite, and clear it at the
> >      beginning of main();
> 
> Ugh.  Requiring another global variable doesn't smell like good design.

I agree.  We already have a lot of globals.  We need another one like
we need a hole in the head.  Especially a global like this one... ;-)
 
> Do we want git-archive specific one-way conversions that are capable of
> creating files like git.spec?  Or is this just a shiny toy hypnotizing
> me? 8-)

But aren't shiny toys fun?  ;-)

-- 
Shawn.