[PATCH 1/3] receive-pack: Two small code cleanups

[PATCH 1/3] receive-pack: Two small code cleanups

[Andreas Gruenbacher]

Rename show_ref()'s path parameter to refname.

In read_head_info(), lines may have trailing capability strings.  Throw
away such strings after evaluation; they are not needed in the command
structs.

Signed-off-by: Andreas Gruenbacher <agruen@suse.de>
---
 builtin-receive-pack.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/builtin-receive-pack.c b/builtin-receive-pack.c
index 0559fcc..77cbc2a 100644
--- a/builtin-receive-pack.c
+++ b/builtin-receive-pack.c
@@ -105,13 +105,13 @@ static int receive_pack_config(const char *var, const char *value, void *cb)
 	return git_default_config(var, value, cb);
 }
 
-static int show_ref(const char *path, const unsigned char *sha1, int flag, void *cb_data)
+static int show_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)
 {
 	if (sent_capabilities)
-		packet_write(1, "%s %s\n", sha1_to_hex(sha1), path);
+		packet_write(1, "%s %s\n", sha1_to_hex(sha1), refname);
 	else
 		packet_write(1, "%s %s%c%s%s\n",
-			     sha1_to_hex(sha1), path, 0,
+			     sha1_to_hex(sha1), refname, 0,
 			     " report-status delete-refs side-band-64k",
 			     prefer_ofs_delta ? " ofs-delta" : "");
 	sent_capabilities = 1;
@@ -524,7 +524,7 @@ static void read_head_info(void)
 		static char line[1000];
 		unsigned char old_sha1[20], new_sha1[20];
 		struct command *cmd;
-		char *refname;
+		const char *refname;
 		int len, reflen;
 
 		len = packet_read_line(0, line, sizeof(line));
@@ -548,10 +548,10 @@ static void read_head_info(void)
 			if (strstr(refname + reflen + 1, "side-band-64k"))
 				use_sideband = LARGE_PACKET_MAX;
 		}
-		cmd = xmalloc(sizeof(struct command) + len - 80);
+		cmd = xmalloc(sizeof(struct command) + reflen + 1);
 		hashcpy(cmd->old_sha1, old_sha1);
 		hashcpy(cmd->new_sha1, new_sha1);
-		memcpy(cmd->ref_name, line + 82, len - 81);
+		memcpy(cmd->ref_name, refname, reflen + 1);
 		cmd->error_string = NULL;
 		cmd->next = NULL;
 		*p = cmd;
-- 
1.6.6.243.gff6d2

[PATCH 2/3] Different views on a repository

[Andreas Gruenbacher]

Add --view options in upload-pack and receive-pack so that a repository
on the server side can be made to look like several independent
repositories on the client side.

This is implemented by transforming ref names: for example, with
--view=one/, refs/heads/one/master on the server will look like
refs/heads/master to the client, refs/tags/one/v1 will look like
refs/tags/v1, etc.

This allows to transparently share repositories on the server which
have a lot of objects in common without complicating things for the
client, and without breaking garbage collection.

Signed-off-by: Andreas Gruenbacher <agruen@suse.de>
---
 Documentation/git-receive-pack.txt |    8 +++++-
 Documentation/git-upload-pack.txt  |    9 ++++++-
 builtin-receive-pack.c             |   20 ++++++++++++++++
 refs.c                             |   44 ++++++++++++++++++++++++++++++++++++
 refs.h                             |    3 ++
 upload-pack.c                      |   11 +++++++++
 6 files changed, 93 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-receive-pack.txt b/Documentation/git-receive-pack.txt
index 2790eeb..09d7d0c 100644
--- a/Documentation/git-receive-pack.txt
+++ b/Documentation/git-receive-pack.txt
@@ -8,7 +8,7 @@ git-receive-pack - Receive what is pushed into the repository
 
 SYNOPSIS
 --------
-'git-receive-pack' <directory>
+'git-receive-pack' [--view=<prefix>] <directory>
 
 DESCRIPTION
 -----------
@@ -34,6 +34,12 @@ are not fast-forwards.
 
 OPTIONS
 -------
+--view=<prefix>::
+	Prepend <prefix> to all ref names.  For example, --view=one/ will
+	turn refs/tags/v1 into refs/tags/one/v1 on the receiving end.  Together
+	with the --view option of linkgit:git-upload-pack[1], this allows to
+	make one respository look like multiple independent repositories.
+
 <directory>::
 	The repository to sync into.
 
diff --git a/Documentation/git-upload-pack.txt b/Documentation/git-upload-pack.txt
index 71ca4ef..0eee0ba 100644
--- a/Documentation/git-upload-pack.txt
+++ b/Documentation/git-upload-pack.txt
@@ -8,7 +8,7 @@ git-upload-pack - Send objects packed back to git-fetch-pack
 
 SYNOPSIS
 --------
-'git-upload-pack' [--strict] [--timeout=<n>] <directory>
+'git-upload-pack' [--strict] [--timeout=<n>] [--view=<prefix>] <directory>
 
 DESCRIPTION
 -----------
@@ -30,6 +30,13 @@ OPTIONS
 --timeout=<n>::
 	Interrupt transfer after <n> seconds of inactivity.
 
+--view=<prefix>::
+	Only upload refs which start with <prefix>, and hide <prefix> from the
+	remote side.  For example, --view=one/ will skip refs/heads/master
+	and turn refs/tags/one/v1 into refs/tags/v1.  Together with the --view
+	option of linkgit:git-receive-pack, this allows to make one respository
+	look like multiple independent repositories.
+
 <directory>::
 	The repository to sync from.
 
diff --git a/builtin-receive-pack.c b/builtin-receive-pack.c
index 77cbc2a..44d7055 100644
--- a/builtin-receive-pack.c
+++ b/builtin-receive-pack.c
@@ -32,6 +32,7 @@ static int use_sideband;
 static int prefer_ofs_delta = 1;
 static int auto_update_server_info;
 static int auto_gc = 1;
+static const char *view;
 static const char *head_name;
 static int sent_capabilities;
 
@@ -107,6 +108,12 @@ static int receive_pack_config(const char *var, const char *value, void *cb)
 
 static int show_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)
 {
+	if (view) {
+		refname = ref_to_view(refname, view);
+		if (!refname)
+			return 0;
+	}
+
 	if (sent_capabilities)
 		packet_write(1, "%s %s\n", sha1_to_hex(sha1), refname);
 	else
@@ -548,6 +555,15 @@ static void read_head_info(void)
 			if (strstr(refname + reflen + 1, "side-band-64k"))
 				use_sideband = LARGE_PACKET_MAX;
 		}
+		if (view) {
+			const char *r;
+
+			r = view_to_ref(refname, view);
+			if (r) {
+				refname = r;
+				reflen = strlen(refname);
+			}
+		}
 		cmd = xmalloc(sizeof(struct command) + reflen + 1);
 		hashcpy(cmd->old_sha1, old_sha1);
 		hashcpy(cmd->new_sha1, new_sha1);
@@ -736,6 +752,10 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 				stateless_rpc = 1;
 				continue;
 			}
+			if (!prefixcmp(arg, "--view=")) {
+				view = arg + 7;
+				continue;
+			}
 
 			usage(receive_pack_usage);
 		}
diff --git a/refs.c b/refs.c
index f3fcbe0..b1f3951 100644
--- a/refs.c
+++ b/refs.c
@@ -1829,3 +1829,47 @@ char *shorten_unambiguous_ref(const char *ref, int strict)
 	free(short_name);
 	return xstrdup(ref);
 }
+
+const char *ref_to_view(const char *refname, const char *view)
+{
+	static char *buffer;
+	int prefix_len, view_len, suffix_len;
+	const char *r, *suffix;
+
+	if (prefixcmp(refname, "refs/"))
+		return NULL;
+	r = strchr(refname + 5, '/');
+	if (!r)
+		return NULL;
+	r++;
+	view_len = strlen(view);
+	if (strncmp(r, view, view_len))
+		return NULL;
+	suffix = r + view_len;
+	prefix_len = r - refname;
+	suffix_len = strlen(suffix);
+	buffer = xrealloc(buffer, prefix_len + suffix_len + 1);
+	sprintf(buffer, "%.*s%s", prefix_len, refname, suffix);
+	return buffer;
+}
+
+const char *view_to_ref(const char *refname, const char *view)
+{
+	static char *buffer;
+	int prefix_len, view_len, suffix_len;
+	const char *r, *suffix;
+
+	view_len = strlen(view);
+	if (prefixcmp(refname, "refs/"))
+		return NULL;
+	r = strchr(refname + 5, '/');
+	if (!r)
+		return NULL;
+	r++;
+	prefix_len = r - refname;
+	suffix = r + view_len;
+	suffix_len = strlen(suffix);
+	buffer = xrealloc(buffer, prefix_len + view_len + suffix_len + 1);
+	sprintf(buffer, "%.*s%s%s", prefix_len, refname, view, suffix);
+	return buffer;
+}
diff --git a/refs.h b/refs.h
index f7648b9..390e812 100644
--- a/refs.h
+++ b/refs.h
@@ -98,4 +98,7 @@ int update_ref(const char *action, const char *refname,
 		const unsigned char *sha1, const unsigned char *oldval,
 		int flags, enum action_on_err onerr);
 
+extern const char *ref_to_view(const char *refname, const char *view);
+extern const char *view_to_ref(const char *refname, const char *view);
+
 #endif /* REFS_H */
diff --git a/upload-pack.c b/upload-pack.c
index dc464d7..bc72471 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -34,6 +34,7 @@ static struct object_array have_obj;
 static struct object_array want_obj;
 static struct object_array extra_edge_obj;
 static unsigned int timeout;
+static const char *view;
 /* 0 for no sideband,
  * otherwise maximum packet size (up to 65520 bytes).
  */
@@ -629,6 +630,12 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo
 	if (!o)
 		die("git upload-pack: cannot find object %s:", sha1_to_hex(sha1));
 
+	if (view) {
+		refname = ref_to_view(refname, view);
+		if (!refname)
+			return 0;
+	}
+
 	if (capabilities)
 		packet_write(1, "%s %s%c%s\n", sha1_to_hex(sha1), refname,
 			0, capabilities);
@@ -711,6 +718,10 @@ int main(int argc, char **argv)
 			daemon_mode = 1;
 			continue;
 		}
+		if (!prefixcmp(arg, "--view=")) {
+			view = arg + 7;
+			continue;
+		}
 		if (!strcmp(arg, "--")) {
 			i++;
 			break;
-- 
1.6.6.243.gff6d2

[PATCH 3/3] Different views on a repository: HEAD mapping

[Andreas Gruenbacher]

The HEAD ref is not located under .git/refs/heads/, so the trivial
view mapping doesn't work.  Fix this by making refs/heads/<view>HEAD
appear as HEAD on the client when --view=<view> is used in upload-pack
and receive-pack.

Signed-off-by: Andreas Gruenbacher <agruen@suse.de>
---
 Documentation/git-receive-pack.txt |    7 ++++---
 Documentation/git-upload-pack.txt  |    9 +++++----
 refs.c                             |   21 ++++++++++++++++++---
 refs.h                             |    1 +
 upload-pack.c                      |    9 +++++++--
 5 files changed, 35 insertions(+), 12 deletions(-)

diff --git a/Documentation/git-receive-pack.txt b/Documentation/git-receive-pack.txt
index 09d7d0c..07e0159 100644
--- a/Documentation/git-receive-pack.txt
+++ b/Documentation/git-receive-pack.txt
@@ -36,9 +36,10 @@ OPTIONS
 -------
 --view=<prefix>::
 	Prepend <prefix> to all ref names.  For example, --view=one/ will
-	turn refs/tags/v1 into refs/tags/one/v1 on the receiving end.  Together
-	with the --view option of linkgit:git-upload-pack[1], this allows to
-	make one respository look like multiple independent repositories.
+	turn refs/tags/v1 into refs/tags/one/v1 and HEAD into refs/heads/one/HEAD
+	on the receiving end.  Together with the --view option of
+	linkgit:git-upload-pack[1], this allows to make one respository look like
+	multiple independent repositories.
 
 <directory>::
 	The repository to sync into.
diff --git a/Documentation/git-upload-pack.txt b/Documentation/git-upload-pack.txt
index 0eee0ba..1e8b76b 100644
--- a/Documentation/git-upload-pack.txt
+++ b/Documentation/git-upload-pack.txt
@@ -32,10 +32,11 @@ OPTIONS
 
 --view=<prefix>::
 	Only upload refs which start with <prefix>, and hide <prefix> from the
-	remote side.  For example, --view=one/ will skip refs/heads/master
-	and turn refs/tags/one/v1 into refs/tags/v1.  Together with the --view
-	option of linkgit:git-receive-pack, this allows to make one respository
-	look like multiple independent repositories.
+	remote side.  For example, --view=one/ will skip refs/heads/master,
+	turn refs/tags/one/v1 into refs/tags/v1, and refs/heads/one/HEAD into
+	HEAD.  Together with the --view option of linkgit:git-receive-pack,
+	this allows to make one respository look like multiple independent
+	repositories.
 
 <directory>::
 	The repository to sync from.
diff --git a/refs.c b/refs.c
index b1f3951..77a6267 100644
--- a/refs.c
+++ b/refs.c
@@ -650,16 +650,21 @@ end_each:
 	return retval;
 }
 
-int head_ref(each_ref_fn fn, void *cb_data)
+int one_ref(each_ref_fn fn, void *cb_data, const char *refname)
 {
 	unsigned char sha1[20];
 	int flag;
 
-	if (resolve_ref("HEAD", sha1, 1, &flag))
-		return fn("HEAD", sha1, flag, cb_data);
+	if (resolve_ref(refname, sha1, 1, &flag))
+		return fn(refname, sha1, flag, cb_data);
 	return 0;
 }
 
+int head_ref(each_ref_fn fn, void *cb_data)
+{
+	return one_ref(fn, cb_data, "HEAD");
+}
+
 int for_each_ref(each_ref_fn fn, void *cb_data)
 {
 	return do_for_each_ref("refs/", fn, 0, 0, cb_data);
@@ -1846,6 +1851,10 @@ const char *ref_to_view(const char *refname, const char *view)
 	if (strncmp(r, view, view_len))
 		return NULL;
 	suffix = r + view_len;
+	if (!strncmp(refname + 5, "heads/", 6) &&
+	    !strcmp(suffix, "HEAD"))
+		return "HEAD";
+
 	prefix_len = r - refname;
 	suffix_len = strlen(suffix);
 	buffer = xrealloc(buffer, prefix_len + suffix_len + 1);
@@ -1860,6 +1869,12 @@ const char *view_to_ref(const char *refname, const char *view)
 	const char *r, *suffix;
 
 	view_len = strlen(view);
+	if (!strcmp(refname, "HEAD")) {
+		buffer = xrealloc(buffer, view_len + 16);
+		sprintf(buffer, "refs/heads/%sHEAD", view);
+		return buffer;
+	}
+
 	if (prefixcmp(refname, "refs/"))
 		return NULL;
 	r = strchr(refname + 5, '/');
diff --git a/refs.h b/refs.h
index 390e812..addcc2d 100644
--- a/refs.h
+++ b/refs.h
@@ -18,6 +18,7 @@ struct ref_lock {
  * and returns the value
  */
 typedef int each_ref_fn(const char *refname, const unsigned char *sha1, int flags, void *cb_data);
+extern int one_ref(each_ref_fn, void *, const char *);
 extern int head_ref(each_ref_fn, void *);
 extern int for_each_ref(each_ref_fn, void *);
 extern int for_each_ref_in(const char *, each_ref_fn, void *);
diff --git a/upload-pack.c b/upload-pack.c
index bc72471..b3bf20f 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -668,13 +668,18 @@ static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag
 
 static void upload_pack(void)
 {
+	const char *head = "HEAD";
+
+	if (view)
+		head = view_to_ref(head, view);
+
 	if (advertise_refs || !stateless_rpc) {
 		reset_timeout();
-		head_ref(send_ref, NULL);
+		one_ref(send_ref, NULL, head);
 		for_each_ref(send_ref, NULL);
 		packet_flush(1);
 	} else {
-		head_ref(mark_our_ref, NULL);
+		one_ref(mark_our_ref, NULL, head);
 		for_each_ref(mark_our_ref, NULL);
 	}
 	if (advertise_refs)
-- 
1.6.6.243.gff6d2

[RFC][PATCH 0/3] Different views on a repository

[Andreas Gruenbacher]

Hello,

we have a use case with groups of repositories which share lots of
objects, but which are logically independent.  There is no strict
hierarchy between the repositories, the development modl is arbitrary.
The alternates mechanism for sharig objects between repositories won't
work.

The best idea I came up with so far to solve this was to keep everything
in the same repository on the server.  Then, to keep the logically
independent repositories separate, directories are used below refs/heads
and refs/tags.  Receive-pack and upload-pack are modified to hide this
directory structure from clients so that repositories will continue to
look "normal" to users.  For example, the following structure on the
server:

	refs/heads/one/master
	refs/tags/one/tag1
	refs/heads/two/master
	refs/heads/two/branch2

would appear as two independent repositories to different clients:

	refs/heads/master
	refs/tags/tag1

and:

	refs/heads/master
	refs/heads/branch2

The following three patches implement this.  What do you guys think --
does the basic idea and implementation look sensible, or am I
overlooking a way to solve this kind of problem with other means?

Thanks!


  receive-pack: Two small code cleanups
  Different views on a repository
  Different views on a repository: HEAD mapping

 Documentation/git-receive-pack.txt |    9 ++++-
 Documentation/git-upload-pack.txt  |   10 +++++-
 builtin-receive-pack.c             |   32 ++++++++++++++---
 refs.c                             |   65 ++++++++++++++++++++++++++++++++++--
 refs.h                             |    4 ++
 upload-pack.c                      |   20 ++++++++++-
 6 files changed, 127 insertions(+), 13 deletions(-)

Re: [PATCH 1/3] receive-pack: Two small code cleanups

[Shawn O. Pearce]

Andreas Gruenbacher <agruen@suse.de> wrote:
> Rename show_ref()'s path parameter to refname.
> 
> In read_head_info(), lines may have trailing capability strings.  Throw
> away such strings after evaluation; they are not needed in the command
> structs.
> 
> Signed-off-by: Andreas Gruenbacher <agruen@suse.de>

Acked-by: Shawn O. Pearce <spearce@spearce.org>

-- 
Shawn.

Re: [PATCH 2/3] Different views on a repository

[Shawn O. Pearce]

Andreas Gruenbacher <agruen@suse.de> wrote:
> Add --view options in upload-pack and receive-pack so that a repository
> on the server side can be made to look like several independent
> repositories on the client side.

Before saying this is good... I'd like to know how a repository owner
is supposed to set these options on the user started invocations
of other remote side program.

Right now, I don't see how this is too different from just
doing the following on a client:

  git init
  git remote add origin URL
  git config remote.origin.fetch refs/heads/one/*:refs/remotes/origin/*

and therefore shouldn't just be handled on the *client* side of the
connection, as part of the remote setup and push matching refs rules.

(Of course, the push matching ref logic is messy too... adding yet
more into that pile might also be ugly.)

> +const char *view_to_ref(const char *refname, const char *view)
> +{
> +	static char *buffer;
...
> +	buffer = xrealloc(buffer, prefix_len + view_len + suffix_len + 1);
> +	sprintf(buffer, "%.*s%s%s", prefix_len, refname, view, suffix);
> +	return buffer;

I'd rather not use a static buffer like this.  Why not alloc and let
the caller free?  Or have the caller pass in a strbuf you populate
for them?

-- 
Shawn.

Re: [PATCH 2/3] Different views on a repository

[Michael J Gruber]

Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57:
> Add --view options in upload-pack and receive-pack so that a repository
> on the server side can be made to look like several independent
> repositories on the client side.
> 
> This is implemented by transforming ref names: for example, with
> --view=one/, refs/heads/one/master on the server will look like
> refs/heads/master to the client, refs/tags/one/v1 will look like
> refs/tags/v1, etc.
> 
> This allows to transparently share repositories on the server which
> have a lot of objects in common without complicating things for the
> client, and without breaking garbage collection.

Just from this description, I can't see why the same can't be done with
appropriate refspecs. (A helper for doing that would be more welcome, of
course.)

Maybe a few tests and documentation (i.e. examples, not just the option
description) would clear this up?

Michael

Re: [PATCH 2/3] Different views on a repository

[Andreas Gruenbacher]

On Thursday 25 February 2010 10:01:43 Michael J Gruber wrote:
> Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57:
> > Add --view options in upload-pack and receive-pack so that a repository
> > on the server side can be made to look like several independent
> > repositories on the client side.
> >
> > This is implemented by transforming ref names: for example, with
> > --view=one/, refs/heads/one/master on the server will look like
> > refs/heads/master to the client, refs/tags/one/v1 will look like
> > refs/tags/v1, etc.
> >
> > This allows to transparently share repositories on the server which
> > have a lot of objects in common without complicating things for the
> > client, and without breaking garbage collection.
> 
> Just from this description, I can't see why the same can't be done with
> appropriate refspecs. (A helper for doing that would be more welcome, of
> course.)

You mean on the client side? The problem then is that a simple "git clone" 
would not do the right thing anymore; you would still expose server-side 
implementation details to clients. Clients shouldn't have to bother with this 
added complexity. (They might not even have access to some of the views.) When 
you do the mapping server-side, you can split or merge repositories as needed 
without the clients even noticing.

> Maybe a few tests and documentation (i.e. examples, not just the option
> description) would clear this up?

Indeed, I should add some more background info.

Thanks,
Andreas

Re: [PATCH 2/3] Different views on a repository

[Michael J Gruber]

Andreas Gruenbacher venit, vidit, dixit 25.02.2010 10:25:
> On Thursday 25 February 2010 10:01:43 Michael J Gruber wrote:
>> Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57:
>>> Add --view options in upload-pack and receive-pack so that a repository
>>> on the server side can be made to look like several independent
>>> repositories on the client side.
>>>
>>> This is implemented by transforming ref names: for example, with
>>> --view=one/, refs/heads/one/master on the server will look like
>>> refs/heads/master to the client, refs/tags/one/v1 will look like
>>> refs/tags/v1, etc.
>>>
>>> This allows to transparently share repositories on the server which
>>> have a lot of objects in common without complicating things for the
>>> client, and without breaking garbage collection.
>>
>> Just from this description, I can't see why the same can't be done with
>> appropriate refspecs. (A helper for doing that would be more welcome, of
>> course.)
> 
> You mean on the client side? The problem then is that a simple "git clone" 
> would not do the right thing anymore; you would still expose server-side 
> implementation details to clients. Clients shouldn't have to bother with this 
> added complexity. (They might not even have access to some of the views.) When 
> you do the mapping server-side, you can split or merge repositories as needed 
> without the clients even noticing.

But the client has to request a specific view, doesn't it? You have to
tell all clients "don't just clone, use...", where the "..." don't seem
to be part of the series yet. [I could see 0/3 on gmane only now, by the
way.]

I just can't help the impression that this is a use case which does not
need a new feature, at least not upload/receive-pack wise. It's more a
matter of ensuring that all clients use a specific configuration (which
you would have to with your patch as well, AFAICT), and this more
general issue is creeping up again and again, with no agreeable solution
so far.

Cheers,
Michael

Re: [PATCH 2/3] Different views on a repository

[Andreas Gruenbacher]

On Thursday 25 February 2010 13:30:24 Michael J Gruber wrote:
> Andreas Gruenbacher venit, vidit, dixit 25.02.2010 10:25:
> > On Thursday 25 February 2010 10:01:43 Michael J Gruber wrote:
> >> Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57:
> >>> Add --view options in upload-pack and receive-pack so that a repository
> >>> on the server side can be made to look like several independent
> >>> repositories on the client side.
> >>>
> >>> This is implemented by transforming ref names: for example, with
> >>> --view=one/, refs/heads/one/master on the server will look like
> >>> refs/heads/master to the client, refs/tags/one/v1 will look like
> >>> refs/tags/v1, etc.
> >>>
> >>> This allows to transparently share repositories on the server which
> >>> have a lot of objects in common without complicating things for the
> >>> client, and without breaking garbage collection.
> >>
> >> Just from this description, I can't see why the same can't be done with
> >> appropriate refspecs. (A helper for doing that would be more welcome, of
> >> course.)
> >
> > You mean on the client side? The problem then is that a simple "git
> > clone" would not do the right thing anymore; you would still expose
> > server-side implementation details to clients. Clients shouldn't have to
> > bother with this added complexity. (They might not even have access to
> > some of the views.) When you do the mapping server-side, you can split or
> > merge repositories as needed without the clients even noticing.
> 
> But the client has to request a specific view, doesn't it?

No, it's a server side thing. The git commands affected are upload-pack and 
receice-pack, and those run on the remote end of a fetch. For example, the 
client would ask the server for repository /foo/one or /foo/two, and the 
server would map that to different views of /bar/shared: when the client asks 
the server to run "git-upload-pack /foo/one", the server runs "git-upload-pack 
--view=one/ /bar/shared" instead.

This is relatively easy to set up over ssh using a simple script; for direct 
git access, a small wrapper daemon would be needed. I'm not sure how this 
could be hacked into http access, but it doesn't seem all that hard, either.

> I just can't help the impression that this is a use case which does not
> need a new feature, at least not upload/receive-pack wise.

Still, even with the additional explanation above?

> It's more a matter of ensuring that all clients use a specific configuration
> (which you would have to with your patch as well, AFAICT), and this more
> general issue is creeping up again and again, with no agreeable solution
> so far.

Well that's another problem which we indeed also have for enabling things like 
local consistency checks and merge drivers. I don't have a good answer here :)

Thanks,
Andreas

Re: [PATCH 2/3] Different views on a repository

[Junio C Hamano]

Andreas Gruenbacher <agruen@suse.de> writes:

> No, it's a server side thing.

If it were a server side thing, then I would expect no change to
send/receive pack.  Instead your clients will access distinct URL as if
they are different repositories.

    git clone git://example.com/pub/scm/git/A
    git push example.com:/pub/scm/git/B master
    git pull http://example.com/pub/scm/git/C

They should not have to care that the server is cheating to save disk
space, and they should be able to access your server with Git v1.6.0.

Instead, the server side would:

 - have separate repositories, A, B and C, as normal repositories;

 - these repositories share their object stores by having their
   .git/objects pointing at a shared location via a symlink;

 - on the server side, gc/prune/fsck will have to be updated so that when
   the object store of a repository (say A) is shared with something else,
   they will consider refs in other repositories (B and C) also as the
   root of traversal.

So if this were a server side solution, I would expect the series would
add:

 - a way to set up a shared object store;

 - a way to maintain a list of backlinks to repositories that share an
   object store;

 - a way to create a new repository that shares the object store
   (e.g. create a symlink to the shared store instead of having its own
   .git/objects/, and add itself to the list of backlinks for the shared
   object store);

 - a way to retire an existing such repository (rm -rf and remove itself
   from the list of backlinks);

 - update gc/prune/fsck to honor such a list of backlinks.

This would help a "forks" setup commonly seen at places like repo.or.cz
and github.com among others.

One thing that is missing from the above handwaving outline that your
"different views" offers is a "consolidated view", a pseudo-repository
that allows you to see refs from individual real (from the client's and
project participant's point of view) repositories as if they are in
individual subhierarchies of the ref namespace.

I however suspect that you didn't want such a view in the first place if
there weren't issues around reachability.  In other words, I suspect that
you invented it merely as one possible solution to the reachability issue,
and it was not your goal to have such a consolidated view by itself.

Re: [RFC][PATCH 0/3] Different views on a repository

[James Pickens]

On Wed, Feb 24, 2010, Andreas Gruenbacher <agruen@suse.de> wrote:
> we have a use case with groups of repositories which share lots of
> objects, but which are logically independent.  There is no strict
> hierarchy between the repositories, the development modl is arbitrary.
> The alternates mechanism for sharig objects between repositories won't
> work.

Can you elaborate on why alternates won't work?

> The best idea I came up with so far to solve this

Solve what?  You didn't mention any specific problem.  Are you just trying
to save disk space by not storing multiple copies of the same objects?

James

Re: [PATCH 2/3] Different views on a repository

[Andreas Gruenbacher]

On Thursday 25 February 2010 18:28:46 Junio C Hamano wrote:
> Andreas Gruenbacher <agruen@suse.de> writes:
> > No, it's a server side thing.
> 
> If it were a server side thing, then I would expect no change to
> send/receive pack.
>
> Instead your clients will access distinct URL as if they are different
> repositories.
> 
>     git clone git://example.com/pub/scm/git/A
>     git push example.com:/pub/scm/git/B master
>     git pull http://example.com/pub/scm/git/C
>
> They should not have to care that the server is cheating to save disk
> space, and they should be able to access your server with Git v1.6.0.
> 
> Instead, the server side would:
> 
>  - have separate repositories, A, B and C, as normal repositories;
> 
>  - these repositories share their object stores by having their
>    .git/objects pointing at a shared location via a symlink;
> 
>  - on the server side, gc/prune/fsck will have to be updated so that when
>    the object store of a repository (say A) is shared with something else,
>    they will consider refs in other repositories (B and C) also as the
>    root of traversal.

I was proposing to change receive-pack and upload-pack, both of which are 
running on the server; there was no mention of send-pack.  What I've proposed 
is not a complete solution, but it should suffice to show the idea.

Your alternative proposal would also solve my problem, in a different way.  
With either approach, what looks like separate repositories A, B, and C to 
clients looks like one repository to gc/prune/fsck.

> So if this were a server side solution, I would expect the series would
> add:
> 
>  - a way to set up a shared object store;
> 
>  - a way to maintain a list of backlinks to repositories that share an
>    object store;
> 
>  - a way to create a new repository that shares the object store
>    (e.g. create a symlink to the shared store instead of having its own
>    .git/objects/, and add itself to the list of backlinks for the shared
>    object store);
> 
>  - a way to retire an existing such repository (rm -rf and remove itself
>    from the list of backlinks);
> 
>  - update gc/prune/fsck to honor such a list of backlinks.
> 
> This would help a "forks" setup commonly seen at places like repo.or.cz
> and github.com among others.

Yes. I'm don't know how big a problem this is for those kinds of hosters; in 
our case, it is a big problem.

> One thing that is missing from the above handwaving outline that your
> "different views" offers is a "consolidated view", a pseudo-repository
> that allows you to see refs from individual real (from the client's and
> project participant's point of view) repositories as if they are in
> individual subhierarchies of the ref namespace.

I have been talking about a repository and different subsets or views of that 
repository; you call the former a consolidated view and the latter a 
repository.  Those are really just two sides of the same coin.

> I however suspect that you didn't want such a view in the first place if
> there weren't issues around reachability.  In other words, I suspect that
> you invented it merely as one possible solution to the reachability issue,
> and it was not your goal to have such a consolidated view by itself.

I'm actually not sure.  The "consolidated view" as you put it may be useful 
all by itself; it would be a proper, self sufficient git repository -- a 
really nice property.  it may be too painful to maintain this view though.

When sharing objects across repositories, the worst-case scenario is that 
something goes wrong with the backlinks.  You will eventually lose objects, 
but it may take a while until it happens and until you notice, with a lot of 
damage.  That's nasty.

A combination of the two approaches would be to "link forward" instead of 
"linking back", so that the consolidated view would maintain itself, with a 
server repo setup like this:

	/repos/ABC:
		objects
		refs/tags/A/
		refs/tags/B/
		refs/heads/A/
		refs/heads/B/

	/repos/A:
		refs/tags -> /repos/ABC/refs/tags/A/
		refs/heads -> /repos/ABC/refs/heads/A/
		objects -> /repos/ABC/objects/

	/repos/B:
		refs/tags -> /repos/ABC/refs/tags/B/
		refs/heads -> /repos/ABC/refs/heads/B/
		objects -> /repos/ABC/objects/

This could be made safe by not doing garbage collection if objects is a 
symlink instead of a directory.  (The ABC repo could be garbage collected as 
usual.)  Am I overlooking anything why this can't work?


Thanks,
Andreas

Re: [RFC][PATCH 0/3] Different views on a repository

[Adam Brewster]

On Wed, Feb 24, 2010 at 11:41 AM, Andreas Gruenbacher <agruen@suse.de>wrote:

> Hello,
>
> we have a use case with groups of repositories which share lots of
> objects, but which are logically independent.  There is no strict
> hierarchy between the repositories, the development modl is arbitrary.
> The alternates mechanism for sharig objects between repositories won't
> work.
>

I don't know what you're trying to accomplish, but for what it's
worth, I do something similar with a couple of shell scripts.

My goal was to make bundles (basically thin packs) smaller by taking
advantage of files I knew were available on the far side of the
air-gap even if they weren't in a the repository I was bundling that
particular day.

The idea was to push from all of my repositories into a super
repository with a fancy (and auto-generated) refspec.  The actual code
is impenetrable, but reconstructing it in everybody's favorite IDE,
gmail, I came up with

#!/bin/bash
PROJECTS=$HOME/projects
SUPER=$HOME/projects/.git-super-repo

[[ -d "$SUPER" ]] || \
 (mkdir "$SUPER"; git --git-dir="$SUPER" init)

for i in $PROJECTS/*/.git; do
 name=$(basename "$(dirname "$0")")
 echo "$SUPER/objects" > $i/.git/objects/info/alternates
 git --git-dir="$i" push -f "$SUPER" "*:refs/$name/*"
done

git --git-dir="$SUPER" gc --aggressive

for i in $PROJECTS/*/.git; do
 git --git-dir="$i" repack -Ad #unnecessary?
 git --git-dir="$i" gc --aggressive
done

Clearly I can lose data if I try to rebase $SUPER or something, but I
think it's pretty safe for normal use.

In your case, the "projects" are called "views".

Adam

Re: [RFC][PATCH 0/3] Different views on a repository

[Andreas Gruenbacher]

On Friday 26 February 2010 05:12:45 Adam Brewster wrote:
> The idea was to push from all of my repositories into a super repository
> with a fancy (and auto-generated) refspec.  The actual code is
> impenetrable, but reconstructing it in everybody's favorite IDE, gmail, I
> came up with
> 
> #!/bin/bash
> PROJECTS=$HOME/projects
> SUPER=$HOME/projects/.git-super-repo
> 
> [[ -d "$SUPER" ]] || \
>   (mkdir "$SUPER"; git --git-dir="$SUPER" init)
> 
> for i in $PROJECTS/*/.git; do
>   name=$(basename "$(dirname "$0")")
>   echo "$SUPER/objects" > $i/.git/objects/info/alternates
>   git --git-dir="$i" push -f "$SUPER" "*:refs/$name/*"
> done
> 
> git --git-dir="$SUPER" gc --aggressive
> 
> for i in $PROJECTS/*/.git; do
>   git --git-dir="$i" repack -Ad #unnecessary?
>   git --git-dir="$i" gc --aggressive
> done

I see, when multiple projects share the same objects, you push them into those 
projects independently first, and the script will later move them to $SUPER.

> Clearly I can lose data if I try to rebase $SUPER or something, but I think
> it's pretty safe for normal use.

It looks safe unless somebody messes with $SUPER.  A lot of repacking will 
still occur as part of moving stuff to $SUPER, though.

I was trying to set things up so that this extra work won't be necessary in 
the first place.

Thanks!

Andreas

Re: [PATCH 2/3] Different views on a repository

[Andreas Gruenbacher]

On Friday 26 February 2010 01:45:33 Andreas Gruenbacher wrote:
> A combination of the two approaches would be to "link forward" instead of
> "linking back", so that the consolidated view would maintain itself, with a
> server repo setup like this:
> 
> 	/repos/ABC:
> 		objects
> 		refs/tags/A/
> 		refs/tags/B/
> 		refs/heads/A/
> 		refs/heads/B/
> 
> 	/repos/A:
> 		refs/tags -> /repos/ABC/refs/tags/A/
> 		refs/heads -> /repos/ABC/refs/heads/A/
> 		objects -> /repos/ABC/objects/
> 
> 	/repos/B:
> 		refs/tags -> /repos/ABC/refs/tags/B/
> 		refs/heads -> /repos/ABC/refs/heads/B/
> 		objects -> /repos/ABC/objects/
> 
> This could be made safe by not doing garbage collection if objects is a
> symlink instead of a directory.  (The ABC repo could be garbage collected
>  as usual.)  Am I overlooking anything why this can't work?

Self reply: reference packing breaks this kind of setup.  Crap.

Andreas