[PATCH v3 1/3] Support multiple virtual repositories with a single object store and refs

[PATCH v3 1/3] Support multiple virtual repositories with a single object store and refs

[Jamey Sharp]

From: Josh Triplett <josh@joshtriplett.org>

Given many repositories with copies of the same objects (such as branches of
the same source), sharing a common object store will avoid duplication.
Alternates provide a single baseline, but don't handle ongoing activity in the
various repositories.  Furthermore, operations such as git-gc need to know
about all of the refs.

Git supports storing multiple virtual repositories within the object store and
references of a single underlying repository.  The underlying repository
stores the objects for all of the virtual repositories, and includes all the
refs and heads of the virtual repositories using prefixed names.

git-upload-pack and git-receive-pack rewrite the names of refs and heads as
specified by the --ref-prefix and --head options.  For instance,
--ref-prefix=virtual/reponame/ will use refs/virtual/reponame/heads/* and
refs/virtual/reponame/tags/*.  git-upload-pack and git-receive-pack will
ignore any references that do not match the specified prefix.

These options implement the underlying mechanism for virtual
repositories; the higher-level protocol handler (such as http-backend or
a custom server) can pass these options when invoking upload-pack or
receive-pack, providing values based on components of the repository
path.  For a simple local test, git-remote-ext works:

git clone ext::'git %s --ref-prefix=virtual/reponame/ --head=virtual-HEAD/reponame storage.git'

Commit by Josh Triplett and Jamey Sharp.

Signed-off-by: Josh Triplett <josh@joshtriplett.org>
Signed-off-by: Jamey Sharp <jamey@minilop.net>
---
v2: remove accidentally-included debug message, and add patch 2/2 for
    git-http-backend.
v3: add patch 3/3 with documentation for virtual repositories, and
    incorporated feedback from Jeff King and Junio C Hamano.

 builtin/receive-pack.c |   36 +++++++++++++++++++++++++++---------
 upload-pack.c          |   34 +++++++++++++++++++++++++++-------
 2 files changed, 54 insertions(+), 16 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index e1ba4dc..76dacd0 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -34,6 +34,8 @@ static int prefer_ofs_delta = 1;
 static int auto_update_server_info;
 static int auto_gc = 1;
 static const char *head_name;
+static const char *head_path = "HEAD";
+static const char *ref_prefix = "refs/";
 static int sent_capabilities;
 
 static enum deny_action parse_deny_action(const char *var, const char *value)
@@ -108,11 +110,12 @@ static int receive_pack_config(const char *var, const char *value, void *cb)
 
 static int show_ref(const char *path, const unsigned char *sha1, int flag, void *cb_data)
 {
+	const char *refnameprefix = cb_data;
 	if (sent_capabilities)
-		packet_write(1, "%s %s\n", sha1_to_hex(sha1), path);
+		packet_write(1, "%s %s%s\n", sha1_to_hex(sha1), refnameprefix, path);
 	else
-		packet_write(1, "%s %s%c%s%s\n",
-			     sha1_to_hex(sha1), path, 0,
+		packet_write(1, "%s %s%s%c%s%s\n",
+			     sha1_to_hex(sha1), refnameprefix, path, 0,
 			     " report-status delete-refs side-band-64k",
 			     prefer_ofs_delta ? " ofs-delta" : "");
 	sent_capabilities = 1;
@@ -121,9 +124,9 @@ static int show_ref(const char *path, const unsigned char *sha1, int flag, void
 
 static void write_head_info(void)
 {
-	for_each_ref(show_ref, NULL);
+	for_each_ref_in(ref_prefix, show_ref, "refs/");
 	if (!sent_capabilities)
-		show_ref("capabilities^{}", null_sha1, 0, NULL);
+		show_ref("capabilities^{}", null_sha1, 0, "");
 
 }
 
@@ -332,6 +335,8 @@ static void refuse_unconfigured_deny_delete_current(void)
 static const char *update(struct command *cmd)
 {
 	const char *name = cmd->ref_name;
+	struct strbuf prefixed_name_buf = STRBUF_INIT;
+	const char *prefixed_name;
 	unsigned char *old_sha1 = cmd->old_sha1;
 	unsigned char *new_sha1 = cmd->new_sha1;
 	struct ref_lock *lock;
@@ -342,7 +347,10 @@ static const char *update(struct command *cmd)
 		return "funny refname";
 	}
 
-	if (is_ref_checked_out(name)) {
+	strbuf_addf(&prefixed_name_buf, "%s%s", ref_prefix, name + 5);
+	prefixed_name = strbuf_detach(&prefixed_name_buf, NULL);
+
+	if (is_ref_checked_out(prefixed_name)) {
 		switch (deny_current_branch) {
 		case DENY_IGNORE:
 			break;
@@ -370,7 +378,7 @@ static const char *update(struct command *cmd)
 			return "deletion prohibited";
 		}
 
-		if (!strcmp(name, head_name)) {
+		if (!strcmp(prefixed_name, head_name)) {
 			switch (deny_delete_current) {
 			case DENY_IGNORE:
 				break;
@@ -426,14 +434,14 @@ static const char *update(struct command *cmd)
 			rp_warning("Allowing deletion of corrupt ref.");
 			old_sha1 = NULL;
 		}
-		if (delete_ref(name, old_sha1, 0)) {
+		if (delete_ref(prefixed_name, old_sha1, 0)) {
 			rp_error("failed to delete %s", name);
 			return "failed to delete";
 		}
 		return NULL; /* good */
 	}
 	else {
-		lock = lock_any_ref_for_update(name, old_sha1, 0);
+		lock = lock_any_ref_for_update(prefixed_name, old_sha1, 0);
 		if (!lock) {
 			rp_error("failed to lock %s", name);
 			return "failed to lock";
@@ -760,6 +768,16 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 				advertise_refs = 1;
 				continue;
 			}
+			if (!prefixcmp(arg, "--head=")) {
+				head_path = arg+7;
+				continue;
+			}
+			if (!prefixcmp(arg, "--ref-prefix=")) {
+				struct strbuf prefixbuf = STRBUF_INIT;
+				strbuf_addf(&prefixbuf, "refs/%s", arg+13);
+				ref_prefix = strbuf_detach(&prefixbuf, NULL);
+				continue;
+			}
 			if (!strcmp(arg, "--stateless-rpc")) {
 				stateless_rpc = 1;
 				continue;
diff --git a/upload-pack.c b/upload-pack.c
index ce5cbbe..a1e495f 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -34,6 +34,8 @@ static int shallow_nr;
 static struct object_array have_obj;
 static struct object_array want_obj;
 static struct object_array extra_edge_obj;
+static const char *head_path = "HEAD";
+static const char *ref_prefix = "";
 static unsigned int timeout;
 /* 0 for no sideband,
  * otherwise maximum packet size (up to 65520 bytes).
@@ -640,17 +642,18 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo
 	static const char *capabilities = "multi_ack thin-pack side-band"
 		" side-band-64k ofs-delta shallow no-progress"
 		" include-tag multi_ack_detailed";
+	const char *refnameprefix = cb_data;
 	struct object *o = parse_object(sha1);
 
 	if (!o)
 		die("git upload-pack: cannot find object %s:", sha1_to_hex(sha1));
 
 	if (capabilities)
-		packet_write(1, "%s %s%c%s%s\n", sha1_to_hex(sha1), refname,
+		packet_write(1, "%s %s%s%c%s%s\n", sha1_to_hex(sha1), refnameprefix, refname,
 			     0, capabilities,
 			     stateless_rpc ? " no-done" : "");
 	else
-		packet_write(1, "%s %s\n", sha1_to_hex(sha1), refname);
+		packet_write(1, "%s %s%s\n", sha1_to_hex(sha1), refnameprefix, refname);
 	capabilities = NULL;
 	if (!(o->flags & OUR_REF)) {
 		o->flags |= OUR_REF;
@@ -659,7 +662,7 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo
 	if (o->type == OBJ_TAG) {
 		o = deref_tag(o, refname, 0);
 		if (o)
-			packet_write(1, "%s %s^{}\n", sha1_to_hex(o->sha1), refname);
+			packet_write(1, "%s %s%s^{}\n", sha1_to_hex(o->sha1), refnameprefix, refname);
 	}
 	return 0;
 }
@@ -678,15 +681,24 @@ static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag
 
 static void upload_pack(void)
 {
+	struct strbuf prefix = STRBUF_INIT;
+	unsigned char sha1[20];
+	int flag;
+
+	strbuf_addf(&prefix, "refs/%s", ref_prefix);
 	if (advertise_refs || !stateless_rpc) {
 		reset_timeout();
-		head_ref(send_ref, NULL);
-		for_each_ref(send_ref, NULL);
+		if (resolve_ref(head_path, sha1, 1, &flag))
+			send_ref("HEAD", sha1, flag, "");
+		for_each_ref_in(prefix.buf, send_ref, "refs/");
 		packet_flush(1);
 	} else {
-		head_ref(mark_our_ref, NULL);
-		for_each_ref(mark_our_ref, NULL);
+		if (resolve_ref(head_path, sha1, 1, &flag))
+			mark_our_ref("HEAD", sha1, flag, NULL);
+		for_each_ref_in(prefix.buf, mark_our_ref, NULL);
 	}
+	strbuf_release(&prefix);
+
 	if (advertise_refs)
 		return;
 
@@ -716,6 +728,14 @@ int main(int argc, char **argv)
 			advertise_refs = 1;
 			continue;
 		}
+		if (!prefixcmp(arg, "--head=")) {
+			head_path = arg+7;
+			continue;
+		}
+		if (!prefixcmp(arg, "--ref-prefix=")) {
+			ref_prefix = arg+13;
+			continue;
+		}
 		if (!strcmp(arg, "--stateless-rpc")) {
 			stateless_rpc = 1;
 			continue;
-- 
1.7.5.1

[PATCH v3 2/3] Support virtual repositories in smart http-backend, specified by environment

[Jamey Sharp]

From: Josh Triplett <josh@joshtriplett.org>

Translate the GIT_REF_PREFIX and GIT_HEAD environment variables to the
--ref-prefix= and --head= options to upload-pack and receive-pack.

Add documentation, including a sample Apache configuration snippet.

Commit by Josh Triplett and Jamey Sharp.

Signed-off-by: Josh Triplett <josh@joshtriplett.org>
Signed-off-by: Jamey Sharp <jamey@minilop.net>
---
v2: add patch 2/2 for git-http-backend.
v3: add patch 3/3 with documentation for virtual repositories, and
    incorporated feedback from Jeff King and Junio C Hamano.

 Documentation/git-http-backend.txt |   14 ++++++++++++
 http-backend.c                     |   39 +++++++++++++++++++++++++++++------
 2 files changed, 46 insertions(+), 7 deletions(-)

diff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt
index 277d9e1..4e0b243 100644
--- a/Documentation/git-http-backend.txt
+++ b/Documentation/git-http-backend.txt
@@ -119,6 +119,14 @@ ScriptAliasMatch \
 
 ScriptAlias /git/ /var/www/cgi-bin/gitweb.cgi/
 ----------------------------------------------------------------
++
+To serve multiple virtual repositories from a single storage
+repository:
++
+----------------------------------------------------------------
+SetEnvIf Request_URI "^/git/([^/]*)" GIT_REF_PREFIX=$1/ GIT_HEAD=$1-HEAD
+ScriptAliasMatch ^/git/[^/]*(.*) /usr/libexec/git-core/git-http-backend/storage.git$1
+----------------------------------------------------------------
 
 Accelerated static Apache 2.x::
 	Similar to the above, but Apache can be used to return static
@@ -167,6 +175,12 @@ The GIT_HTTP_EXPORT_ALL environmental variable may be passed to
 'git-http-backend' to bypass the check for the "git-daemon-export-ok"
 file in each repository before allowing export of that repository.
 
+The GIT_REF_PREFIX and GIT_HEAD environment variables allow
+http-backend to support multiple virtual repositories within a single
+storage repository.  If set, 'git http-backend' will operate on
+'refs/${GIT_REF_PREFIX}' rather than 'refs/', and '$GIT_HEAD' rather
+than HEAD.
+
 The backend process sets GIT_COMMITTER_NAME to '$REMOTE_USER' and
 GIT_COMMITTER_EMAIL to '$\{REMOTE_USER}@http.$\{REMOTE_ADDR\}',
 ensuring that any reflogs created by 'git-receive-pack' contain some
diff --git a/http-backend.c b/http-backend.c
index 8501504..3d9e3b1 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -315,16 +315,23 @@ done:
 	close(out);
 }
 
-static void run_service(const char **argv)
+#define RUN_SERVICE_LAST_EXTRA_ARG 2
+#define RUN_SERVICE_EXTRA_ARGS NULL, NULL, NULL
+
+static void run_service(const char *argv0, const char **argv)
 {
 	const char *encoding = getenv("HTTP_CONTENT_ENCODING");
 	const char *user = getenv("REMOTE_USER");
 	const char *host = getenv("REMOTE_ADDR");
+	char *ref_prefix = getenv("GIT_REF_PREFIX");
+	char *head = getenv("GIT_HEAD");
 	char *env[3];
 	struct strbuf buf = STRBUF_INIT;
 	int gzipped_request = 0;
 	struct child_process cld;
 
+	argv += RUN_SERVICE_LAST_EXTRA_ARG;
+
 	if (encoding && !strcmp(encoding, "gzip"))
 		gzipped_request = 1;
 	else if (encoding && !strcmp(encoding, "x-gzip"))
@@ -343,6 +350,24 @@ static void run_service(const char **argv)
 	env[1] = strbuf_detach(&buf, NULL);
 	env[2] = NULL;
 
+	if (ref_prefix && !*ref_prefix)
+		ref_prefix = NULL;
+	if (ref_prefix) {
+		strbuf_addf(&buf, "--ref-prefix=%s", ref_prefix);
+		ref_prefix = strbuf_detach(&buf, NULL);
+		*argv-- = ref_prefix;
+	}
+
+	if (head && !*head)
+		head = NULL;
+	if (head) {
+		strbuf_addf(&buf, "--head=%s", head);
+		head = strbuf_detach(&buf, NULL);
+		*argv-- = head;
+	}
+
+	*argv = argv0;
+
 	memset(&cld, 0, sizeof(cld));
 	cld.argv = argv;
 	cld.env = (const char *const *)env;
@@ -362,6 +387,8 @@ static void run_service(const char **argv)
 		exit(1);
 	free(env[0]);
 	free(env[1]);
+	free(ref_prefix);
+	free(head);
 	strbuf_release(&buf);
 }
 
@@ -391,7 +418,7 @@ static void get_info_refs(char *arg)
 	hdr_nocache();
 
 	if (service_name) {
-		const char *argv[] = {NULL /* service name */,
+		const char *argv[] = {RUN_SERVICE_EXTRA_ARGS,
 			"--stateless-rpc", "--advertise-refs",
 			".", NULL};
 		struct rpc_service *svc = select_service(service_name);
@@ -404,8 +431,7 @@ static void get_info_refs(char *arg)
 		packet_write(1, "# service=git-%s\n", svc->name);
 		packet_flush(1);
 
-		argv[0] = svc->name;
-		run_service(argv);
+		run_service(svc->name, argv);
 
 	} else {
 		select_getanyfile();
@@ -462,7 +488,7 @@ static void check_content_type(const char *accepted_type)
 
 static void service_rpc(char *service_name)
 {
-	const char *argv[] = {NULL, "--stateless-rpc", ".", NULL};
+	const char *argv[] = {RUN_SERVICE_EXTRA_ARGS, "--stateless-rpc", ".", NULL};
 	struct rpc_service *svc = select_service(service_name);
 	struct strbuf buf = STRBUF_INIT;
 
@@ -478,8 +504,7 @@ static void service_rpc(char *service_name)
 
 	end_headers();
 
-	argv[0] = svc->name;
-	run_service(argv);
+	run_service(svc->name, argv);
 	strbuf_release(&buf);
 }
 
-- 
1.7.5.1

[PATCH v3 3/3] Add documentation for virtual repositories

[Jamey Sharp]

From: Josh Triplett <josh@joshtriplett.org>

Add a new gitvirtual(5) document, and cross-reference it from
git-http-backend(1).

Thanks to Jeff King for providing the material for the SECURITY section
and inspiring the CONVENTIONS section.

Commit by Josh Triplett and Jamey Sharp.

Signed-off-by: Josh Triplett <josh@joshtriplett.org>
Signed-off-by: Jamey Sharp <jamey@minilop.net>
---
v3: add patch 3/3 with documentation for virtual repositories, and
    incorporated feedback from Jeff King and Junio C Hamano.

 Documentation/Makefile                 |    2 +-
 Documentation/git-http-backend.txt     |    4 +-
 Documentation/gitvirtual.txt           |   76 ++++++++++++++++++++++++++++++++
 contrib/completion/git-completion.bash |    2 +-
 4 files changed, 80 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/gitvirtual.txt

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 36989b7..4b4bd2f 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -6,7 +6,7 @@ MAN5_TXT=gitattributes.txt gitignore.txt gitmodules.txt githooks.txt \
 	gitrepository-layout.txt
 MAN7_TXT=gitcli.txt gittutorial.txt gittutorial-2.txt \
 	gitcvs-migration.txt gitcore-tutorial.txt gitglossary.txt \
-	gitdiffcore.txt gitrevisions.txt gitworkflows.txt
+	gitdiffcore.txt gitrevisions.txt gitvirtual.txt gitworkflows.txt
 
 MAN_TXT = $(MAN1_TXT) $(MAN5_TXT) $(MAN7_TXT)
 MAN_XML=$(patsubst %.txt,%.xml,$(MAN_TXT))
diff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt
index 4e0b243..4a7c42b 100644
--- a/Documentation/git-http-backend.txt
+++ b/Documentation/git-http-backend.txt
@@ -120,8 +120,8 @@ ScriptAliasMatch \
 ScriptAlias /git/ /var/www/cgi-bin/gitweb.cgi/
 ----------------------------------------------------------------
 +
-To serve multiple virtual repositories from a single storage
-repository:
+To serve multiple virtual repositories (linkgit:gitvirtual[7]) from a
+single storage repository:
 +
 ----------------------------------------------------------------
 SetEnvIf Request_URI "^/git/([^/]*)" GIT_REF_PREFIX=$1/ GIT_HEAD=$1-HEAD
diff --git a/Documentation/gitvirtual.txt b/Documentation/gitvirtual.txt
new file mode 100644
index 0000000..ed5a4c5
--- /dev/null
+++ b/Documentation/gitvirtual.txt
@@ -0,0 +1,76 @@
+gitvirtual(7)
+=============
+
+NAME
+----
+gitvirtual - Git virtual repositories
+
+DESCRIPTION
+-----------
+
+Given many repositories with copies of the same objects (such as
+branches of the same source), sharing a common object store will avoid
+duplication.  Alternates provide a single baseline, but don't handle
+ongoing activity in the various repositories.  Furthermore, operations
+such as linkgit:git-gc[1] need to know about all of the refs.
+
+Git supports storing multiple virtual repositories within the object
+store and references of a single underlying repository.  The underlying
+repository stores the objects for all of the virtual repositories, and
+includes all the refs and heads of the virtual repositories using
+prefixed names.
+
+linkgit:git-upload-pack[1] and linkgit:git-receive-pack[1] rewrite the
+names of refs and heads as specified by the --ref-prefix and --head
+options.  For instance, --ref-prefix=`virtual/reponame/` will use
++pass:[refs/virtual/reponame/heads/*]+ and
++pass:[refs/virtual/reponame/tags/*]+.  git-upload-pack and
+git-receive-pack will ignore any references that do not match the
+specified prefix.
+
+These options implement the underlying mechanism for virtual
+repositories.  The smart HTTP server, linkgit:git-http-backend[1], can
+accept a ref prefix and HEAD path via environment variables, and pass
+them to the backend programs.  For a simple local test, you can use
+git-remote-ext:
+
+----------
+git clone ext::'git %s --ref-prefix=virtual/prefix/ --head=prefix-HEAD /tmp/prefixed.git'
+----------
+
+CONVENTIONS
+-----------
+
+The --ref-prefix and --head options provide quite a bit of flexibility
+in organizing the refs of virtual repositories within those of the
+underlying repository.  In the absence of a strong reason to do
+otherwise, consider following these conventions:
+
+--ref-prefix=`virtual/reponame/`::
+	This puts refs under `refs/virtual/reponame/`, which avoids a
+	namespace conflict between `reponame` and built-in ref
+	directories such as `heads` and `tags`.
+
+--head=`virtual-HEAD/reponame`::
+	This puts HEADs under `virtual-HEAD/` to avoid namespace
+	conflicts with top-level filenames in a git repository.
+
+SECURITY
+--------
+
+Anyone with access to any virtual repository can potentially access
+objects from any other virtual repository stored in the same underlying
+repository.  You can't directly say "give me object ABCD" if you don't
+have a ref to it, but you can do some other sneaky things like:
+
+. Claiming to push ABCD, at which point the server will optimize out the
+  need for you to actually send it. Now you have a ref to ABCD and can
+  fetch it (claiming not to have it, of course).
+
+. Requesting other refs, claiming that you have ABCD, at which point the
+  server may generate deltas against ABCD.
+
+None of this causes a problem if you only host public repositories, or
+if everyone who may read one virtual repo may also read everything in
+every other virtual repo (for instance, if everyone in an organization
+has read permission to every repository).
diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index bb8d7d0..a58d27e 100755
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -1469,7 +1469,7 @@ _git_help ()
 		attributes cli core-tutorial cvs-migration
 		diffcore gitk glossary hooks ignore modules
 		repository-layout tutorial tutorial-2
-		workflows
+		virtual workflows
 		"
 }
 
-- 
1.7.5.1

Re: [PATCH v3 1/3] Support multiple virtual repositories with a single object store and refs

[Junio C Hamano]

Jamey Sharp <jamey@minilop.net> writes:

> From: Josh Triplett <josh@joshtriplett.org>
>
> Given many repositories with copies of the same objects (such as branches of
> the same source), sharing a common object store will avoid duplication.
> Alternates provide a single baseline, but don't handle ongoing activity in the
> various repositories.  Furthermore, operations such as git-gc need to know
> about all of the refs.
>
> Git supports storing multiple virtual repositories within the object store and
> references of a single underlying repository.  The underlying repository
> stores the objects for all of the virtual repositories, and includes all the
> refs and heads of the virtual repositories using prefixed names.

I do not see anything changed up to this point since the previous
round... sent a wrong patch?

In any case, I _think_ what you are trying to say is:

 - Implemented in the most naïve way, you can host multiple instances of
   related projects, but that is wasteful; their object stores will have
   duplicated objects without sharing. (This is the crucial part missing
   from your description that confused me when trying to _guess_ what
   problem you are trying to solve in the first place).

 - You _could_ use alternates mechanism to alleviate that problem, but it
   has issues, e.g. gc needs to be aware of other repositories (This is in
   your first paragraph).

 - Instead, we could store a single, large, repository and carve out its
   refs namespaces into multiple hierarchies, to make it look as if there
   are multiple repositories. (The first sentence of the second paragraph
   also confused me, as you said "Git supports storing multiple ..." in
   present tense).

One thing you would want to be careful with is what to do with the HEAD
symrefs, which should appear to read "ref: refs/heads/<some-branch>" from
the point of view of the clients that are under the illusion that they are
interacting with one specific repository among others, while for the
purpose of gc and things in the huge single repository they should be
pointing at something like "refs/hosted-1-project/heads/<that-branch>",
but other than that, after a lot of guesswork, the problem you are trying
to solve seems clearer to me.

But please do not make me guess.

Re: [PATCH v3 3/3] Add documentation for virtual repositories

[Jeff King]

On Tue, May 24, 2011 at 05:46:32PM -0700, Jamey Sharp wrote:

>  Documentation/Makefile                 |    2 +-
>  Documentation/git-http-backend.txt     |    4 +-
>  Documentation/gitvirtual.txt           |   76 ++++++++++++++++++++++++++++++++
>  contrib/completion/git-completion.bash |    2 +-

Maybe it would make sense to mention your new options to upload-pack and
receive-pack in their manpages; the description can be short, but refer
the user to gitvirtual.

> +Given many repositories with copies of the same objects (such as
> +branches of the same source), sharing a common object store will avoid
> +duplication.  Alternates provide a single baseline, but don't handle
> +ongoing activity in the various repositories.  Furthermore, operations
> +such as linkgit:git-gc[1] need to know about all of the refs.

It's not quite true that alternates provide only a single baseline. They
can be updated and objects consolidated over time (e.g., with a nightly
repack). The problem is that they require management to do so (this is
also a benefit, if you want a sharing policy besides "all repos have all
objects").

> +linkgit:git-upload-pack[1] and linkgit:git-receive-pack[1] rewrite the
> +names of refs and heads as specified by the --ref-prefix and --head
> +options.  For instance, --ref-prefix=`virtual/reponame/` will use
> ++pass:[refs/virtual/reponame/heads/*]+ and
> ++pass:[refs/virtual/reponame/tags/*]+.  git-upload-pack and
> +git-receive-pack will ignore any references that do not match the
> +specified prefix.

Thinking on the whole idea a bit more, is there a reason to restrict
this to upload-pack and receive-pack? Sure, they are the most obvious
places to use it for hosting, but might I not want to be able to do:

  cd /path/to/mega-repository.git
  git --ref-prefix=virtual/repo1 log master

to do server-side scripting inside the virtual repos (or more likely,
setting GIT_REF_PREFIX at the top of your script).

> +The --ref-prefix and --head options provide quite a bit of flexibility
> +in organizing the refs of virtual repositories within those of the
> +underlying repository.  In the absence of a strong reason to do
> +otherwise, consider following these conventions:
> +
> +--ref-prefix=`virtual/reponame/`::
> +	This puts refs under `refs/virtual/reponame/`, which avoids a
> +	namespace conflict between `reponame` and built-in ref
> +	directories such as `heads` and `tags`.
> +
> +--head=`virtual-HEAD/reponame`::
> +	This puts HEADs under `virtual-HEAD/` to avoid namespace
> +	conflicts with top-level filenames in a git repository.

I'm curious if you have a use for this much flexibility. In particular,
why do the HEAD and refs prefixes need the ability to be separate? Also,
what about other non-HEAD top-level refs? IOW, a true "virtual
repository" to me would just be:

  GIT_REF_PREFIX=refs/virtual/repo1

and then _every_ ref resolution would just prefix that, whether it was
in refs/ or not. So you would have:

  .git/refs/virtual/repo1/HEAD
  .git/refs/virtual/repo1/refs/heads/master
  .git/refs/virtual/repo1/refs/tags/v1.0

and so on. And this fits in with the idea of it not just being an
upload-pack and receive-pack thing. I could do:

  GIT_REF_PREFIX=refs/virtual/repo1; export GIT_REF_PREFIX
  git fetch some-remote

and it would write to .git/refs/virtual/repo1/FETCH_HEAD.

So the virtual repository is basically just a "chroot" of the ref
namespace. And it's dirt simple to implement, because you do the
translation at the refs.c layer.

> +SECURITY
> +--------
> +
> +Anyone with access to any virtual repository can potentially access
> +objects from any other virtual repository stored in the same underlying
> +repository.  You can't directly say "give me object ABCD" if you don't
> +have a ref to it, but you can do some other sneaky things like:
> +
> +. Claiming to push ABCD, at which point the server will optimize out the
> +  need for you to actually send it. Now you have a ref to ABCD and can
> +  fetch it (claiming not to have it, of course).
> +
> +. Requesting other refs, claiming that you have ABCD, at which point the
> +  server may generate deltas against ABCD.
> +
> +None of this causes a problem if you only host public repositories, or
> +if everyone who may read one virtual repo may also read everything in
> +every other virtual repo (for instance, if everyone in an organization
> +has read permission to every repository).

Well, this text is obviously correct and written by a very smart person.
;)

You might want to mention that if you do need to handle these security
concerns, then the alternates route, even though it creates more
management headache, is going to be more flexible with respect to which
objects are shared.

In fact, given what I said at the very top of the email, I wonder if the
documentation would be better structured as "here are two methods for
sharing objects, here are reasons why you might choose one or the other,
and here is how to use each".

-Peff

Re: [PATCH v3 1/3] Support multiple virtual repositories with a single object store and refs

[Jamey Sharp]

[-- Attachment #1: Type: text/plain, Size: 4045 bytes --]

On Tue, May 24, 2011 at 06:21:00PM -0700, Junio C Hamano wrote:
> Jamey Sharp <jamey@minilop.net> writes:
> 
> > From: Josh Triplett <josh@joshtriplett.org>
> >
> > Given many repositories with copies of the same objects (such as branches of
> > the same source), sharing a common object store will avoid duplication.
> > Alternates provide a single baseline, but don't handle ongoing activity in the
> > various repositories.  Furthermore, operations such as git-gc need to know
> > about all of the refs.
> >
> > Git supports storing multiple virtual repositories within the object store and
> > references of a single underlying repository.  The underlying repository
> > stores the objects for all of the virtual repositories, and includes all the
> > refs and heads of the virtual repositories using prefixed names.
> 
> I do not see anything changed up to this point since the previous
> round... sent a wrong patch?

Apparently so. I watched Josh fix up that commit message, and then I
don't know where it went.

> In any case, I _think_ what you are trying to say is:
> 
>  - Implemented in the most naïve way, you can host multiple instances of
>    related projects, but that is wasteful; their object stores will have
>    duplicated objects without sharing. (This is the crucial part missing
>    from your description that confused me when trying to _guess_ what
>    problem you are trying to solve in the first place).
> 
>  - You _could_ use alternates mechanism to alleviate that problem, but it
>    has issues, e.g. gc needs to be aware of other repositories (This is in
>    your first paragraph).
> 
>  - Instead, we could store a single, large, repository and carve out its
>    refs namespaces into multiple hierarchies, to make it look as if there
>    are multiple repositories. (The first sentence of the second paragraph
>    also confused me, as you said "Git supports storing multiple ..." in
>    present tense).

Yes. I hope you won't mind if we blatantly steal this description. :-)

> One thing you would want to be careful with is what to do with the HEAD
> symrefs, which should appear to read "ref: refs/heads/<some-branch>" from
> the point of view of the clients that are under the illusion that they are
> interacting with one specific repository among others, while for the
> purpose of gc and things in the huge single repository they should be
> pointing at something like "refs/hosted-1-project/heads/<that-branch>",

As far as I can tell, that isn't true. Judging by the pack-protocol
documentation, my reading of the implementation, and the results of some
tests I ran, symrefs are resolved to hashes before being sent over the
wire, and then HEAD is magically re-inferred back into a symref on the
other end.

(This has the odd property that if you create a repository containing
two branches with identical heads, then clone that repository, the
clone's origin/HEAD will point to a randomly-selected one of the two
branches. Tested in version 1.7.4.4, and seems to be a necessary
consequence of the protocol design.)

As a result, symrefs only need to be valid in the underlying repository;
there's no mapping needed for the protocol. However, you probably do
want a different HEAD for each virtual repository, which is why we added
the --head option.

We didn't actually think about impact of these virtual HEADs on gc. As
long as they're all symrefs, they can't matter for gc, right? The head
they reference is already a suitable gc root. If the virtual HEADs do
need to participate in gc, then I guess we should update the conventions
documentation to recommend that they live somewhere under refs/.

> but other than that, after a lot of guesswork, the problem you are trying
> to solve seems clearer to me.
> 
> But please do not make me guess.

Indeed. We'll get that right next round, honest this time. :-/

Now that you have the problem statement down, is the proposed solution
acceptable for merge?

Jamey

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCH v3 3/3] Add documentation for virtual repositories

[Shawn Pearce]

On Wed, May 25, 2011 at 09:07, Jeff King <peff@peff.net> wrote:
>
> Thinking on the whole idea a bit more, is there a reason to restrict
> this to upload-pack and receive-pack? Sure, they are the most obvious
> places to use it for hosting, but might I not want to be able to do:

No, there isn't. I had the same impression reading this series... that
doing it in upload-pack receive-pack was wrong, but I couldn't put my
finger on why. I think you did (below), so thank you.

>  cd /path/to/mega-repository.git
>  git --ref-prefix=virtual/repo1 log master
>
> to do server-side scripting inside the virtual repos (or more likely,
> setting GIT_REF_PREFIX at the top of your script).
>
>> +The --ref-prefix and --head options provide quite a bit of flexibility
...
> I'm curious if you have a use for this much flexibility. In particular,
> why do the HEAD and refs prefixes need the ability to be separate? Also,
> what about other non-HEAD top-level refs? IOW, a true "virtual
> repository" to me would just be:
>
>  GIT_REF_PREFIX=refs/virtual/repo1
>
> and then _every_ ref resolution would just prefix that, whether it was
> in refs/ or not. So you would have:
>
>  .git/refs/virtual/repo1/HEAD
>  .git/refs/virtual/repo1/refs/heads/master
>  .git/refs/virtual/repo1/refs/tags/v1.0
>
> and so on. And this fits in with the idea of it not just being an
> upload-pack and receive-pack thing. I could do:
>
>  GIT_REF_PREFIX=refs/virtual/repo1; export GIT_REF_PREFIX
>  git fetch some-remote

+1 * 1000. This should be a single environment variable / top level
option. HEAD should also use the GIT_REF_PREFIX, like any other ref.

FETCH_HEAD and MERGE_HEAD probably should as well if GIT_REF_PREFIX is
set, however these are going to be a bit harder to move. Not all tools
that read them are GIT_REF_PREFIX aware, or go through C code that can
be modified to be GIT_REF_PREFIX aware. (git-gui and EGit, I'm talking
about you here!)  They can obviously be fixed, but until then using a
working directory with GIT_REF_PREFIX set will be slightly
interesting.

> So the virtual repository is basically just a "chroot" of the ref
> namespace. And it's dirt simple to implement, because you do the
> translation at the refs.c layer.

Yes, exactly.

-- 
Shawn.

Re: [PATCH v3 3/3] Add documentation for virtual repositories

[Jeff King]

On Wed, May 25, 2011 at 10:01:20AM -0700, Shawn O. Pearce wrote:

> > and so on. And this fits in with the idea of it not just being an
> > upload-pack and receive-pack thing. I could do:
> >
> >  GIT_REF_PREFIX=refs/virtual/repo1; export GIT_REF_PREFIX
> >  git fetch some-remote
> 
> +1 * 1000. This should be a single environment variable / top level
> option. HEAD should also use the GIT_REF_PREFIX, like any other ref.

Good, I am glad I'm not the only one thinking this. :)

> FETCH_HEAD and MERGE_HEAD probably should as well if GIT_REF_PREFIX is
> set, however these are going to be a bit harder to move. Not all tools
> that read them are GIT_REF_PREFIX aware, or go through C code that can
> be modified to be GIT_REF_PREFIX aware. (git-gui and EGit, I'm talking
> about you here!)  They can obviously be fixed, but until then using a
> working directory with GIT_REF_PREFIX set will be slightly
> interesting.

Yeah, most scripts these days will need to go through the C programs to
handle packed-refs. But the top-level pseudo-refs are a bit more
magical. That is perhaps why they split HEAD and refs handling in the
original patch. Still, I don't think it's insurmountable.

> > So the virtual repository is basically just a "chroot" of the ref
> > namespace. And it's dirt simple to implement, because you do the
> > translation at the refs.c layer.
> 
> Yes, exactly.

Like chroots, there is a sticky point with symbolic links. What should
"refs/virtual/repo1/HEAD" have in it? Either:

  ref: refs/virtual/repo1/refs/heads/master

or

  ref: refs/heads/master

?

If the former, then we will have to make sure the ref is inside our
prefix, and strip it out. If the latter, then you will get different
results for:

  git show refs/virtual/repo1/HEAD

versus

  GIT_REF_PREFIX=refs/virtual/repo1 git show HEAD

which I think is a bad thing.

-Peff

Re: [PATCH v3 3/3] Add documentation for virtual repositories

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I'm curious if you have a use for this much flexibility. In particular,
> why do the HEAD and refs prefixes need the ability to be separate?...
> ...
> So the virtual repository is basically just a "chroot" of the ref
> namespace.

Yes, that makes much more sense than singling HEAD out.

Re: [PATCH v3 3/3] Add documentation for virtual repositories

[Josh Triplett]

On Wed, May 25, 2011 at 12:07:08PM -0400, Jeff King wrote:
> On Tue, May 24, 2011 at 05:46:32PM -0700, Jamey Sharp wrote:
> 
> >  Documentation/Makefile                 |    2 +-
> >  Documentation/git-http-backend.txt     |    4 +-
> >  Documentation/gitvirtual.txt           |   76 ++++++++++++++++++++++++++++++++
> >  contrib/completion/git-completion.bash |    2 +-
> 
> Maybe it would make sense to mention your new options to upload-pack and
> receive-pack in their manpages; the description can be short, but refer
> the user to gitvirtual.

Fair enough.  We'll go ahead and document them (in patch 1/3 with a
reference to gitvirtual added in patch 3/3), and avoid making the pile
of undocumented upload-pack and receive-pack options larger. :)

> > +Given many repositories with copies of the same objects (such as
> > +branches of the same source), sharing a common object store will avoid
> > +duplication.  Alternates provide a single baseline, but don't handle
> > +ongoing activity in the various repositories.  Furthermore, operations
> > +such as linkgit:git-gc[1] need to know about all of the refs.
> 
> It's not quite true that alternates provide only a single baseline. They
> can be updated and objects consolidated over time (e.g., with a nightly
> repack). The problem is that they require management to do so (this is
> also a benefit, if you want a sharing policy besides "all repos have all
> objects").

True enough.  We wanted something that automatically worked without
background maintenance, but alternates can help if you keep moving
common objects to the alternate repository.

> > +linkgit:git-upload-pack[1] and linkgit:git-receive-pack[1] rewrite the
> > +names of refs and heads as specified by the --ref-prefix and --head
> > +options.  For instance, --ref-prefix=`virtual/reponame/` will use
> > ++pass:[refs/virtual/reponame/heads/*]+ and
> > ++pass:[refs/virtual/reponame/tags/*]+.  git-upload-pack and
> > +git-receive-pack will ignore any references that do not match the
> > +specified prefix.
> 
> Thinking on the whole idea a bit more, is there a reason to restrict
> this to upload-pack and receive-pack? Sure, they are the most obvious
> places to use it for hosting, but might I not want to be able to do:
> 
>   cd /path/to/mega-repository.git
>   git --ref-prefix=virtual/repo1 log master
> 
> to do server-side scripting inside the virtual repos (or more likely,
> setting GIT_REF_PREFIX at the top of your script).

Many git commands will need special handling for this, though.  For
instance, gc needs to know about all refs, not just a prefix of refs;
otherwise it will break the repository.  Or, for an example within a
single command, the checks for updating a currently-checked-out ref in a
repository need to use the repository's HEAD, not the virtual HEAD.
And similarly, git checkout with a ref-prefix set would construct a
repository where HEAD doesn't match the workdir.

Having this handled "transparently" for all git commands seems likely to
run into this kind of corner case, where parts of a git command run
correctly with ref-prefix but other parts or other invoked git commands
must not run with ref-prefix.

I do agree that some other git programs could learn to use ref-prefix,
and it makes sense to move the functionality into refs.c as a general
mechanism for those programs to use.  However, I don't think it makes
sense to transparently make all git programs use ref-prefix without
checking them individually to see if it makes sense.

> > +The --ref-prefix and --head options provide quite a bit of flexibility
> > +in organizing the refs of virtual repositories within those of the
> > +underlying repository.  In the absence of a strong reason to do
> > +otherwise, consider following these conventions:
> > +
> > +--ref-prefix=`virtual/reponame/`::
> > +	This puts refs under `refs/virtual/reponame/`, which avoids a
> > +	namespace conflict between `reponame` and built-in ref
> > +	directories such as `heads` and `tags`.
> > +
> > +--head=`virtual-HEAD/reponame`::
> > +	This puts HEADs under `virtual-HEAD/` to avoid namespace
> > +	conflicts with top-level filenames in a git repository.
> 
> I'm curious if you have a use for this much flexibility. In particular,
> why do the HEAD and refs prefixes need the ability to be separate? Also,
> what about other non-HEAD top-level refs? IOW, a true "virtual
> repository" to me would just be:
> 
>   GIT_REF_PREFIX=refs/virtual/repo1
> 
> and then _every_ ref resolution would just prefix that, whether it was
> in refs/ or not. So you would have:
> 
>   .git/refs/virtual/repo1/HEAD
>   .git/refs/virtual/repo1/refs/heads/master
>   .git/refs/virtual/repo1/refs/tags/v1.0

Ah, *now* I see what you meant by including the repeated "refs/", and
using that to allow putting HEAD in the same namespace makes sense.

We don't actually need the flexibility of putting HEAD in a different
place, and this layout makes sense, so we can change the ref-prefix
mechanism to drop the separate --head entirely.

> > +SECURITY
> > +--------
> > +
> > +Anyone with access to any virtual repository can potentially access
> > +objects from any other virtual repository stored in the same underlying
> > +repository.  You can't directly say "give me object ABCD" if you don't
> > +have a ref to it, but you can do some other sneaky things like:
> > +
> > +. Claiming to push ABCD, at which point the server will optimize out the
> > +  need for you to actually send it. Now you have a ref to ABCD and can
> > +  fetch it (claiming not to have it, of course).
> > +
> > +. Requesting other refs, claiming that you have ABCD, at which point the
> > +  server may generate deltas against ABCD.
> > +
> > +None of this causes a problem if you only host public repositories, or
> > +if everyone who may read one virtual repo may also read everything in
> > +every other virtual repo (for instance, if everyone in an organization
> > +has read permission to every repository).
> 
> Well, this text is obviously correct and written by a very smart person.
> ;)
> 
> You might want to mention that if you do need to handle these security
> concerns, then the alternates route, even though it creates more
> management headache, is going to be more flexible with respect to which
> objects are shared.
> 
> In fact, given what I said at the very top of the email, I wonder if the
> documentation would be better structured as "here are two methods for
> sharing objects, here are reasons why you might choose one or the other,
> and here is how to use each".

I think it makes sense to reference alternates in the gitvirtual page, but
I don't think it makes sense to put the full documentation for both in
the same page.

- Josh Triplett

Re: [PATCH v3 3/3] Add documentation for virtual repositories

[Shawn Pearce]

On Wed, May 25, 2011 at 10:10, Jeff King <peff@peff.net> wrote:
> Like chroots, there is a sticky point with symbolic links. What should
> "refs/virtual/repo1/HEAD" have in it? Either:
>
>  ref: refs/virtual/repo1/refs/heads/master
...
> If the former, then we will have to make sure the ref is inside our
> prefix, and strip it out.

Yes. And if its outside of our chroot space, we have to hide the
symbolic reference as though it is broken. This is fairly simple since
symbolic references are supposed to be "absolute", we can do a quick
prefix test and either strip the prefix on read (or add it on write),
or pretend like the reference is broken and hide it if its outside the
prefix.

> If the latter, then you will get different
> results for:
>
>  git show refs/virtual/repo1/HEAD
>
> versus
>
>  GIT_REF_PREFIX=refs/virtual/repo1 git show HEAD
>
> which I think is a bad thing.

Yes, I agree, this would be bad. Hence we should store the real path
in the symbolic reference, but strip/add the prefix when
GIT_REF_PREFIX is set.

-- 
Shawn.