* [RFC][PATCH 0/3] Different views on a repository
@ 2010-02-24 16:41 Andreas Gruenbacher
2010-02-24 15:33 ` [PATCH 1/3] receive-pack: Two small code cleanups Andreas Gruenbacher
` (3 more replies)
0 siblings, 4 replies; 16+ messages in thread
From: Andreas Gruenbacher @ 2010-02-24 16:41 UTC (permalink / raw)
To: git
Hello,
we have a use case with groups of repositories which share lots of
objects, but which are logically independent. There is no strict
hierarchy between the repositories, the development modl is arbitrary.
The alternates mechanism for sharig objects between repositories won't
work.
The best idea I came up with so far to solve this was to keep everything
in the same repository on the server. Then, to keep the logically
independent repositories separate, directories are used below refs/heads
and refs/tags. Receive-pack and upload-pack are modified to hide this
directory structure from clients so that repositories will continue to
look "normal" to users. For example, the following structure on the
server:
refs/heads/one/master
refs/tags/one/tag1
refs/heads/two/master
refs/heads/two/branch2
would appear as two independent repositories to different clients:
refs/heads/master
refs/tags/tag1
and:
refs/heads/master
refs/heads/branch2
The following three patches implement this. What do you guys think --
does the basic idea and implementation look sensible, or am I
overlooking a way to solve this kind of problem with other means?
Thanks!
receive-pack: Two small code cleanups
Different views on a repository
Different views on a repository: HEAD mapping
Documentation/git-receive-pack.txt | 9 ++++-
Documentation/git-upload-pack.txt | 10 +++++-
builtin-receive-pack.c | 32 ++++++++++++++---
refs.c | 65 ++++++++++++++++++++++++++++++++++--
refs.h | 4 ++
upload-pack.c | 20 ++++++++++-
6 files changed, 127 insertions(+), 13 deletions(-)
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH 1/3] receive-pack: Two small code cleanups 2010-02-24 16:41 [RFC][PATCH 0/3] Different views on a repository Andreas Gruenbacher @ 2010-02-24 15:33 ` Andreas Gruenbacher 2010-02-24 15:57 ` [PATCH 2/3] Different views on a repository Andreas Gruenbacher 2010-02-24 17:29 ` [PATCH 1/3] receive-pack: Two small code cleanups Shawn O. Pearce 2010-02-25 20:13 ` [RFC][PATCH 0/3] Different views on a repository James Pickens ` (2 subsequent siblings) 3 siblings, 2 replies; 16+ messages in thread From: Andreas Gruenbacher @ 2010-02-24 15:33 UTC (permalink / raw) To: git Rename show_ref()'s path parameter to refname. In read_head_info(), lines may have trailing capability strings. Throw away such strings after evaluation; they are not needed in the command structs. Signed-off-by: Andreas Gruenbacher <agruen@suse.de> --- builtin-receive-pack.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/builtin-receive-pack.c b/builtin-receive-pack.c index 0559fcc..77cbc2a 100644 --- a/builtin-receive-pack.c +++ b/builtin-receive-pack.c @@ -105,13 +105,13 @@ static int receive_pack_config(const char *var, const char *value, void *cb) return git_default_config(var, value, cb); } -static int show_ref(const char *path, const unsigned char *sha1, int flag, void *cb_data) +static int show_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data) { if (sent_capabilities) - packet_write(1, "%s %s\n", sha1_to_hex(sha1), path); + packet_write(1, "%s %s\n", sha1_to_hex(sha1), refname); else packet_write(1, "%s %s%c%s%s\n", - sha1_to_hex(sha1), path, 0, + sha1_to_hex(sha1), refname, 0, " report-status delete-refs side-band-64k", prefer_ofs_delta ? " ofs-delta" : ""); sent_capabilities = 1; @@ -524,7 +524,7 @@ static void read_head_info(void) static char line[1000]; unsigned char old_sha1[20], new_sha1[20]; struct command *cmd; - char *refname; + const char *refname; int len, reflen; len = packet_read_line(0, line, sizeof(line)); @@ -548,10 +548,10 @@ static void read_head_info(void) if (strstr(refname + reflen + 1, "side-band-64k")) use_sideband = LARGE_PACKET_MAX; } - cmd = xmalloc(sizeof(struct command) + len - 80); + cmd = xmalloc(sizeof(struct command) + reflen + 1); hashcpy(cmd->old_sha1, old_sha1); hashcpy(cmd->new_sha1, new_sha1); - memcpy(cmd->ref_name, line + 82, len - 81); + memcpy(cmd->ref_name, refname, reflen + 1); cmd->error_string = NULL; cmd->next = NULL; *p = cmd; -- 1.6.6.243.gff6d2 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 2/3] Different views on a repository 2010-02-24 15:33 ` [PATCH 1/3] receive-pack: Two small code cleanups Andreas Gruenbacher @ 2010-02-24 15:57 ` Andreas Gruenbacher 2010-02-24 16:14 ` [PATCH 3/3] Different views on a repository: HEAD mapping Andreas Gruenbacher ` (2 more replies) 2010-02-24 17:29 ` [PATCH 1/3] receive-pack: Two small code cleanups Shawn O. Pearce 1 sibling, 3 replies; 16+ messages in thread From: Andreas Gruenbacher @ 2010-02-24 15:57 UTC (permalink / raw) To: git Add --view options in upload-pack and receive-pack so that a repository on the server side can be made to look like several independent repositories on the client side. This is implemented by transforming ref names: for example, with --view=one/, refs/heads/one/master on the server will look like refs/heads/master to the client, refs/tags/one/v1 will look like refs/tags/v1, etc. This allows to transparently share repositories on the server which have a lot of objects in common without complicating things for the client, and without breaking garbage collection. Signed-off-by: Andreas Gruenbacher <agruen@suse.de> --- Documentation/git-receive-pack.txt | 8 +++++- Documentation/git-upload-pack.txt | 9 ++++++- builtin-receive-pack.c | 20 ++++++++++++++++ refs.c | 44 ++++++++++++++++++++++++++++++++++++ refs.h | 3 ++ upload-pack.c | 11 +++++++++ 6 files changed, 93 insertions(+), 2 deletions(-) diff --git a/Documentation/git-receive-pack.txt b/Documentation/git-receive-pack.txt index 2790eeb..09d7d0c 100644 --- a/Documentation/git-receive-pack.txt +++ b/Documentation/git-receive-pack.txt @@ -8,7 +8,7 @@ git-receive-pack - Receive what is pushed into the repository SYNOPSIS -------- -'git-receive-pack' <directory> +'git-receive-pack' [--view=<prefix>] <directory> DESCRIPTION ----------- @@ -34,6 +34,12 @@ are not fast-forwards. OPTIONS ------- +--view=<prefix>:: + Prepend <prefix> to all ref names. For example, --view=one/ will + turn refs/tags/v1 into refs/tags/one/v1 on the receiving end. Together + with the --view option of linkgit:git-upload-pack[1], this allows to + make one respository look like multiple independent repositories. + <directory>:: The repository to sync into. diff --git a/Documentation/git-upload-pack.txt b/Documentation/git-upload-pack.txt index 71ca4ef..0eee0ba 100644 --- a/Documentation/git-upload-pack.txt +++ b/Documentation/git-upload-pack.txt @@ -8,7 +8,7 @@ git-upload-pack - Send objects packed back to git-fetch-pack SYNOPSIS -------- -'git-upload-pack' [--strict] [--timeout=<n>] <directory> +'git-upload-pack' [--strict] [--timeout=<n>] [--view=<prefix>] <directory> DESCRIPTION ----------- @@ -30,6 +30,13 @@ OPTIONS --timeout=<n>:: Interrupt transfer after <n> seconds of inactivity. +--view=<prefix>:: + Only upload refs which start with <prefix>, and hide <prefix> from the + remote side. For example, --view=one/ will skip refs/heads/master + and turn refs/tags/one/v1 into refs/tags/v1. Together with the --view + option of linkgit:git-receive-pack, this allows to make one respository + look like multiple independent repositories. + <directory>:: The repository to sync from. diff --git a/builtin-receive-pack.c b/builtin-receive-pack.c index 77cbc2a..44d7055 100644 --- a/builtin-receive-pack.c +++ b/builtin-receive-pack.c @@ -32,6 +32,7 @@ static int use_sideband; static int prefer_ofs_delta = 1; static int auto_update_server_info; static int auto_gc = 1; +static const char *view; static const char *head_name; static int sent_capabilities; @@ -107,6 +108,12 @@ static int receive_pack_config(const char *var, const char *value, void *cb) static int show_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data) { + if (view) { + refname = ref_to_view(refname, view); + if (!refname) + return 0; + } + if (sent_capabilities) packet_write(1, "%s %s\n", sha1_to_hex(sha1), refname); else @@ -548,6 +555,15 @@ static void read_head_info(void) if (strstr(refname + reflen + 1, "side-band-64k")) use_sideband = LARGE_PACKET_MAX; } + if (view) { + const char *r; + + r = view_to_ref(refname, view); + if (r) { + refname = r; + reflen = strlen(refname); + } + } cmd = xmalloc(sizeof(struct command) + reflen + 1); hashcpy(cmd->old_sha1, old_sha1); hashcpy(cmd->new_sha1, new_sha1); @@ -736,6 +752,10 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix) stateless_rpc = 1; continue; } + if (!prefixcmp(arg, "--view=")) { + view = arg + 7; + continue; + } usage(receive_pack_usage); } diff --git a/refs.c b/refs.c index f3fcbe0..b1f3951 100644 --- a/refs.c +++ b/refs.c @@ -1829,3 +1829,47 @@ char *shorten_unambiguous_ref(const char *ref, int strict) free(short_name); return xstrdup(ref); } + +const char *ref_to_view(const char *refname, const char *view) +{ + static char *buffer; + int prefix_len, view_len, suffix_len; + const char *r, *suffix; + + if (prefixcmp(refname, "refs/")) + return NULL; + r = strchr(refname + 5, '/'); + if (!r) + return NULL; + r++; + view_len = strlen(view); + if (strncmp(r, view, view_len)) + return NULL; + suffix = r + view_len; + prefix_len = r - refname; + suffix_len = strlen(suffix); + buffer = xrealloc(buffer, prefix_len + suffix_len + 1); + sprintf(buffer, "%.*s%s", prefix_len, refname, suffix); + return buffer; +} + +const char *view_to_ref(const char *refname, const char *view) +{ + static char *buffer; + int prefix_len, view_len, suffix_len; + const char *r, *suffix; + + view_len = strlen(view); + if (prefixcmp(refname, "refs/")) + return NULL; + r = strchr(refname + 5, '/'); + if (!r) + return NULL; + r++; + prefix_len = r - refname; + suffix = r + view_len; + suffix_len = strlen(suffix); + buffer = xrealloc(buffer, prefix_len + view_len + suffix_len + 1); + sprintf(buffer, "%.*s%s%s", prefix_len, refname, view, suffix); + return buffer; +} diff --git a/refs.h b/refs.h index f7648b9..390e812 100644 --- a/refs.h +++ b/refs.h @@ -98,4 +98,7 @@ int update_ref(const char *action, const char *refname, const unsigned char *sha1, const unsigned char *oldval, int flags, enum action_on_err onerr); +extern const char *ref_to_view(const char *refname, const char *view); +extern const char *view_to_ref(const char *refname, const char *view); + #endif /* REFS_H */ diff --git a/upload-pack.c b/upload-pack.c index dc464d7..bc72471 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -34,6 +34,7 @@ static struct object_array have_obj; static struct object_array want_obj; static struct object_array extra_edge_obj; static unsigned int timeout; +static const char *view; /* 0 for no sideband, * otherwise maximum packet size (up to 65520 bytes). */ @@ -629,6 +630,12 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo if (!o) die("git upload-pack: cannot find object %s:", sha1_to_hex(sha1)); + if (view) { + refname = ref_to_view(refname, view); + if (!refname) + return 0; + } + if (capabilities) packet_write(1, "%s %s%c%s\n", sha1_to_hex(sha1), refname, 0, capabilities); @@ -711,6 +718,10 @@ int main(int argc, char **argv) daemon_mode = 1; continue; } + if (!prefixcmp(arg, "--view=")) { + view = arg + 7; + continue; + } if (!strcmp(arg, "--")) { i++; break; -- 1.6.6.243.gff6d2 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 3/3] Different views on a repository: HEAD mapping 2010-02-24 15:57 ` [PATCH 2/3] Different views on a repository Andreas Gruenbacher @ 2010-02-24 16:14 ` Andreas Gruenbacher 2010-02-24 17:42 ` [PATCH 2/3] Different views on a repository Shawn O. Pearce 2010-02-25 9:01 ` Michael J Gruber 2 siblings, 0 replies; 16+ messages in thread From: Andreas Gruenbacher @ 2010-02-24 16:14 UTC (permalink / raw) To: git The HEAD ref is not located under .git/refs/heads/, so the trivial view mapping doesn't work. Fix this by making refs/heads/<view>HEAD appear as HEAD on the client when --view=<view> is used in upload-pack and receive-pack. Signed-off-by: Andreas Gruenbacher <agruen@suse.de> --- Documentation/git-receive-pack.txt | 7 ++++--- Documentation/git-upload-pack.txt | 9 +++++---- refs.c | 21 ++++++++++++++++++--- refs.h | 1 + upload-pack.c | 9 +++++++-- 5 files changed, 35 insertions(+), 12 deletions(-) diff --git a/Documentation/git-receive-pack.txt b/Documentation/git-receive-pack.txt index 09d7d0c..07e0159 100644 --- a/Documentation/git-receive-pack.txt +++ b/Documentation/git-receive-pack.txt @@ -36,9 +36,10 @@ OPTIONS ------- --view=<prefix>:: Prepend <prefix> to all ref names. For example, --view=one/ will - turn refs/tags/v1 into refs/tags/one/v1 on the receiving end. Together - with the --view option of linkgit:git-upload-pack[1], this allows to - make one respository look like multiple independent repositories. + turn refs/tags/v1 into refs/tags/one/v1 and HEAD into refs/heads/one/HEAD + on the receiving end. Together with the --view option of + linkgit:git-upload-pack[1], this allows to make one respository look like + multiple independent repositories. <directory>:: The repository to sync into. diff --git a/Documentation/git-upload-pack.txt b/Documentation/git-upload-pack.txt index 0eee0ba..1e8b76b 100644 --- a/Documentation/git-upload-pack.txt +++ b/Documentation/git-upload-pack.txt @@ -32,10 +32,11 @@ OPTIONS --view=<prefix>:: Only upload refs which start with <prefix>, and hide <prefix> from the - remote side. For example, --view=one/ will skip refs/heads/master - and turn refs/tags/one/v1 into refs/tags/v1. Together with the --view - option of linkgit:git-receive-pack, this allows to make one respository - look like multiple independent repositories. + remote side. For example, --view=one/ will skip refs/heads/master, + turn refs/tags/one/v1 into refs/tags/v1, and refs/heads/one/HEAD into + HEAD. Together with the --view option of linkgit:git-receive-pack, + this allows to make one respository look like multiple independent + repositories. <directory>:: The repository to sync from. diff --git a/refs.c b/refs.c index b1f3951..77a6267 100644 --- a/refs.c +++ b/refs.c @@ -650,16 +650,21 @@ end_each: return retval; } -int head_ref(each_ref_fn fn, void *cb_data) +int one_ref(each_ref_fn fn, void *cb_data, const char *refname) { unsigned char sha1[20]; int flag; - if (resolve_ref("HEAD", sha1, 1, &flag)) - return fn("HEAD", sha1, flag, cb_data); + if (resolve_ref(refname, sha1, 1, &flag)) + return fn(refname, sha1, flag, cb_data); return 0; } +int head_ref(each_ref_fn fn, void *cb_data) +{ + return one_ref(fn, cb_data, "HEAD"); +} + int for_each_ref(each_ref_fn fn, void *cb_data) { return do_for_each_ref("refs/", fn, 0, 0, cb_data); @@ -1846,6 +1851,10 @@ const char *ref_to_view(const char *refname, const char *view) if (strncmp(r, view, view_len)) return NULL; suffix = r + view_len; + if (!strncmp(refname + 5, "heads/", 6) && + !strcmp(suffix, "HEAD")) + return "HEAD"; + prefix_len = r - refname; suffix_len = strlen(suffix); buffer = xrealloc(buffer, prefix_len + suffix_len + 1); @@ -1860,6 +1869,12 @@ const char *view_to_ref(const char *refname, const char *view) const char *r, *suffix; view_len = strlen(view); + if (!strcmp(refname, "HEAD")) { + buffer = xrealloc(buffer, view_len + 16); + sprintf(buffer, "refs/heads/%sHEAD", view); + return buffer; + } + if (prefixcmp(refname, "refs/")) return NULL; r = strchr(refname + 5, '/'); diff --git a/refs.h b/refs.h index 390e812..addcc2d 100644 --- a/refs.h +++ b/refs.h @@ -18,6 +18,7 @@ struct ref_lock { * and returns the value */ typedef int each_ref_fn(const char *refname, const unsigned char *sha1, int flags, void *cb_data); +extern int one_ref(each_ref_fn, void *, const char *); extern int head_ref(each_ref_fn, void *); extern int for_each_ref(each_ref_fn, void *); extern int for_each_ref_in(const char *, each_ref_fn, void *); diff --git a/upload-pack.c b/upload-pack.c index bc72471..b3bf20f 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -668,13 +668,18 @@ static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag static void upload_pack(void) { + const char *head = "HEAD"; + + if (view) + head = view_to_ref(head, view); + if (advertise_refs || !stateless_rpc) { reset_timeout(); - head_ref(send_ref, NULL); + one_ref(send_ref, NULL, head); for_each_ref(send_ref, NULL); packet_flush(1); } else { - head_ref(mark_our_ref, NULL); + one_ref(mark_our_ref, NULL, head); for_each_ref(mark_our_ref, NULL); } if (advertise_refs) -- 1.6.6.243.gff6d2 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH 2/3] Different views on a repository 2010-02-24 15:57 ` [PATCH 2/3] Different views on a repository Andreas Gruenbacher 2010-02-24 16:14 ` [PATCH 3/3] Different views on a repository: HEAD mapping Andreas Gruenbacher @ 2010-02-24 17:42 ` Shawn O. Pearce 2010-02-25 9:01 ` Michael J Gruber 2 siblings, 0 replies; 16+ messages in thread From: Shawn O. Pearce @ 2010-02-24 17:42 UTC (permalink / raw) To: Andreas Gruenbacher; +Cc: git Andreas Gruenbacher <agruen@suse.de> wrote: > Add --view options in upload-pack and receive-pack so that a repository > on the server side can be made to look like several independent > repositories on the client side. Before saying this is good... I'd like to know how a repository owner is supposed to set these options on the user started invocations of other remote side program. Right now, I don't see how this is too different from just doing the following on a client: git init git remote add origin URL git config remote.origin.fetch refs/heads/one/*:refs/remotes/origin/* and therefore shouldn't just be handled on the *client* side of the connection, as part of the remote setup and push matching refs rules. (Of course, the push matching ref logic is messy too... adding yet more into that pile might also be ugly.) > +const char *view_to_ref(const char *refname, const char *view) > +{ > + static char *buffer; ... > + buffer = xrealloc(buffer, prefix_len + view_len + suffix_len + 1); > + sprintf(buffer, "%.*s%s%s", prefix_len, refname, view, suffix); > + return buffer; I'd rather not use a static buffer like this. Why not alloc and let the caller free? Or have the caller pass in a strbuf you populate for them? -- Shawn. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 2/3] Different views on a repository 2010-02-24 15:57 ` [PATCH 2/3] Different views on a repository Andreas Gruenbacher 2010-02-24 16:14 ` [PATCH 3/3] Different views on a repository: HEAD mapping Andreas Gruenbacher 2010-02-24 17:42 ` [PATCH 2/3] Different views on a repository Shawn O. Pearce @ 2010-02-25 9:01 ` Michael J Gruber 2010-02-25 9:25 ` Andreas Gruenbacher 2 siblings, 1 reply; 16+ messages in thread From: Michael J Gruber @ 2010-02-25 9:01 UTC (permalink / raw) To: Andreas Gruenbacher; +Cc: git Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57: > Add --view options in upload-pack and receive-pack so that a repository > on the server side can be made to look like several independent > repositories on the client side. > > This is implemented by transforming ref names: for example, with > --view=one/, refs/heads/one/master on the server will look like > refs/heads/master to the client, refs/tags/one/v1 will look like > refs/tags/v1, etc. > > This allows to transparently share repositories on the server which > have a lot of objects in common without complicating things for the > client, and without breaking garbage collection. Just from this description, I can't see why the same can't be done with appropriate refspecs. (A helper for doing that would be more welcome, of course.) Maybe a few tests and documentation (i.e. examples, not just the option description) would clear this up? Michael ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 2/3] Different views on a repository 2010-02-25 9:01 ` Michael J Gruber @ 2010-02-25 9:25 ` Andreas Gruenbacher 2010-02-25 12:30 ` Michael J Gruber 0 siblings, 1 reply; 16+ messages in thread From: Andreas Gruenbacher @ 2010-02-25 9:25 UTC (permalink / raw) To: Michael J Gruber; +Cc: git On Thursday 25 February 2010 10:01:43 Michael J Gruber wrote: > Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57: > > Add --view options in upload-pack and receive-pack so that a repository > > on the server side can be made to look like several independent > > repositories on the client side. > > > > This is implemented by transforming ref names: for example, with > > --view=one/, refs/heads/one/master on the server will look like > > refs/heads/master to the client, refs/tags/one/v1 will look like > > refs/tags/v1, etc. > > > > This allows to transparently share repositories on the server which > > have a lot of objects in common without complicating things for the > > client, and without breaking garbage collection. > > Just from this description, I can't see why the same can't be done with > appropriate refspecs. (A helper for doing that would be more welcome, of > course.) You mean on the client side? The problem then is that a simple "git clone" would not do the right thing anymore; you would still expose server-side implementation details to clients. Clients shouldn't have to bother with this added complexity. (They might not even have access to some of the views.) When you do the mapping server-side, you can split or merge repositories as needed without the clients even noticing. > Maybe a few tests and documentation (i.e. examples, not just the option > description) would clear this up? Indeed, I should add some more background info. Thanks, Andreas ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 2/3] Different views on a repository 2010-02-25 9:25 ` Andreas Gruenbacher @ 2010-02-25 12:30 ` Michael J Gruber 2010-02-25 14:35 ` Andreas Gruenbacher 0 siblings, 1 reply; 16+ messages in thread From: Michael J Gruber @ 2010-02-25 12:30 UTC (permalink / raw) To: Andreas Gruenbacher; +Cc: git Andreas Gruenbacher venit, vidit, dixit 25.02.2010 10:25: > On Thursday 25 February 2010 10:01:43 Michael J Gruber wrote: >> Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57: >>> Add --view options in upload-pack and receive-pack so that a repository >>> on the server side can be made to look like several independent >>> repositories on the client side. >>> >>> This is implemented by transforming ref names: for example, with >>> --view=one/, refs/heads/one/master on the server will look like >>> refs/heads/master to the client, refs/tags/one/v1 will look like >>> refs/tags/v1, etc. >>> >>> This allows to transparently share repositories on the server which >>> have a lot of objects in common without complicating things for the >>> client, and without breaking garbage collection. >> >> Just from this description, I can't see why the same can't be done with >> appropriate refspecs. (A helper for doing that would be more welcome, of >> course.) > > You mean on the client side? The problem then is that a simple "git clone" > would not do the right thing anymore; you would still expose server-side > implementation details to clients. Clients shouldn't have to bother with this > added complexity. (They might not even have access to some of the views.) When > you do the mapping server-side, you can split or merge repositories as needed > without the clients even noticing. But the client has to request a specific view, doesn't it? You have to tell all clients "don't just clone, use...", where the "..." don't seem to be part of the series yet. [I could see 0/3 on gmane only now, by the way.] I just can't help the impression that this is a use case which does not need a new feature, at least not upload/receive-pack wise. It's more a matter of ensuring that all clients use a specific configuration (which you would have to with your patch as well, AFAICT), and this more general issue is creeping up again and again, with no agreeable solution so far. Cheers, Michael ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 2/3] Different views on a repository 2010-02-25 12:30 ` Michael J Gruber @ 2010-02-25 14:35 ` Andreas Gruenbacher 2010-02-25 17:28 ` Junio C Hamano 0 siblings, 1 reply; 16+ messages in thread From: Andreas Gruenbacher @ 2010-02-25 14:35 UTC (permalink / raw) To: Michael J Gruber; +Cc: git On Thursday 25 February 2010 13:30:24 Michael J Gruber wrote: > Andreas Gruenbacher venit, vidit, dixit 25.02.2010 10:25: > > On Thursday 25 February 2010 10:01:43 Michael J Gruber wrote: > >> Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57: > >>> Add --view options in upload-pack and receive-pack so that a repository > >>> on the server side can be made to look like several independent > >>> repositories on the client side. > >>> > >>> This is implemented by transforming ref names: for example, with > >>> --view=one/, refs/heads/one/master on the server will look like > >>> refs/heads/master to the client, refs/tags/one/v1 will look like > >>> refs/tags/v1, etc. > >>> > >>> This allows to transparently share repositories on the server which > >>> have a lot of objects in common without complicating things for the > >>> client, and without breaking garbage collection. > >> > >> Just from this description, I can't see why the same can't be done with > >> appropriate refspecs. (A helper for doing that would be more welcome, of > >> course.) > > > > You mean on the client side? The problem then is that a simple "git > > clone" would not do the right thing anymore; you would still expose > > server-side implementation details to clients. Clients shouldn't have to > > bother with this added complexity. (They might not even have access to > > some of the views.) When you do the mapping server-side, you can split or > > merge repositories as needed without the clients even noticing. > > But the client has to request a specific view, doesn't it? No, it's a server side thing. The git commands affected are upload-pack and receice-pack, and those run on the remote end of a fetch. For example, the client would ask the server for repository /foo/one or /foo/two, and the server would map that to different views of /bar/shared: when the client asks the server to run "git-upload-pack /foo/one", the server runs "git-upload-pack --view=one/ /bar/shared" instead. This is relatively easy to set up over ssh using a simple script; for direct git access, a small wrapper daemon would be needed. I'm not sure how this could be hacked into http access, but it doesn't seem all that hard, either. > I just can't help the impression that this is a use case which does not > need a new feature, at least not upload/receive-pack wise. Still, even with the additional explanation above? > It's more a matter of ensuring that all clients use a specific configuration > (which you would have to with your patch as well, AFAICT), and this more > general issue is creeping up again and again, with no agreeable solution > so far. Well that's another problem which we indeed also have for enabling things like local consistency checks and merge drivers. I don't have a good answer here :) Thanks, Andreas ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 2/3] Different views on a repository 2010-02-25 14:35 ` Andreas Gruenbacher @ 2010-02-25 17:28 ` Junio C Hamano 2010-02-26 0:45 ` Andreas Gruenbacher 0 siblings, 1 reply; 16+ messages in thread From: Junio C Hamano @ 2010-02-25 17:28 UTC (permalink / raw) To: Andreas Gruenbacher; +Cc: Michael J Gruber, git Andreas Gruenbacher <agruen@suse.de> writes: > No, it's a server side thing. If it were a server side thing, then I would expect no change to send/receive pack. Instead your clients will access distinct URL as if they are different repositories. git clone git://example.com/pub/scm/git/A git push example.com:/pub/scm/git/B master git pull http://example.com/pub/scm/git/C They should not have to care that the server is cheating to save disk space, and they should be able to access your server with Git v1.6.0. Instead, the server side would: - have separate repositories, A, B and C, as normal repositories; - these repositories share their object stores by having their .git/objects pointing at a shared location via a symlink; - on the server side, gc/prune/fsck will have to be updated so that when the object store of a repository (say A) is shared with something else, they will consider refs in other repositories (B and C) also as the root of traversal. So if this were a server side solution, I would expect the series would add: - a way to set up a shared object store; - a way to maintain a list of backlinks to repositories that share an object store; - a way to create a new repository that shares the object store (e.g. create a symlink to the shared store instead of having its own .git/objects/, and add itself to the list of backlinks for the shared object store); - a way to retire an existing such repository (rm -rf and remove itself from the list of backlinks); - update gc/prune/fsck to honor such a list of backlinks. This would help a "forks" setup commonly seen at places like repo.or.cz and github.com among others. One thing that is missing from the above handwaving outline that your "different views" offers is a "consolidated view", a pseudo-repository that allows you to see refs from individual real (from the client's and project participant's point of view) repositories as if they are in individual subhierarchies of the ref namespace. I however suspect that you didn't want such a view in the first place if there weren't issues around reachability. In other words, I suspect that you invented it merely as one possible solution to the reachability issue, and it was not your goal to have such a consolidated view by itself. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 2/3] Different views on a repository 2010-02-25 17:28 ` Junio C Hamano @ 2010-02-26 0:45 ` Andreas Gruenbacher 2010-02-26 21:35 ` Andreas Gruenbacher 0 siblings, 1 reply; 16+ messages in thread From: Andreas Gruenbacher @ 2010-02-26 0:45 UTC (permalink / raw) To: Junio C Hamano; +Cc: Michael J Gruber, git On Thursday 25 February 2010 18:28:46 Junio C Hamano wrote: > Andreas Gruenbacher <agruen@suse.de> writes: > > No, it's a server side thing. > > If it were a server side thing, then I would expect no change to > send/receive pack. > > Instead your clients will access distinct URL as if they are different > repositories. > > git clone git://example.com/pub/scm/git/A > git push example.com:/pub/scm/git/B master > git pull http://example.com/pub/scm/git/C > > They should not have to care that the server is cheating to save disk > space, and they should be able to access your server with Git v1.6.0. > > Instead, the server side would: > > - have separate repositories, A, B and C, as normal repositories; > > - these repositories share their object stores by having their > .git/objects pointing at a shared location via a symlink; > > - on the server side, gc/prune/fsck will have to be updated so that when > the object store of a repository (say A) is shared with something else, > they will consider refs in other repositories (B and C) also as the > root of traversal. I was proposing to change receive-pack and upload-pack, both of which are running on the server; there was no mention of send-pack. What I've proposed is not a complete solution, but it should suffice to show the idea. Your alternative proposal would also solve my problem, in a different way. With either approach, what looks like separate repositories A, B, and C to clients looks like one repository to gc/prune/fsck. > So if this were a server side solution, I would expect the series would > add: > > - a way to set up a shared object store; > > - a way to maintain a list of backlinks to repositories that share an > object store; > > - a way to create a new repository that shares the object store > (e.g. create a symlink to the shared store instead of having its own > .git/objects/, and add itself to the list of backlinks for the shared > object store); > > - a way to retire an existing such repository (rm -rf and remove itself > from the list of backlinks); > > - update gc/prune/fsck to honor such a list of backlinks. > > This would help a "forks" setup commonly seen at places like repo.or.cz > and github.com among others. Yes. I'm don't know how big a problem this is for those kinds of hosters; in our case, it is a big problem. > One thing that is missing from the above handwaving outline that your > "different views" offers is a "consolidated view", a pseudo-repository > that allows you to see refs from individual real (from the client's and > project participant's point of view) repositories as if they are in > individual subhierarchies of the ref namespace. I have been talking about a repository and different subsets or views of that repository; you call the former a consolidated view and the latter a repository. Those are really just two sides of the same coin. > I however suspect that you didn't want such a view in the first place if > there weren't issues around reachability. In other words, I suspect that > you invented it merely as one possible solution to the reachability issue, > and it was not your goal to have such a consolidated view by itself. I'm actually not sure. The "consolidated view" as you put it may be useful all by itself; it would be a proper, self sufficient git repository -- a really nice property. it may be too painful to maintain this view though. When sharing objects across repositories, the worst-case scenario is that something goes wrong with the backlinks. You will eventually lose objects, but it may take a while until it happens and until you notice, with a lot of damage. That's nasty. A combination of the two approaches would be to "link forward" instead of "linking back", so that the consolidated view would maintain itself, with a server repo setup like this: /repos/ABC: objects refs/tags/A/ refs/tags/B/ refs/heads/A/ refs/heads/B/ /repos/A: refs/tags -> /repos/ABC/refs/tags/A/ refs/heads -> /repos/ABC/refs/heads/A/ objects -> /repos/ABC/objects/ /repos/B: refs/tags -> /repos/ABC/refs/tags/B/ refs/heads -> /repos/ABC/refs/heads/B/ objects -> /repos/ABC/objects/ This could be made safe by not doing garbage collection if objects is a symlink instead of a directory. (The ABC repo could be garbage collected as usual.) Am I overlooking anything why this can't work? Thanks, Andreas ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 2/3] Different views on a repository 2010-02-26 0:45 ` Andreas Gruenbacher @ 2010-02-26 21:35 ` Andreas Gruenbacher 0 siblings, 0 replies; 16+ messages in thread From: Andreas Gruenbacher @ 2010-02-26 21:35 UTC (permalink / raw) To: Junio C Hamano; +Cc: Michael J Gruber, git On Friday 26 February 2010 01:45:33 Andreas Gruenbacher wrote: > A combination of the two approaches would be to "link forward" instead of > "linking back", so that the consolidated view would maintain itself, with a > server repo setup like this: > > /repos/ABC: > objects > refs/tags/A/ > refs/tags/B/ > refs/heads/A/ > refs/heads/B/ > > /repos/A: > refs/tags -> /repos/ABC/refs/tags/A/ > refs/heads -> /repos/ABC/refs/heads/A/ > objects -> /repos/ABC/objects/ > > /repos/B: > refs/tags -> /repos/ABC/refs/tags/B/ > refs/heads -> /repos/ABC/refs/heads/B/ > objects -> /repos/ABC/objects/ > > This could be made safe by not doing garbage collection if objects is a > symlink instead of a directory. (The ABC repo could be garbage collected > as usual.) Am I overlooking anything why this can't work? Self reply: reference packing breaks this kind of setup. Crap. Andreas ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 1/3] receive-pack: Two small code cleanups 2010-02-24 15:33 ` [PATCH 1/3] receive-pack: Two small code cleanups Andreas Gruenbacher 2010-02-24 15:57 ` [PATCH 2/3] Different views on a repository Andreas Gruenbacher @ 2010-02-24 17:29 ` Shawn O. Pearce 1 sibling, 0 replies; 16+ messages in thread From: Shawn O. Pearce @ 2010-02-24 17:29 UTC (permalink / raw) To: Andreas Gruenbacher; +Cc: git Andreas Gruenbacher <agruen@suse.de> wrote: > Rename show_ref()'s path parameter to refname. > > In read_head_info(), lines may have trailing capability strings. Throw > away such strings after evaluation; they are not needed in the command > structs. > > Signed-off-by: Andreas Gruenbacher <agruen@suse.de> Acked-by: Shawn O. Pearce <spearce@spearce.org> -- Shawn. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC][PATCH 0/3] Different views on a repository 2010-02-24 16:41 [RFC][PATCH 0/3] Different views on a repository Andreas Gruenbacher 2010-02-24 15:33 ` [PATCH 1/3] receive-pack: Two small code cleanups Andreas Gruenbacher @ 2010-02-25 20:13 ` James Pickens 2010-02-26 4:30 ` Adam Brewster [not found] ` <c376da901002252012s507a6921q922e606bdce4b4fa@mail.gmail.com> 3 siblings, 0 replies; 16+ messages in thread From: James Pickens @ 2010-02-25 20:13 UTC (permalink / raw) To: Andreas Gruenbacher; +Cc: git On Wed, Feb 24, 2010, Andreas Gruenbacher <agruen@suse.de> wrote: > we have a use case with groups of repositories which share lots of > objects, but which are logically independent. There is no strict > hierarchy between the repositories, the development modl is arbitrary. > The alternates mechanism for sharig objects between repositories won't > work. Can you elaborate on why alternates won't work? > The best idea I came up with so far to solve this Solve what? You didn't mention any specific problem. Are you just trying to save disk space by not storing multiple copies of the same objects? James ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC][PATCH 0/3] Different views on a repository 2010-02-24 16:41 [RFC][PATCH 0/3] Different views on a repository Andreas Gruenbacher 2010-02-24 15:33 ` [PATCH 1/3] receive-pack: Two small code cleanups Andreas Gruenbacher 2010-02-25 20:13 ` [RFC][PATCH 0/3] Different views on a repository James Pickens @ 2010-02-26 4:30 ` Adam Brewster [not found] ` <c376da901002252012s507a6921q922e606bdce4b4fa@mail.gmail.com> 3 siblings, 0 replies; 16+ messages in thread From: Adam Brewster @ 2010-02-26 4:30 UTC (permalink / raw) To: Andreas Gruenbacher; +Cc: git On Wed, Feb 24, 2010 at 11:41 AM, Andreas Gruenbacher <agruen@suse.de>wrote: > Hello, > > we have a use case with groups of repositories which share lots of > objects, but which are logically independent. There is no strict > hierarchy between the repositories, the development modl is arbitrary. > The alternates mechanism for sharig objects between repositories won't > work. > I don't know what you're trying to accomplish, but for what it's worth, I do something similar with a couple of shell scripts. My goal was to make bundles (basically thin packs) smaller by taking advantage of files I knew were available on the far side of the air-gap even if they weren't in a the repository I was bundling that particular day. The idea was to push from all of my repositories into a super repository with a fancy (and auto-generated) refspec. The actual code is impenetrable, but reconstructing it in everybody's favorite IDE, gmail, I came up with #!/bin/bash PROJECTS=$HOME/projects SUPER=$HOME/projects/.git-super-repo [[ -d "$SUPER" ]] || \ (mkdir "$SUPER"; git --git-dir="$SUPER" init) for i in $PROJECTS/*/.git; do name=$(basename "$(dirname "$0")") echo "$SUPER/objects" > $i/.git/objects/info/alternates git --git-dir="$i" push -f "$SUPER" "*:refs/$name/*" done git --git-dir="$SUPER" gc --aggressive for i in $PROJECTS/*/.git; do git --git-dir="$i" repack -Ad #unnecessary? git --git-dir="$i" gc --aggressive done Clearly I can lose data if I try to rebase $SUPER or something, but I think it's pretty safe for normal use. In your case, the "projects" are called "views". Adam ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <c376da901002252012s507a6921q922e606bdce4b4fa@mail.gmail.com>]
* Re: [RFC][PATCH 0/3] Different views on a repository [not found] ` <c376da901002252012s507a6921q922e606bdce4b4fa@mail.gmail.com> @ 2010-02-26 12:01 ` Andreas Gruenbacher 0 siblings, 0 replies; 16+ messages in thread From: Andreas Gruenbacher @ 2010-02-26 12:01 UTC (permalink / raw) To: Adam Brewster; +Cc: git On Friday 26 February 2010 05:12:45 Adam Brewster wrote: > The idea was to push from all of my repositories into a super repository > with a fancy (and auto-generated) refspec. The actual code is > impenetrable, but reconstructing it in everybody's favorite IDE, gmail, I > came up with > > #!/bin/bash > PROJECTS=$HOME/projects > SUPER=$HOME/projects/.git-super-repo > > [[ -d "$SUPER" ]] || \ > (mkdir "$SUPER"; git --git-dir="$SUPER" init) > > for i in $PROJECTS/*/.git; do > name=$(basename "$(dirname "$0")") > echo "$SUPER/objects" > $i/.git/objects/info/alternates > git --git-dir="$i" push -f "$SUPER" "*:refs/$name/*" > done > > git --git-dir="$SUPER" gc --aggressive > > for i in $PROJECTS/*/.git; do > git --git-dir="$i" repack -Ad #unnecessary? > git --git-dir="$i" gc --aggressive > done I see, when multiple projects share the same objects, you push them into those projects independently first, and the script will later move them to $SUPER. > Clearly I can lose data if I try to rebase $SUPER or something, but I think > it's pretty safe for normal use. It looks safe unless somebody messes with $SUPER. A lot of repacking will still occur as part of moving stuff to $SUPER, though. I was trying to set things up so that this extra work won't be necessary in the first place. Thanks! Andreas ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2010-02-26 21:35 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-02-24 16:41 [RFC][PATCH 0/3] Different views on a repository Andreas Gruenbacher
2010-02-24 15:33 ` [PATCH 1/3] receive-pack: Two small code cleanups Andreas Gruenbacher
2010-02-24 15:57 ` [PATCH 2/3] Different views on a repository Andreas Gruenbacher
2010-02-24 16:14 ` [PATCH 3/3] Different views on a repository: HEAD mapping Andreas Gruenbacher
2010-02-24 17:42 ` [PATCH 2/3] Different views on a repository Shawn O. Pearce
2010-02-25 9:01 ` Michael J Gruber
2010-02-25 9:25 ` Andreas Gruenbacher
2010-02-25 12:30 ` Michael J Gruber
2010-02-25 14:35 ` Andreas Gruenbacher
2010-02-25 17:28 ` Junio C Hamano
2010-02-26 0:45 ` Andreas Gruenbacher
2010-02-26 21:35 ` Andreas Gruenbacher
2010-02-24 17:29 ` [PATCH 1/3] receive-pack: Two small code cleanups Shawn O. Pearce
2010-02-25 20:13 ` [RFC][PATCH 0/3] Different views on a repository James Pickens
2010-02-26 4:30 ` Adam Brewster
[not found] ` <c376da901002252012s507a6921q922e606bdce4b4fa@mail.gmail.com>
2010-02-26 12:01 ` Andreas Gruenbacher
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.