* Adding more namespace support to git @ 2016-08-20 19:07 Richard 2016-08-21 2:05 ` Josh Triplett 2016-08-24 17:49 ` Jeff King 0 siblings, 2 replies; 7+ messages in thread From: Richard @ 2016-08-20 19:07 UTC (permalink / raw) To: git I work on a git server called gitano. We've been using and recommending cgit for the web UI. I've been working on adding git namespace support to both, so that we can separate administrative branches from code branches. Because git is not namespace aware for anything but git-upload-pack and git-receive-pack, I've had to implement namespace parsing in cgit for listing branches, showing logs, displaying notes and commit decorations. It might be more useful if this support was added to git itself, so other git servers could make use of it so there's less duplicated code. I think the way to do this would be to make the low-level ref reading functions, read_raw_ref, for_each_reflog_ent*, reflog_exists etc., interpret the ref they are passed as being relative to the current git namespace. Since when upload-pack and receive-pack run hooks they leave GIT_NAMESPACE set there are hook scripts that expect that the current namespace is ignored, so commands that now want to be namespace aware would have to opt-in. I tried adding a flag to the functions with the intention of changing everything to conditionally set the flag and adding wrapper functions to provide the old ABIs, but quickly abandoned that after seeing how much change would be required. Is anyone more familiar with the codebase able to recommend how I might make it work, and would there any interest in accepting such patches if I got it to work? Please reply-all in response, I am not subscribed to the mailing list. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Adding more namespace support to git 2016-08-20 19:07 Adding more namespace support to git Richard @ 2016-08-21 2:05 ` Josh Triplett 2016-08-21 11:30 ` Richard 2016-08-24 17:49 ` Jeff King 1 sibling, 1 reply; 7+ messages in thread From: Josh Triplett @ 2016-08-21 2:05 UTC (permalink / raw) To: Richard; +Cc: git On Sat, Aug 20, 2016 at 08:07:00PM +0100, Richard wrote: > I work on a git server called gitano. > We've been using and recommending cgit for the web UI. > > I've been working on adding git namespace support to both, > so that we can separate administrative branches from code branches. > > Because git is not namespace aware for anything but git-upload-pack > and git-receive-pack, > I've had to implement namespace parsing in cgit > for listing branches, showing logs, displaying notes and commit decorations. > It might be more useful if this support was added to git itself, > so other git servers could make use of it so there's less duplicated code. > > I think the way to do this would be to make the low-level ref reading functions, > read_raw_ref, for_each_reflog_ent*, reflog_exists etc., > interpret the ref they are passed as being relative to the current git > namespace. > > Since when upload-pack and receive-pack run hooks they leave GIT_NAMESPACE set > there are hook scripts that expect that the current namespace is ignored, > so commands that now want to be namespace aware would have to opt-in. That seems really unfortunate. While at the time we wanted to start with namespace support in upload-pack and receive-pack (and http-backend) because those would allow using it as a server-side storage format, I don't think we realized that leaving GIT_NAMESPACE in the hook environment would completely prevent other git commands from automatically handling namespaces. And conversely, we can't just have upload-pack and receive-pack start removing it from the hook environment, because a hook might expect to read the current namespace from it (and then run git commands that the hook expects will ignore it). (This also affects libgit2; I recently added a function to libgit2 to interpret various git environment variables, including GIT_NAMESPACE. If git commands can't just use that automatically, that'll need to change too, to avoid unexpected behavior in hooks.) For that matter, someone could run "GIT_NAMESPACE=foo git push remotename branchname" or "GIT_NAMESPACE=foo git clone remotename", and based on the current behavior, they'd expect to have the namespace apply to the remote end, but not the local end. Unfortunately, I think at this point, GIT_NAMESPACE has to exclusively refer to the namespace for the remote end, to avoid breakage. Which means any automatic pervasive support for namespaces on the local side would need to use a different mechanism. (In addition to applying to ref enumeration, this would also need to apply to the local end of refspecs.) And this new mechanism would need to not affect the remote end, to allow remapping the local end while accessing an un-namespaced (or differently namespaced) remote. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Adding more namespace support to git 2016-08-21 2:05 ` Josh Triplett @ 2016-08-21 11:30 ` Richard 2016-08-21 14:07 ` Josh Triplett 0 siblings, 1 reply; 7+ messages in thread From: Richard @ 2016-08-21 11:30 UTC (permalink / raw) To: Josh Triplett; +Cc: git On 21 August 2016 at 03:05, Josh Triplett <josh@joshtriplett.org> wrote: > On Sat, Aug 20, 2016 at 08:07:00PM +0100, Richard wrote: >> Since when upload-pack and receive-pack run hooks they leave GIT_NAMESPACE set >> there are hook scripts that expect that the current namespace is ignored, >> so commands that now want to be namespace aware would have to opt-in. > > That seems really unfortunate. While at the time we wanted to start > with namespace support in upload-pack and receive-pack (and > http-backend) because those would allow using it as a server-side > storage format, I don't think we realized that leaving GIT_NAMESPACE in > the hook environment would completely prevent other git commands from > automatically handling namespaces. > > And conversely, we can't just have upload-pack and receive-pack start > removing it from the hook environment, because a hook might expect to > read the current namespace from it (and then run git commands that the > hook expects will ignore it). This is exactly what I've had to do for my proof of concept https://git.gitano.org.uk/cgit.git/commit/?h=richardmaw/namespaces&id=379124469a8a13208f976eb816375b00901ae77f > For that matter, someone could run "GIT_NAMESPACE=foo git push > remotename branchname" or > "GIT_NAMESPACE=foo git clone remotename", and based on the current > behavior, they'd expect to have the namespace apply to the remote end, > but not the local end. I'm fairly sure this isn't the case, at least from what I've tried. At one point it appeared to be working, but that was just because it started the upload-pack as a subprocess, which inherited the GIT_NAMESPACE environment variable rather than being passed it. I think this is why the test suite always sets up a remote with the ext:: helper so it can set --namespace=foo in the command. This is one of the reasons why I have been working on namespace support in the git server, you have to encode the namespace in the url somehow since it isn't passed through the git protocol. We were thinking of adding ssh://git@server/~username/repo/path.git syntax for letting users have their own private namespace in a repository, and later extending the backend of the git server's repository storage so that other repositories could just be namespaces of a different repository so we could do something like repository forks provided the repositories have the same availability. > Unfortunately, I think at this point, GIT_NAMESPACE has to exclusively > refer to the namespace for the remote end, to avoid breakage. Which > means any automatic pervasive support for namespaces on the local side > would need to use a different mechanism. (In addition to applying to > ref enumeration, this would also need to apply to the local end of > refspecs.) And this new mechanism would need to not affect the remote > end, to allow remapping the local end while accessing an un-namespaced > (or differently namespaced) remote. The problem for hooks is that it is implicitly inherited, so it could work if upload-pack receive-pack and http-backend work with GIT_NAMESPACE set, but everything else that wants to use a namespace has to set --namespace on the command-line. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Adding more namespace support to git 2016-08-21 11:30 ` Richard @ 2016-08-21 14:07 ` Josh Triplett [not found] ` <CAB2Vqoa+1QKD-7zjNUJBuu2qNowgY7n9fLM77zUdzf8aJi4XaQ@mail.gmail.com> 0 siblings, 1 reply; 7+ messages in thread From: Josh Triplett @ 2016-08-21 14:07 UTC (permalink / raw) To: Richard; +Cc: git On Sun, Aug 21, 2016 at 12:30:16PM +0100, Richard wrote: > On 21 August 2016 at 03:05, Josh Triplett <josh@joshtriplett.org> wrote: > > Unfortunately, I think at this point, GIT_NAMESPACE has to exclusively > > refer to the namespace for the remote end, to avoid breakage. Which > > means any automatic pervasive support for namespaces on the local side > > would need to use a different mechanism. (In addition to applying to > > ref enumeration, this would also need to apply to the local end of > > refspecs.) And this new mechanism would need to not affect the remote > > end, to allow remapping the local end while accessing an un-namespaced > > (or differently namespaced) remote. > > The problem for hooks is that it is implicitly inherited, > so it could work if upload-pack receive-pack and http-backend work > with GIT_NAMESPACE set, > but everything else that wants to use a namespace has to set > --namespace on the command-line. I'd like to see it work more automatically than that. Perhaps a separate environment variable to set the client-side namespace? ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <CAB2Vqoa+1QKD-7zjNUJBuu2qNowgY7n9fLM77zUdzf8aJi4XaQ@mail.gmail.com>]
* Re: Adding more namespace support to git [not found] ` <CAB2Vqoa+1QKD-7zjNUJBuu2qNowgY7n9fLM77zUdzf8aJi4XaQ@mail.gmail.com> @ 2016-08-22 19:16 ` Josh Triplett 2016-08-22 19:36 ` Richard 0 siblings, 1 reply; 7+ messages in thread From: Josh Triplett @ 2016-08-22 19:16 UTC (permalink / raw) To: Richard; +Cc: git On Mon, Aug 22, 2016 at 07:36:31PM +0100, Richard wrote: > On 21 Aug 2016 15:07, "Josh Triplett" <josh@joshtriplett.org> wrote: > > I'd like to see it work more automatically than that. Perhaps a > > separate environment variable to set the client-side namespace? > > How about a config option? That could be set globally, per repository, in > the environment or on the command line. That might work, though you wouldn't normally want to set it globally or per-repository (since it affects access to a repository and you'd typically want to use multiple different values or it wouldn't have much point). - Josh Triplett ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Adding more namespace support to git 2016-08-22 19:16 ` Josh Triplett @ 2016-08-22 19:36 ` Richard 0 siblings, 0 replies; 7+ messages in thread From: Richard @ 2016-08-22 19:36 UTC (permalink / raw) To: Josh Triplett; +Cc: git On 22 August 2016 at 20:16, Josh Triplett <josh@joshtriplett.org> wrote: > On Mon, Aug 22, 2016 at 07:36:31PM +0100, Richard wrote: >> On 21 Aug 2016 15:07, "Josh Triplett" <josh@joshtriplett.org> wrote: >> > I'd like to see it work more automatically than that. Perhaps a >> > separate environment variable to set the client-side namespace? >> >> How about a config option? That could be set globally, per repository, in >> the environment or on the command line. > > That might work, though you wouldn't normally want to set it globally or > per-repository (since it affects access to a repository and you'd > typically want to use multiple different values or it wouldn't have much > point). Globally is a bit contrived, but could be used to keep the top-level namespace clean so you might opt to default to fetching into a namespace called "main" so that if you need to temporarily fetch into a different namespace it wouldn't be problematic. Perhaps it's a kernel tree from a vendor with a messy branch naming scheme so you don't want to fetch it into your primary namespace and make it difficult to find your branches, but you don't know which of their branches you need until you've got them all. So you fetch into the different namespace rather than a fresh clone to avoid re-fetching everything (numerous alternative solutions exist) Then once you've found out which branch you need, you make a note, switch back to the "main" namespace and re-fetch just that branch. A per repository default namespace could also be useful if an upstream repository has multiple namespaces (code vs documentation maybe) you could fetch them all and then switch between them when you need to work on different parts, and if it's config rather than an environment variable it will persist between shell sessions easier. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Adding more namespace support to git 2016-08-20 19:07 Adding more namespace support to git Richard 2016-08-21 2:05 ` Josh Triplett @ 2016-08-24 17:49 ` Jeff King 1 sibling, 0 replies; 7+ messages in thread From: Jeff King @ 2016-08-24 17:49 UTC (permalink / raw) To: Richard; +Cc: Michael Haggerty, git On Sat, Aug 20, 2016 at 08:07:00PM +0100, Richard wrote: > Because git is not namespace aware for anything but git-upload-pack > and git-receive-pack, I've had to implement namespace parsing in cgit > for listing branches, showing logs, displaying notes and commit > decorations. It might be more useful if this support was added to git > itself, so other git servers could make use of it so there's less > duplicated code. > > I think the way to do this would be to make the low-level ref reading > functions, read_raw_ref, for_each_reflog_ent*, reflog_exists etc., > interpret the ref they are passed as being relative to the current git > namespace. At GitHub, we store many forks for a single repository, and we considered using namespaces for our storage strategy. But like you, we ran into the problem that you they only work for certain operations. :) Our solution is to use separate repositories, each with their own ref storage, but pointing to a shared object store. It works, but there are a lot of gotchas and performance issues with migrating objects around, running repacks. Michael Haggerty (cc'd) has picked up the pluggable ref-backend work started by others, and I know has some ideas on doing namespaces at that level. Basically, the concept of "namespaces" should be able to plug in between the actual storage backend and the rest of the git code. Git code wouldn't have to care whether the namespace plugin was in use, and the namespace plugin wouldn't have to care which storage backend was in use (it would just silently translate "refs/heads/foo" into "refs/namespaces/123/heads/foo", and vice versa). That's a very "complete" solution in the sense that the git code does not know about the namespaces, and cannot even access refs outside of it. But I think in general it would do what you want. Most operations would run in a certain namespace (i.e., pretend nothing outside of that namespace exists, for fetches, diffs, etc), and others would want to look at the whole namespace (e.g., repacking, pruning). I don't know of any operations that want to see both views in the same process. -Peff ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-08-24 17:51 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-08-20 19:07 Adding more namespace support to git Richard 2016-08-21 2:05 ` Josh Triplett 2016-08-21 11:30 ` Richard 2016-08-21 14:07 ` Josh Triplett [not found] ` <CAB2Vqoa+1QKD-7zjNUJBuu2qNowgY7n9fLM77zUdzf8aJi4XaQ@mail.gmail.com> 2016-08-22 19:16 ` Josh Triplett 2016-08-22 19:36 ` Richard 2016-08-24 17:49 ` Jeff King
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.