proper remote ref namespaces

proper remote ref namespaces

[Jacob Keller]

Hello,

Recently there was some discussion about git-notes and how we do not
fetch notes from remotes by default. The big problem with doing so is
because refs/remotes/* hierarchy is only setup for branches (heads),
so we don't have any clean location to put them.

Around the time of git 1.8.0, Johan Herland made a proposal for
remotes to put all their refs in refs/remtoes/*, by moving heads into
refs/remotes/<remote>heads/* [1]

In addition, his proposal was to include remote tags into
refs/remotes/<remote>/tags and also refs/remotes/<remote>/replace and
notes similarly.

During this discussion there was many people who liked the idea, and
others who rejected it. The main rejection reason  was two fold:

(a) tags are "global" per project, so their namespace should be
treated global as it is now.

The proposal's counter to this is that tags aren't guaranteed to be
global, because today two remotes you fetch might have tags that are
the same name with different pointers. This is currently hidden, and
git silently picks the tag it fetched first.

(b) script compatibility, as changing the ref layout  such that new
git can't work with old repository would be bad

the counter to this, is that we make git smart enough to recognize old
remote format, and continue to work with it. Scripts which depend on
this layout will break, but that may not be such a huge concern.

Personally, I think this proposal at least for heads, notes, replace,
and other remote refs we'd like to pull is very useful. I don't
rightly know the answer for tags. The linked discussion below covers
several pages of back and forth between a few people about which
method is best.

I like the idea of simplifying tags and branches and notes and others
to all fetch the same way. local stuff is in refs/heads or refs/notes
and remote stuff is (by default) in refs/remotes/<remote>/tags etc

But it does bring up some discussion as today we "auto follow" tags
into refs/tags, and it can get weird for tags since now "ambiguous"
tags must mean if there are tags of same name which point to different
refs, and we'd need to teach a bunch of logic to the ref lookup code.

I am looking at ways to help git-notes be easier to use, so that we by
default fetch notes, and enable easier merge, since we'd have default
locations to merge from and to. This would make the sharing of notes a
lot easier, which is one of their primary sticking points.. you can't
really share them without *everyone* working to do it the same way you
do. By making a default policy, sharing becomes natural, and users can
easily add *public* notes to commits for things such as bug ids and
other things which are not discovered until after the commit is
created.

In addition, the easy ability to share replaces might also be helpful,
though IMHO not as valuable as git-notes.

I think that the only logical refs layout is
"refs/remotes/<remote>/(heads|tags|notes|replace)"

and adding "refs/remote-notes" and "refs/remote-replace" is not really
a clean solution.

Given that the 1.8.0 proposal mostly died, does anyone have any thoughts now?

The proposal suggested by Johan makes sense to me, and I believe we
can code up logic to make it easy for new git to keep logic of the old
layout.

Personally, I think the best solution is to only store that layout for
a given clone, using a config option that defaults to false, where
new-git sets it to true for all clones. Then, provide a command to
renew remotes-layout that does this if the user wishes. Thus, clones
for the old style will be handled, and new clones would have the new
layout. (ie: no mixing layouts in a single repository).

I'm really not sure if this is the best solution, but seems like the
cleanest solution.

Regards,
Jake

[1] http://thread.gmane.org/gmane.comp.version-control.git/165799/focus=165885

Re: proper remote ref namespaces

[Marc Branchaud]

On 15-08-12 02:43 AM, Jacob Keller wrote:
> Hello,
> 
> Recently there was some discussion about git-notes and how we do not fetch
> notes from remotes by default. The big problem with doing so is because
> refs/remotes/* hierarchy is only setup for branches (heads), so we don't
> have any clean location to put them.
> 
> Around the time of git 1.8.0, Johan Herland made a proposal for remotes to
> put all their refs in refs/remtoes/*, by moving heads into 
> refs/remotes/<remote>heads/* [1]

Thanks for resurrecting this discussion.  I feel this is a fundamental
inconsistency in git's core design.  Not a fatal flaw by any means, but
something that keeps git from reaching its full potential.

> In addition, his proposal was to include remote tags into 
> refs/remotes/<remote>/tags and also refs/remotes/<remote>/replace and 
> notes similarly.
> 
> During this discussion there was many people who liked the idea, and 
> others who rejected it. The main rejection reason  was two fold:
> 
> (a) tags are "global" per project, so their namespace should be treated
> global as it is now.
> 
> The proposal's counter to this is that tags aren't guaranteed to be 
> global, because today two remotes you fetch might have tags that are the
> same name with different pointers. This is currently hidden, and git
> silently picks the tag it fetched first.
> 
> (b) script compatibility, as changing the ref layout  such that new git
> can't work with old repository would be bad
> 
> the counter to this, is that we make git smart enough to recognize old 
> remote format, and continue to work with it. Scripts which depend on this
> layout will break, but that may not be such a huge concern.
> 
> Personally, I think this proposal at least for heads, notes, replace, and
> other remote refs we'd like to pull is very useful. I don't rightly know
> the answer for tags. The linked discussion below covers several pages of
> back and forth between a few people about which method is best.
> 
> I like the idea of simplifying tags and branches and notes and others to
> all fetch the same way. local stuff is in refs/heads or refs/notes and
> remote stuff is (by default) in refs/remotes/<remote>/tags etc
> 
> But it does bring up some discussion as today we "auto follow" tags into
> refs/tags, and it can get weird for tags since now "ambiguous" tags must
> mean if there are tags of same name which point to different refs,

The tags would be disambiguated by their remote name, e.g. "origin/tags/v5.6"
vs. "hacker/tags/v5.6".  The change would actually simplify tag auto-following.

> and we'd need to teach a bunch of logic to the ref lookup code.

Not a lot.  Existing DWIMery already handles ambiguous branches, by
preferring a local branch name over any remote ones.  The only teaching
that's really needed is to resolve shorthand like "origin/v5.6" to
"refs/remotes/origin/tags/v5.6" (i.e. to look for anything matching
"refs/remotes/origin/*/v5.6") but that doesn't seem very difficult.

There is the question of how the user can even know if there's ambiguity.
Aside from paying attention to "git fetch" output, I think we could extend
"git tag" (and "git branch") in the case where the user specifies an existing
tag (or branch).  Right now both commands fail because the name already
exists.  All we need to do is extend that error message a bit, e.g.:

	> git tag v5.6
	fatal: tag 'next' already exists as
	v5.6 -> a3a0e5d67d554a685abd897bc3ce4ffa4e74c812
	origin/v5.6 remoteX/v5.6 -> 504346bbf9b02387b51f232f4db9c1860789b135
	hacker/v5.6 -> fb4aa3533f81700789b3fb119e527410678e8d8d

Here we see that our local v5.6, and hacker's v5.6, are unique, but origin
and remoteX both have the same v5.6.

Well, same-ish.  I think those SHA IDs should be the things the tags point
at, not the tag objects' IDs.  This keeps things consistent between
lightweight and annotated/signed tags.  It does mean though that origin/v5.6
and remoteX/v5.6 might be different tag objects that happen to point at the
same thing.  I'm not sure the distinction is all that germane, and it's
something that the user could disambiguate with "git tag -v" or "git show".

> I am looking at ways to help git-notes be easier to use, so that we by 
> default fetch notes, and enable easier merge, since we'd have default 
> locations to merge from and to. This would make the sharing of notes a lot
> easier, which is one of their primary sticking points.. you can't really
> share them without *everyone* working to do it the same way you do. By
> making a default policy, sharing becomes natural, and users can easily add
> *public* notes to commits for things such as bug ids and other things
> which are not discovered until after the commit is created.
> 
> In addition, the easy ability to share replaces might also be helpful, 
> though IMHO not as valuable as git-notes.
> 
> I think that the only logical refs layout is 
> "refs/remotes/<remote>/(heads|tags|notes|replace)"
> 
> and adding "refs/remote-notes" and "refs/remote-replace" is not really a
> clean solution.
> 
> Given that the 1.8.0 proposal mostly died, does anyone have any thoughts
> now?
> 
> The proposal suggested by Johan makes sense to me, and I believe we can
> code up logic to make it easy for new git to keep logic of the old 
> layout.
> 
> Personally, I think the best solution is to only store that layout for a
> given clone, using a config option that defaults to false, where new-git
> sets it to true for all clones. Then, provide a command to renew
> remotes-layout that does this if the user wishes. Thus, clones for the old
> style will be handled, and new clones would have the new layout. (ie: no
> mixing layouts in a single repository).
> 
> I'm really not sure if this is the best solution, but seems like the 
> cleanest solution.

I think that proposal is good.  Perhaps something like
config.remoterefslayout, where 0 means the existing layout and 1 is the new
layout (with larger numbers reserved for future changes).

Implementing the new layout should, I think, just involve changing the
default fetch refspecs when adding a remote.

Then it's just a matter of making the test suite pass.  :)

		M.

Re: proper remote ref namespaces

[Junio C Hamano]

Jacob Keller <jacob.keller@gmail.com> writes:

> Recently there was some discussion about git-notes and how we do not
> fetch notes from remotes by default. The big problem with doing so is
> because refs/remotes/* hierarchy is only setup for branches (heads),
> so we don't have any clean location to put them.

I wouldn't call this topic "proper" namespaces, though.  What we
have is widely used and it shouldn't be broken.  Call it "enhanced",
perhaps.

Some design boundaries:

 - Moving the remote-tracking branch hierarchy from refs/remotes/$O/*
   to refs/remotes/$O/heads/* would not fly, because it will break
   existing repositories.  Do not even waste time on pursuing
   refs/remotes/$O/{heads,tags,notes...}/*

 - Extending the current refs/remotes/$O/* (for branches) and doing
   refs/remote-tags/$O/* (for tags) may work, would not break
   existing repositories, and could to be extended further to cover
   refs/remote-notes/$O and friends.  It however may not look pretty
   (weak objection) and more importantly, it would make it harder to
   "cull" things that came from a single remote.

Just thinking aloud, perhaps we can introduce a brand new top level
hierarchy refs/remote/$O/{heads,tags,notes,...}, and give backward
compatibility by making a moral equivalent of a symbolic link from
refs/remote/$O/heads to refs/remotes/$O/.  The true remote-tracknig
refs continue to live in refs/remotes/$O/* and old tools that do not
know the new layout would not be hurt.  New tools that want to
ignore and look only at refs/remote/$O/* can do so through the moral
equivalent of a symbolic link.  "remote remove" needs to be taught
about this single exception (i.e. "the symbolic link"), but the
places it needs to touch is limited only to two places and will not
grow.

If somebody got confused, notice that in the above description, I
said refs/remotes/ and refs/remote/.  The former must stay.  The
name of the latter is open to bikeshedding.  Some may prefer a name
that is more distinct (refs/tracking/ or something, perhaps?).  I
happen to prefer a name that is similar, but this preference is very
weak and I can persuaded to go either way.

Re: proper remote ref namespaces

[Junio C Hamano]

Marc Branchaud <marcnarc@xiplink.com> writes:

> Not a lot.  Existing DWIMery already handles ambiguous branches, by
> preferring a local branch name over any remote ones.  The only teaching
> that's really needed is ...

You need to remember that there are five useful things you can do to
mutable things.

 - Creation can be covered by teaching "clone" to put a new-style
   refspec but the same change needs to also go to "remote add".

 - Reading is done by the DWIMery in ref_rev_parse_rules[] (your
   point above).

 - Updating is automatic, as "fetch" does not have any funny
   built-in intuit and blindly follows configured fetch refspec.

 - Deletion by "branch -d -r" and "remote remove" needs to be
   careful about designing how the case where both old and new
   hierarchies exist should be handled (my gut feeling is "delete
   both", but there may be funny corner cases).

 - Enumeration by "branch -l -r" probably shares the same issue as
   deletion.