Re: enhanced remote ref namespaces

Re: enhanced remote ref namespaces

[Jacob Keller]

On Wed, Aug 12, 2015 at 9:10 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Jacob Keller <jacob.keller@gmail.com> writes:
>
>> Recently there was some discussion about git-notes and how we do not
>> fetch notes from remotes by default. The big problem with doing so is
>> because refs/remotes/* hierarchy is only setup for branches (heads),
>> so we don't have any clean location to put them.
>
> I wouldn't call this topic "proper" namespaces, though.  What we
> have is widely used and it shouldn't be broken.  Call it "enhanced",
> perhaps.
>

Ok.

> Some design boundaries:
>
>  - Moving the remote-tracking branch hierarchy from refs/remotes/$O/*
>    to refs/remotes/$O/heads/* would not fly, because it will break
>    existing repositories.  Do not even waste time on pursuing
>    refs/remotes/$O/{heads,tags,notes...}/*
>

even if we maintained new git's abililty to work with this, ie: only
external-to-git scripts would break and only for new clones? Maybe we
don't want to go this route, but it seems like the way that the
original proposal was headed.

>  - Extending the current refs/remotes/$O/* (for branches) and doing
>    refs/remote-tags/$O/* (for tags) may work, would not break
>    existing repositories, and could to be extended further to cover
>    refs/remote-notes/$O and friends.  It however may not look pretty
>    (weak objection) and more importantly, it would make it harder to
>    "cull" things that came from a single remote.
>
> Just thinking aloud, perhaps we can introduce a brand new top level
> hierarchy refs/remote/$O/{heads,tags,notes,...}, and give backward
> compatibility by making a moral equivalent of a symbolic link from
> refs/remote/$O/heads to refs/remotes/$O/.  The true remote-tracknig
> refs continue to live in refs/remotes/$O/* and old tools that do not
> know the new layout would not be hurt.  New tools that want to
> ignore and look only at refs/remote/$O/* can do so through the moral
> equivalent of a symbolic link.  "remote remove" needs to be taught
> about this single exception (i.e. "the symbolic link"), but the
> places it needs to touch is limited only to two places and will not
> grow.
>


I think this proposal makes the sense..  I'd go with something like:

refs/tracking/<origin>/(heads|tags|notes|replace|etc)/*

with a symlink from or into

refs/tracking/<origin>/heads -> refs/remotes/<origin>

I prefer tracking vs remote because "remote" and "remotes" are too
similar for my taste. tracking I think gets the idea across pretty
straight forward. Using a symlink personally I'd symlink the
refs/tracking into refs/remotes rather than make refs/remotes the real
storage.


> If somebody got confused, notice that in the above description, I
> said refs/remotes/ and refs/remote/.  The former must stay.  The
> name of the latter is open to bikeshedding.  Some may prefer a name
> that is more distinct (refs/tracking/ or something, perhaps?).  I
> happen to prefer a name that is similar, but this preference is very
> weak and I can persuaded to go either way.

I don't like it being so similar that we now have typos between
remotes and remote.. ie: remotes/<origin> works for heads, but
"remotes/<origin>/tags" does not... that sounds like it would get
confusing.

Symlinking the old location seems reasonable to me, as it would leave
all the same data in the locations expected by the old tools, while
keeping all actual storage in the new location.

In this way, we would no longer need configuration settings. It
honestly doesn't matter to me which direction we symlink either.

As for the other complex issue is what to do about "refs/tracking/<origin>/tags

The big discussion on the original thread is about how tags would
work. I'm personally ok with *ignoring* tags and leaving it the way it
is for now, and just doing this as a solid place to stick
notes/replace/etc.

Or, we could go the route of continuing to stick tags into "refs/tags"
at the same time as also sticking them into
refs/tracking/<origin>/tags

Or.. we could go the full route of fixing up lookup of tags such that
we put tags in refs/tracking/<origin>/tags and we have it lookup tags
there via something like:

1) local tags preferred

2) any remote tag as long as all remote tags point to the same commit
object (how we select which to use is not very relevant here... we
could actually go with as long as the tag object is the same so two
remotes with annotated tags pointing to the same object but different
tag id would be ambiguous as well)

3) warn the user must disambiguate the tag via the remote name.

We could also teach fetch to warn about remote tags which are
ambiguous (ie: two remotes with same named tag objects pointing to
different things)

How goes this sound? I think it makes sense... I don't know how to do
all this without breaking some backwards compatibility though... I
think we can maintain expectations for the general user but I feel
that any change here will break *someones* scripts.

Regards,
Jake

Re: enhanced remote ref namespaces

[Junio C Hamano]

Jacob Keller <jacob.keller@gmail.com> writes:

>> Just thinking aloud, perhaps we can introduce a brand new top level
>> hierarchy refs/remote/$O/{heads,tags,notes,...}, and give backward
>> compatibility by making a moral equivalent of a symbolic link from
>> refs/remote/$O/heads to refs/remotes/$O/.  The true remote-tracknig
>> refs continue to live in refs/remotes/$O/* and old tools that do not
>> know the new layout would not be hurt.  New tools that want to
>> ignore and look only at refs/remote/$O/* can do so through the moral
>> equivalent of a symbolic link.  "remote remove" needs to be taught
>> about this single exception (i.e. "the symbolic link"), but the
>> places it needs to touch is limited only to two places and will not
>> grow.
>
> I think this proposal makes the sense..  I'd go with something like:
>
> refs/tracking/<origin>/(heads|tags|notes|replace|etc)/*
>
> with a symlink from or into
>
> refs/tracking/<origin>/heads -> refs/remotes/<origin>

I actually do not think we even need the "symlink" thing.

We can just say refs/remotes/$O has been and forever will be where
the remote-tracking branches will live.  With or without the symlink
for backward compatibility, the updated Git will need to be aware of
the two places to deal with older and newer repositories anyway.

"everything is now under refs/tracking/$O, not spread over many
unbounded number of places like refs/remotes-$foo/$O" may appear to
be cleaner but two is already not too bad and will greatly reduce
the transition pain.

Re: enhanced remote ref namespaces

[Jacob Keller]

On Wed, Aug 12, 2015 at 11:54 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Jacob Keller <jacob.keller@gmail.com> writes:
>
>>> Just thinking aloud, perhaps we can introduce a brand new top level
>>> hierarchy refs/remote/$O/{heads,tags,notes,...}, and give backward
>>> compatibility by making a moral equivalent of a symbolic link from
>>> refs/remote/$O/heads to refs/remotes/$O/.  The true remote-tracknig
>>> refs continue to live in refs/remotes/$O/* and old tools that do not
>>> know the new layout would not be hurt.  New tools that want to
>>> ignore and look only at refs/remote/$O/* can do so through the moral
>>> equivalent of a symbolic link.  "remote remove" needs to be taught
>>> about this single exception (i.e. "the symbolic link"), but the
>>> places it needs to touch is limited only to two places and will not
>>> grow.
>>
>> I think this proposal makes the sense..  I'd go with something like:
>>
>> refs/tracking/<origin>/(heads|tags|notes|replace|etc)/*
>>
>> with a symlink from or into
>>
>> refs/tracking/<origin>/heads -> refs/remotes/<origin>
>
> I actually do not think we even need the "symlink" thing.
>
> We can just say refs/remotes/$O has been and forever will be where
> the remote-tracking branches will live.  With or without the symlink
> for backward compatibility, the updated Git will need to be aware of
> the two places to deal with older and newer repositories anyway.
>
> "everything is now under refs/tracking/$O, not spread over many
> unbounded number of places like refs/remotes-$foo/$O" may appear to
> be cleaner but two is already not too bad and will greatly reduce
> the transition pain.

Ok, so.... just have branches be in refs/remotes/$O and all new things
be in refs/tracking/$O/category

that sounds reasonable enough to me.

That still hasn't really resolved the question of how to deal with
tags, but it does solve the question of how to deal with replace and
notes refs.

Regards,
Jake

Re: enhanced remote ref namespaces

[Junio C Hamano]

Jacob Keller <jacob.keller@gmail.com> writes:

> That still hasn't really resolved the question of how to deal with
> tags, but it does solve the question of how to deal with replace and
> notes refs.

I do not think it would be a good change to add a

    [remote "foo"]
        fetch = refs/tags/*:refs/tracking/foo/tags/*

as people do expect to see tags from upstream to appear in their
local tag namespace when cloning from the central meeting point of
their project (e.g. Linus's kernel repository).

I'm willing to believe those who argued that they have both private
tags and public tags in their repository, but I think that would be
something better supported by adding "git tag --local foo" that
creates a local tag in e.g. refs/local-tags/foo, and by teaching
DWIMmery to also look in there when the user says "foo".

Alternatively, they can use their local branch namespace, which is
already private to them.

Re: enhanced remote ref namespaces

[Johan Herland]

On Wed, Aug 12, 2015 at 8:34 PM, Jacob Keller <jacob.keller@gmail.com> wrote:
> On Wed, Aug 12, 2015 at 9:10 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> Some design boundaries:
>>
>>  - Moving the remote-tracking branch hierarchy from refs/remotes/$O/*
>>    to refs/remotes/$O/heads/* would not fly, because it will break
>>    existing repositories.  Do not even waste time on pursuing
>>    refs/remotes/$O/{heads,tags,notes...}/*
>
> even if we maintained new git's abililty to work with this, ie: only
> external-to-git scripts would break and only for new clones? Maybe we
> don't want to go this route, but it seems like the way that the
> original proposal was headed.

I don't think it's worth trying to go that route. Even though it's
theoretically possible to solve it, the complexity added by e.g.
multiple git versions operating on the same repository (maybe even
simultaneously?) suggests that it's much simpler to just go with a new
clean hierarchy that simply Works for new Git versions, and the older
versions don't have to deal with at all. IMHO, maybe even going as far
as incrementing core.repositoryFormatVersion, so that older Git
versions will refuse to work with new-style repos...

[...]
>> If somebody got confused, notice that in the above description, I
>> said refs/remotes/ and refs/remote/.  The former must stay.  The
>> name of the latter is open to bikeshedding.  Some may prefer a name
>> that is more distinct (refs/tracking/ or something, perhaps?).  I
>> happen to prefer a name that is similar, but this preference is very
>> weak and I can persuaded to go either way.
>
> I don't like it being so similar that we now have typos between
> remotes and remote.. ie: remotes/<origin> works for heads, but
> "remotes/<origin>/tags" does not... that sounds like it would get
> confusing.

I like refs/tracking. At some point I was planning to use refs/peers,
but that's merely another color for the bikeshed...

> Symlinking the old location seems reasonable to me, as it would leave
> all the same data in the locations expected by the old tools, while
> keeping all actual storage in the new location.
>
> In this way, we would no longer need configuration settings. It
> honestly doesn't matter to me which direction we symlink either.
>
> As for the other complex issue is what to do about "refs/tracking/<origin>/tags
>
> The big discussion on the original thread is about how tags would
> work. I'm personally ok with *ignoring* tags and leaving it the way it
> is for now, and just doing this as a solid place to stick
> notes/replace/etc.

That is probably the best plan for now: Solve most of the problem, and
punt on the controversial parts.

> Or, we could go the route of continuing to stick tags into "refs/tags"
> at the same time as also sticking them into
> refs/tracking/<origin>/tags

I don't like the copying approach; makes it harder to deduce which tags
are (truly) local, and which came from a remote.

> Or.. we could go the full route of fixing up lookup of tags such that
> we put tags in refs/tracking/<origin>/tags and we have it lookup tags
> there via something like:
>
> 1) local tags preferred
>
> 2) any remote tag as long as all remote tags point to the same commit
> object (how we select which to use is not very relevant here... we
> could actually go with as long as the tag object is the same so two
> remotes with annotated tags pointing to the same object but different
> tag id would be ambiguous as well)
>
> 3) warn the user must disambiguate the tag via the remote name.

As was probably obvious from the old threads, this is where I'd like
to go. Eventually.

> We could also teach fetch to warn about remote tags which are
> ambiguous (ie: two remotes with same named tag objects pointing to
> different things)

Agreed. This would actually be a good improvement Right Now, without
changing any ref layout at all: If get fetch would otherwise copy a
remote tag into refs/tags/*, but doesn't because there is already a
_different_ tag in its place, then warn loudly to the user, so that
user has a chance to discover the tag difference.

> How goes this sound? I think it makes sense... I don't know how to do
> all this without breaking some backwards compatibility though... I
> think we can maintain expectations for the general user but I feel
> that any change here will break *someones* scripts.

As I said above: Punt on tags for now, and you might be able to not
break anyone's scripts (and if you do, it's probably a poorly written
script). Provided that you leave a symlink to/from refs/remotes/$O in
place, you're AFAICS only adding functionality, not (visibly) changing
existing behavior.


BTW, thanks for resurrecting this topic!

...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net