Working with remotes; cloning remote references

Working with remotes; cloning remote references

[Marc Branchaud]

Hi all,

We're a Subversion shop moving to git, and our git-fu is still pretty 
weak.  So I'd appreciate feedback on the setup I'm going to describe, in 
addition to the specific issue I'm going to raise.

Our products involve extensive customization of various 
externally-maintained code bases (mainly in FreeBSD, but also in a few 
other projects too).  We'd like to track the various external 
repositories, to be able to bring their updates into our code base.  We 
don't normally push our own customizations back upstream (yes, not The 
Way Things Should Be, please turn down the flamethrowers).

So we want our main code tree to be composed of our own code and an 
assortment of modified external code:

main/
   external1/
   external2/
   ourstuffA/
   ourstuffB/

Our general approach for supporting this (remember, git novices here) is 
to keep a local git mirror of each external code repository (say, with 
git-svn for the FreeBSD tree).  Then we incorporate the local mirrors 
into our main repo, each under its own directory.

I've looked at two ways to do this: submodules and subtree merges, and 
the latter looks better to me.  The main complication with submodules is 
branching and tagging.  We need to branch or tag the entire main code 
tree, including all the external code bases.  With submodules that seems 
like it would involve several steps to touch all the external mirror 
repos as well as the main repo.  Another complication is that we do a 
lot of our work (50% or more) in the external code trees, and having to 
update the main repo's submodule references as that code changes seems 
rather fragile.

OTOH, a subtree merge nicely puts the external code into a specific spot 
in our main tree, lets that spot get updated with a single command (git 
pull -s subtree External2 master), and still let us branch and tag with 
single commands in the main repo.  Also, the changes we make to 
externally-based code are plain old git pushes to the main repo.  Clean 
and simple.  (BTW, what does braid add beyond just doing subtree merges 
directly?)

So, first the general question: Anything braindead about the above?  Any 
better ways to do this?

Now for the specific issue, which has to do with managing the remotes 
defined in the main repository.  The subtree-merge approach calls for 
the mirrored external repos to be "git remote add"-ed to the main repo. 
  But clones of that repo don't include the remote references (using git 
1.5.4.3 for my testing here).

I think that makes sense in most cases:  Usually a developer will only 
want to clone the main repo and work with what's in there, ad not want 
to bother with the external references.  But when it comes time to pull 
in updates from the external mirror repos the only place where that can 
be done is on the main repo where the original "git remote add" commands 
were issued.

That means merge work has to be done on the main repository's shared 
host (not good) and that testing merged code has to involve committing 
changes on the main shared repo (also not good, even in a branch) and 
testing them on a different host (still no goodness here).

 From this point of view (and please correct me if I'm wrong), it would 
be good if "git clone" had an option to include a repo's remote 
references in the clone.  The clone's origin reference would point to 
the original repo as usual, but all the other remotes would be in the clone.

Pushing and pulling would then also update (non-origin) the remote 
references.

Does this make sense?  Am I missing something here?  (And thanks for 
reading this far!)

		Marc

Re: Working with remotes; cloning remote references

[Peter Harris]

On Thu, Oct 16, 2008 at 2:17 PM, Marc Branchaud wrote:
> From this point of view (and please correct me if I'm wrong), it would be
> good if "git clone" had an option to include a repo's remote references in
> the clone.  The clone's origin reference would point to the original repo as
> usual, but all the other remotes would be in the clone.

"git clone" doesn't have this option, but you can turn it on
immediately after with something similar to:
git config --add remote.origin.fetch +refs/remotes/*:refs/remotes/*
(which I use for fanning-out my git-svn repos)

See "git help fetch" for further documentation.

Peter Harris

Re: Working with remotes; cloning remote references

[Marc Branchaud]

Peter Harris wrote:
> 
> "git clone" doesn't have this option, but you can turn it on
> immediately after with something similar to:
> git config --add remote.origin.fetch +refs/remotes/*:refs/remotes/*
> (which I use for fanning-out my git-svn repos)

Thanks for the pointer (and the quick reply).

That doesn't seem to be what I'm looking for, though -- perhaps I'm 
missing something?  The above puts the remotes in the .git/refs/remotes 
directory, but the .git/config file doesn't have them.

More specifically, if I clone the main repository and run the above and 
then fetch, then AFAICT I'm still not linked to the external mirrors in 
the clone.  In particular, I can't refer to an external project in the 
subtree pull:

	$ git pull -s subtree external2 master
	fatal: 'external2': unable to chdir or not a git archive

(Where 'external2' is the name of the remote in the main repo.)  Also, 
the only remote in the config file is still just the origin.

(As an aside, the above "git config --add" incantation causes problems 
if run inside a clone of a clone:
	$ git clone main clone-of-main
	$ git clone clone-of-main clone-of-clone-of-main
	$ cd clone-of-clone-of-main
	$ git config --add remote.origin.fetch \
		+refs/remotes/*:refs/remotes/*
	$ git pull
	fatal: refs/remotes/origin/master tracks both 
refs/remotes/origin/master and refs/heads/master
)

You did say "something similar" in your reply -- am I not seeing 
something obvious?

		Marc

Re: Working with remotes; cloning remote references

[Peter Harris]

On Thu, Oct 16, 2008 at 4:29 PM, Marc Branchaud wrote:
> Peter Harris wrote:
>>
>> "git clone" doesn't have this option, but you can turn it on
>> immediately after with something similar to:
>> git config --add remote.origin.fetch +refs/remotes/*:refs/remotes/*
>> (which I use for fanning-out my git-svn repos)
>
> Thanks for the pointer (and the quick reply).
>
> That doesn't seem to be what I'm looking for, though -- perhaps I'm missing
> something?  The above puts the remotes in the .git/refs/remotes directory,
> but the .git/config file doesn't have them.

Ah. I believe I misunderstood what you wanted. Perhaps you want "git
remote add", then? Unfortunately, you have to know what your remotes
are outside of git for this. Perhaps a script in the root of your
repository you can run to set this up after the initial clone?

I seem to recall some discussion of allowing a .gitconfig to be in
repositories (similar to .gitignore), but the idea was shot down for
security reasons.

> (As an aside, the above "git config --add" incantation causes problems if
> run inside a clone of a clone:
>
> You did say "something similar" in your reply -- am I not seeing something
> obvious?

I said "something similar" because you probably actually want
git config --add remote.$remote.fetch +refs/remotes/*:refs/remotes/$remote/*
(or other names of your own choosing) so that multiple remote remotes
don't stomp on each other.

Unless you want to be able to address the remotes by name, in which
case you want "git remote add" instead (see above).

Peter Harris

Re: Working with remotes; cloning remote references

[Marc Branchaud]

Peter Harris wrote:
> 
> Ah. I believe I misunderstood what you wanted. Perhaps you want "git
> remote add", then? Unfortunately, you have to know what your remotes
> are outside of git for this. Perhaps a script in the root of your
> repository you can run to set this up after the initial clone?

I think we're converging onto the same track.

Yes, I do want to use "git remote add".  My point is basically that, 
having done various git-remote-adds in the main repository, I'd like to 
avoid having to redo them in a clone of that repository.

A script would work, sure, but to me this seems like something git 
should handle for me.  If I have to re-establish my connections to the 
remotes whenever I want to pull in updates, then I don't see much point 
in keeping the remotes defined in any git repo.

> I seem to recall some discussion of allowing a .gitconfig to be in
> repositories (similar to .gitignore), but the idea was shot down for
> security reasons.

I think I can understand why that would be undesirable, but I don't know 
if something like that would be necessary for what I'm talking about.

		Marc

Re: Working with remotes; cloning remote references

[Michael J Gruber]

Marc Branchaud venit, vidit, dixit 17.10.2008 00:09:
> Peter Harris wrote:
>> Ah. I believe I misunderstood what you wanted. Perhaps you want "git
>> remote add", then? Unfortunately, you have to know what your remotes
>> are outside of git for this. Perhaps a script in the root of your
>> repository you can run to set this up after the initial clone?
> 
> I think we're converging onto the same track.
> 
> Yes, I do want to use "git remote add".  My point is basically that, 
> having done various git-remote-adds in the main repository, I'd like to 
> avoid having to redo them in a clone of that repository.
> 
> A script would work, sure, but to me this seems like something git 
> should handle for me.  If I have to re-establish my connections to the 
> remotes whenever I want to pull in updates, then I don't see much point 
> in keeping the remotes defined in any git repo.
> 
>> I seem to recall some discussion of allowing a .gitconfig to be in
>> repositories (similar to .gitignore), but the idea was shot down for
>> security reasons.
> 
> I think I can understand why that would be undesirable, but I don't know 
> if something like that would be necessary for what I'm talking about.

I don't think there is a direct gittish way for transferring the remote
config from one repo to a clone (other than copying what git submodule
does with .gitmodules etc.).

Would it be sufficient for you if a clone could trigger the main repo to
update its remotes (i.e. git remote update)?

Michael

Re: Working with remotes; cloning remote references

[Marc Branchaud]

Thanks for joining the thread, Michael!

Michael J Gruber wrote:
> 
> I don't think there is a direct gittish way for transferring the remote
> config from one repo to a clone (other than copying what git submodule
> does with .gitmodules etc.).

That's my impression, too, so I'm suggesting that the clone/push/pull 
commands get an option to work with remotes in config.

> Would it be sufficient for you if a clone could trigger the main repo to
> update its remotes (i.e. git remote update)?

Hmmm, I'm not sure...  How would the overall 
merge-in-changes-from-upstream process work in that case?

Say the main repo has the code for "ThingOne" merged under a top-level 
thing-one/ directory:

main/$ git remote add -f ThingOne git://thing/ThingOne.git
main/$ git merge -s ours --no-commit ThingOne/master
main/$ git read-tree --prefix=thing-one/ -i ThingOne/master
main/$ git commit -m "Merged ThingOne into /thing-one/"

Then the ThingOne folks update their code, so we want to incorporate 
their changes into our version of their code.  I think you're suggesting 
that we might make a clone of the main repo then trigger a "git remote 
update" in the clone's origin:

clone/$ make-origin-do-git-remote-update

What's not clear to me is how to proceed from here.  At this point I'm 
hoping there's some way I could do some work in the clone to merge the 
changes the ThingOne folks did into our code.  I'm too new to git to 
know if there is a way forward -- is there?

(With the subtree merge pattern I'd run "git pull -s subtree ThingOne 
master" in the main repo -- is there an equivalent achievable in the 
clone if the main has done a "git remote update"?)

Thanks,

		Marc

Re: Working with remotes; cloning remote references

[Michael J Gruber]

Marc Branchaud venit, vidit, dixit 17.10.2008 16:44:
> Thanks for joining the thread, Michael!

Sure, there's no need to mention it.

Well, maybe there is, given the recent discussion: Everyone can
contribute (with or without C/perl/whatever fu) and thus "pay back" for
free software, even by "only" trying to answer questions. But it
requires participation, combining efforts, rather than splitting off and
not returning... Uh, sorry, your question:

> Michael J Gruber wrote:
>> Would it be sufficient for you if a clone could trigger the main repo to
>> update its remotes (i.e. git remote update)?
> 
> Hmmm, I'm not sure...  How would the overall 
> merge-in-changes-from-upstream process work in that case?

I thought your problem was "transporting" the remote branches from main
(which tracks external) to clone, without having the remote config for
external on the clones. In that case being able to trigger "remote
update" on main from the clones would help.

> Say the main repo has the code for "ThingOne" merged under a top-level 
> thing-one/ directory:
> 
> main/$ git remote add -f ThingOne git://thing/ThingOne.git
> main/$ git merge -s ours --no-commit ThingOne/master
> main/$ git read-tree --prefix=thing-one/ -i ThingOne/master
> main/$ git commit -m "Merged ThingOne into /thing-one/"
> 
> Then the ThingOne folks update their code, so we want to incorporate 
> their changes into our version of their code.  I think you're suggesting 
> that we might make a clone of the main repo then trigger a "git remote 
> update" in the clone's origin:
> 
> clone/$ make-origin-do-git-remote-update
> 
> What's not clear to me is how to proceed from here.  At this point I'm 
> hoping there's some way I could do some work in the clone to merge the 
> changes the ThingOne folks did into our code.  I'm too new to git to 
> know if there is a way forward -- is there?
> 
> (With the subtree merge pattern I'd run "git pull -s subtree ThingOne 
> master" in the main repo -- is there an equivalent achievable in the 
> clone if the main has done a "git remote update"?)

"pull -s strategy repo master" does a fetch followed by "merge -s
strategy repomaster", where repomaster is the ref for master on repo.
So, if you got that branch (repomaster=ThingOne/master) by cloning from
main you can do the merge (subtree or other) on your clone, even without
the remote repo config for ThingOne on clone.

I guess you want to do those merges on several clones, and main, and
push everything back to main. I don't know how well that mixes, but I
don't expect problems, only unexpected ones ;)

Cheers,
Michael

Re: Working with remotes; cloning remote references

[Marc Branchaud]

Michael J Gruber wrote:
> 
> "pull -s strategy repo master" does a fetch followed by "merge -s
> strategy repomaster", where repomaster is the ref for master on repo.
> So, if you got that branch (repomaster=ThingOne/master) by cloning from
> main you can do the merge (subtree or other) on your clone, even without
> the remote repo config for ThingOne on clone.

I'm afraid I'm having trouble translating what you're saying into actual 
git commands (or are you proposing some new git functionality?).  How 
would I get the ThingOne/master branch into the clone?


After some more thought I realized that the clone can just pull directly 
from the ThingOne repository:

clone/$ git pull -s subtree git://thing/ThingOne.git master

(I'm still getting used to git's ability to match commit IDs from 
anywhere -- it's magic! :) )

This goes a long way to where we want to be, in that we don't have to do 
our merging work in the original main repository.

It would be nice, though, if the clone were able to use the main 
repository's definition of the ThingOne remote.  I can think of some 
plausible scenarios where a person could get confused about which 
repo/branch they're supposed to pull.  It's easy to recover from that 
kind of mistake, but there'd be less chance of a mistake if one could 
tell git to "pull from X as defined in the origin repository".

And actually, git's remote functionality feels a bit crippled if clones 
can't make some use of the origin's remotes.  Is there a reason for 
keeping remote definitions out of a clone?

		Marc

Re: Working with remotes; cloning remote references

[Michael J Gruber]

Marc Branchaud venit, vidit, dixit 17.10.2008 21:50:
> Michael J Gruber wrote:
>> "pull -s strategy repo master" does a fetch followed by "merge -s
>> strategy repomaster", where repomaster is the ref for master on repo.
>> So, if you got that branch (repomaster=ThingOne/master) by cloning from
>> main you can do the merge (subtree or other) on your clone, even without
>> the remote repo config for ThingOne on clone.
> 
> I'm afraid I'm having trouble translating what you're saying into actual 
> git commands (or are you proposing some new git functionality?).  How 
> would I get the ThingOne/master branch into the clone?

Sorry for being cryptic. What I meant was: The clones don't need the
full remote config of main in order to get main's remote branches for
the ThingOne remote. Say, main stores the remote tracking branches for
ThingOne in "refs/remotes/ThingOne", using an appropriate remote config.
Then a clone could use the refspec
"refs/remotes/ThingOne/*:refs/remotes/ThingOne/*" (when fetching from
main) in order to fetch those branches, without having a remote config
for ThingOne on the clone. Concretely:

git config remote.main.fetch
'+refs/remotes/ThingOne/*:refs/remotes/ThingOne/*'

on a clone which has a remote config "main" for the main repo.

> After some more thought I realized that the clone can just pull directly 
> from the ThingOne repository:
> 
> clone/$ git pull -s subtree git://thing/ThingOne.git master

I thought that's what you were trying to avoid...

> (I'm still getting used to git's ability to match commit IDs from 
> anywhere -- it's magic! :) )

... but this explains everything ;)

> This goes a long way to where we want to be, in that we don't have to do 
> our merging work in the original main repository.
> 
> It would be nice, though, if the clone were able to use the main 
> repository's definition of the ThingOne remote.  I can think of some 
> plausible scenarios where a person could get confused about which 
> repo/branch they're supposed to pull.  It's easy to recover from that 
> kind of mistake, but there'd be less chance of a mistake if one could 
> tell git to "pull from X as defined in the origin repository".

I think the approach I outlined above could solve this: main prepares
everything to be pulled under refs/remotes/ThingOne, and clones (i.e.
clone users) are told to fetch with a refspec like above. This way they
get a copy of main's remote tracking branches for ThingOne. (Maybe you
use one branch only, then the refspec is even simpler.)

> And actually, git's remote functionality feels a bit crippled if clones 
> can't make some use of the origin's remotes.  Is there a reason for 
> keeping remote definitions out of a clone?

Say A and B are working on a project C. Then, typically, A is interested
in B's work on C, i.e. some of B's local branches, but not on B's remote
 tracking branches: A tracks a "central" C already, just like B does,
and remote tracking branches don't carry any "value adding" by the
cloner, they're just a local unmodified copy of the remote.

But you can always add a refspec like above.

Michael

Re: Working with remotes; cloning remote references

[Marc Branchaud]

Thanks for the explicit instructions, but I think I'm still missing 
something.  Here's exactly what I'm trying, just to make sure I'm not 
doing something odd.

First I set up the remote in the main repo:

main/$ git remote add -f ThingOne git://thing/ThingOne.git
main/$ git merge -s ours --no-commit ThingOne/master
main/$ git read-tree --prefix=thing-one/ -u ThingOne/master
main/$ git commit -m "Adding ThingOne"

Then I make a clone, fetch main's refs to ThingOne, and try to pull in 
changes from ThingOne by using those refs:

$ git clone /path/to/main clone
$ cd clone
clone/$ git config remote.origin.fetch \
         '+refs/remotes/ThingOne/*:refs/remotes/ThingOne/*' 

clone/$ git fetch
 From /path/to/main/
  * [new branch]      ThingOne/master -> ThingOne/master 

clone/$ git pull -s subtree git://thing/ThingOne.git ThingOne/master
fatal: Couldn't find remote ref ThingOne/master

The config & fetch commands indeed add a 
.git/refs/remotes/ThingOne/master file to the clone, but I'm missing the 
magic words for how to use that ref.


Also:

Michael J Gruber wrote:
> 
>> And actually, git's remote functionality feels a bit crippled if clones 
>> can't make some use of the origin's remotes.  Is there a reason for 
>> keeping remote definitions out of a clone?
> 
> Say A and B are working on a project C. Then, typically, A is interested
> in B's work on C, i.e. some of B's local branches, but not on B's remote
>  tracking branches: A tracks a "central" C already, just like B does,
> and remote tracking branches don't carry any "value adding" by the
> cloner, they're just a local unmodified copy of the remote.

I can appreciate that, but the approach leads to two consequences that I 
suggest should be avoidable:

1. A and B both have to set up the references to C themselves.  A can't 
just piggyback on whatever B has already set up.  (This is the impetus 
for my original message.)

2. Suppose B is just a repository that's being shared between A and D -- 
i.e. there's no B entity doing any work directly in B.  In this case if 
A (or D) wants to change B's reference to C (say, to track a new branch 
in C), they can't: B has to be changed directly, and nothing can be done 
in a clone to accomplish this.

I admit I'm picking at nits here -- obviously someone is able to access 
the B repo and do whatever needs doing.  I just feel that there are some 
situations where you want the origin's remotes in your clone, and some 
where you don't, and git should let you decide.

		Marc

Re: Working with remotes; cloning remote references

[Michael J Gruber]

Marc Branchaud venit, vidit, dixit 10/20/08 18:50:
> Thanks for the explicit instructions, but I think I'm still missing 
> something.  Here's exactly what I'm trying, just to make sure I'm not 
> doing something odd.
> 
> First I set up the remote in the main repo:
> 
> main/$ git remote add -f ThingOne git://thing/ThingOne.git
> main/$ git merge -s ours --no-commit ThingOne/master
> main/$ git read-tree --prefix=thing-one/ -u ThingOne/master
> main/$ git commit -m "Adding ThingOne"
> 
> Then I make a clone, fetch main's refs to ThingOne, and try to pull in 
> changes from ThingOne by using those refs:
> 
> $ git clone /path/to/main clone
> $ cd clone
> clone/$ git config remote.origin.fetch \
>          '+refs/remotes/ThingOne/*:refs/remotes/ThingOne/*'

If you want to fetch main's local branches also, use option "--add" here
so that you don't override the default fetch refspec (forgot last time,
sorry).

> clone/$ git fetch
>  From /path/to/main/
>   * [new branch]      ThingOne/master -> ThingOne/master 
> 
> clone/$ git pull -s subtree git://thing/ThingOne.git ThingOne/master
> fatal: Couldn't find remote ref ThingOne/master

because on ThingOne there is no such ref.

> The config & fetch commands indeed add a 
> .git/refs/remotes/ThingOne/master file to the clone, but I'm missing the 
> magic words for how to use that ref.

I thought you wanted to avoid pulling directly from ThingOne to clone?
If you pull directly you might as well set up the same remote config as
on main: for the correct pull line you need to know the same info as for
the correct remote config.

git fetch
git merge -s subtree remotes/ThingOne/master

should do the trick. If that works you can set up things so that pulling
from origin (pulling when you're in your integration branch) does that
merge automatically, using branch.integrationbranch.remote=origin,
branch.integrationbranch.merge=remotes/ThingOne/master (untested ;) ).

To be clear: The idea here is that main decides which ThingOne branch to
store in remotes/ThingOne/master and where to get it from; clones always
pull that one.

> Also:
> 
> Michael J Gruber wrote:
>>> And actually, git's remote functionality feels a bit crippled if clones 
>>> can't make some use of the origin's remotes.  Is there a reason for 
>>> keeping remote definitions out of a clone?
>> Say A and B are working on a project C. Then, typically, A is interested
>> in B's work on C, i.e. some of B's local branches, but not on B's remote
>>  tracking branches: A tracks a "central" C already, just like B does,
>> and remote tracking branches don't carry any "value adding" by the
>> cloner, they're just a local unmodified copy of the remote.
> 
> I can appreciate that, but the approach leads to two consequences that I 
> suggest should be avoidable:
> 
> 1. A and B both have to set up the references to C themselves.  A can't 
> just piggyback on whatever B has already set up.  (This is the impetus 
> for my original message.)
> 
> 2. Suppose B is just a repository that's being shared between A and D -- 
> i.e. there's no B entity doing any work directly in B.  In this case if 
> A (or D) wants to change B's reference to C (say, to track a new branch 
> in C), they can't: B has to be changed directly, and nothing can be done 
> in a clone to accomplish this.
> 
> I admit I'm picking at nits here -- obviously someone is able to access 
> the B repo and do whatever needs doing.  I just feel that there are some 
> situations where you want the origin's remotes in your clone, and some 
> where you don't, and git should let you decide.

Well, it let's you decide: It tracks local branches by default, and
using additional "git config" you can track remotes as well. You can
also use the "--mirror" option to "git clone" or "git remote add", but
that has other side effects.

Cheers,
Michael

Re: Working with remotes; cloning remote references

[Marc Branchaud]

Michael J Gruber wrote:
>
>> clone/$ git config remote.origin.fetch \
>>          '+refs/remotes/ThingOne/*:refs/remotes/ThingOne/*'
> 
> If you want to fetch main's local branches also, use option "--add" here
> so that you don't override the default fetch refspec (forgot last time,
> sorry).

Okay, got it.

> I thought you wanted to avoid pulling directly from ThingOne to clone?

Ah, I see (part of) our disconnect here -- sorry for not explaining it
clearly before!

Yes, indeed, I do want to pull directly from ThingOne into the clone.

The scenario is that there's a bunch of us sharing the main repo, and
when some upstream changes happen in ThingOne I'd like for one of us to
be able to clone the main repo, pull the changes from ThingOne into the 
clone (fixing any conflicts; remember that we have our own changes in 
ThingOne's code), then push the merged changes back to main for everyone 
else to grab.

> If you pull directly you might as well set up the same remote config as
> on main: for the correct pull line you need to know the same info as for
> the correct remote config.

Yes, I see that -- it's why I'm basically satisfied with being able to do

	clone/$ git pull -s subtree /path/to/ThingOne master

from the clone, and all I'm doing now is kvetching about having to 
remember the location and branch of ThingOne when that was already 
configured in main (and thanks for being patient with my kvetching!).

> git fetch
> git merge -s subtree remotes/ThingOne/master
> 
> should do the trick.

AFAICT that only refers to the clone's local branch for the ThingOne 
repo.  The above git-config command doesn't add the ThingOne repo's URL 
to the clone, so I'm still stuck having to use that URL directly to pull 
changes from ThingOne into the clone.  (And when I use the URL directly, 
I have to use a branch name that's defined in the ThingOne repo.  I'd 
like to also be able to use whatever branch name got set up in main when 
"git remote add" was run.)

> If that works you can set up things so that pulling
> from origin (pulling when you're in your integration branch) does that
> merge automatically, using branch.integrationbranch.remote=origin,
> branch.integrationbranch.merge=remotes/ThingOne/master (untested ;) ).
> 
> To be clear: The idea here is that main decides which ThingOne branch to
> store in remotes/ThingOne/master and where to get it from; clones always
> pull that one.

I think that "where to get it from" part is what I'm going on about. 
There doesn't seem to be a way for the main repo to tell the clone where 
the ThingOne repo is, so that the clone can pull in ThingOne changes 
directly.

>> I just feel that there are some 
>> situations where you want the origin's remotes in your clone, and some 
>> where you don't, and git should let you decide.
> 
> Well, it let's you decide: It tracks local branches by default, and
> using additional "git config" you can track remotes as well. You can
> also use the "--mirror" option to "git clone" or "git remote add", but
> that has other side effects.

I think we're mis-communicating, mainly because I'm not yet able to 
express things well in git-speak.  Let me give it another stab...

I believe git lets you track the origin's _branches_ not the origin's 
_remotes_.  I don't think --mirror does what I'm looking for, because 
(side effects aside) it only deals with branches, not remotes.

I find myself getting confused, and I think it's because the files in 
.git/refs/remotes/ are indeed tracking branches on remote repositories. 
  So I think our conversation gets a bit circular because our ideas of a 
"remote" differ.

"git remote add" does two things (maybe more?): It adds a [remote] 
section to the .git/config file, and it adds a branch reference in 
.git/refs/remotes/.  I think what I'd like is for the clone to be able 
to obtain both these things from the origin.  The reason I think it's 
useful is that it would let the clone pull directly from the origin's 
remote repositories, without having to directly specify the remote 
repository's URL and branch name.

Fundamentally, I'm looking to do exactly

	clone/$ git pull -s subtree /path/to/ThingOne master

i.e. pull stuff from one of main's remotes directly into the clone.  But 
I want to replace the "/path/to/ThingOne master" part with something 
that means "use whatever URL and branch name was defined in the origin 
for this remote".

My questions are:  Am I right in thinking this is desirable?  Is there 
already some way to do this?  If not, is it something worth 
implementing?  (I'm happy to roll up my own sleeves here...)

I hope that clarifies things.  Sorry for taking so long to get here!

		Marc

Re: Working with remotes; cloning remote references

[Michael J Gruber]

Marc Branchaud venit, vidit, dixit 10/21/08 17:17:
> I believe git lets you track the origin's _branches_ not the origin's 
> _remotes_.  I don't think --mirror does what I'm looking for, because 
> (side effects aside) it only deals with branches, not remotes.
> 
> I find myself getting confused, and I think it's because the files in 
> .git/refs/remotes/ are indeed tracking branches on remote repositories. 
>   So I think our conversation gets a bit circular because our ideas of a 
> "remote" differ.

Yes, I think there are multiple uses:

- a remote repository (referenced to by an URL)
- a remote config (as created by git remote add)
- a remote branch (a branch in your local repo which is a copy of a
branch in a remote repo; stored under refs/remotes, never to be modified
locally)
- a (remote) tracking branch (a local branch which is set up to pull
from a remote branch by default)

Only the first of these 4 is something resides "remotely". Of course,
"remote URL" could be a path on the local file system or even ".".

> "git remote add" does two things (maybe more?): It adds a [remote] 
> section to the .git/config file, and it adds a branch reference in 
> .git/refs/remotes/. 

Yes.

> I think what I'd like is for the clone to be able 
> to obtain both these things from the origin.  The reason I think it's 
> useful is that it would let the clone pull directly from the origin's 
> remote repositories, without having to directly specify the remote 
> repository's URL and branch name.
> 
> Fundamentally, I'm looking to do exactly
> 
> 	clone/$ git pull -s subtree /path/to/ThingOne master
> 
> i.e. pull stuff from one of main's remotes directly into the clone.  But 
> I want to replace the "/path/to/ThingOne master" part with something 
> that means "use whatever URL and branch name was defined in the origin 
> for this remote".
> 
> My questions are:  Am I right in thinking this is desirable? 

I've got the strong impression you desire it...

Seriously, it seems desirable in cases where the remote URL changes
frequently; say, because the upstream maintainer (maintainer of
ThingOne) gets hit by a bus frequently, or walks the wrong streets in LA
at the wrong time of the day. (I guess my seriousness ended with the
";".) Clones would follow those changes transparently then (if that
feature existed).

> Is there 
> already some way to do this? 

None that I know of.

> If not, is it something worth 
> implementing?  (I'm happy to roll up my own sleeves here...)

First you should decide whether it is worth for you. Comments on the
list tend to kick in once people see a proposed implementation. A viable
strategy could be, mimicking git submodule behaviour in part:

- Implement "git remote export reponame" which takes an existing remote
config from .git/config, writes it to .gitremotes and checks in (or
better: stages for commit) the file .gitremotes [you would use this on main]

- Implement "git remote import" which reads the file .gitremotes and
adds remote config to .git/config. [you would use this on clones]

- Change "git remote update" to take updates to .gitremotes into account
before doing its usual routine (perhaps based on a config with default
off, or a command line option, or better both)

Downside is that .gitremotes is tracked would show up as a file in the
repo, but I can't come up with a better way which is as simple as the
above. .gitremotes could be stored in a specially named branch, though.

> I hope that clarifies things.  Sorry for taking so long to get here!

Don't worry. I take partial blame ;)
And thanks for trying to match up your workflow with git.

Cheers,
Michael

Terminology question: "tracking" branches

[Björn Steinbrink]

On 2008.10.22 16:59:58 +0200, Michael J Gruber wrote:
> - a remote branch (a branch in your local repo which is a copy of a
> branch in a remote repo; stored under refs/remotes, never to be modified
> locally)
> - a (remote) tracking branch (a local branch which is set up to pull
> from a remote branch by default)

(Remote) tracking branches are actually what you called remote branches,
at least according to the git glossary. But I wonder, what is the right
term for a branch that has the --track setup for pull?

On #git I usually fall back to some variation of "a branch that is
configured for 'git pull'" or something similarly verbose (and maybe
that's even partially wrong/inaccurate/incomplete?). And I always try to
stick to saying "remote tracking branch" and not just "tracking branch"
(as the glossary does) to avoid confusion as best as I can. But that
feels quite suboptimal.

So, is there some term that describes a local branch that has been
configured for "git pull"?

Thanks,
Björn

Re: Terminology question: "tracking" branches

[Michael J Gruber]

Björn Steinbrink venit, vidit, dixit 22.10.2008 18:13:
> On 2008.10.22 16:59:58 +0200, Michael J Gruber wrote:
>> - a remote branch (a branch in your local repo which is a copy of a
>> branch in a remote repo; stored under refs/remotes, never to be modified
>> locally)
>> - a (remote) tracking branch (a local branch which is set up to pull
>> from a remote branch by default)
> 
> (Remote) tracking branches are actually what you called remote branches,
> at least according to the git glossary. But I wonder, what is the right
> term for a branch that has the --track setup for pull?

Ooops, I'm sorry. I went by the name of the "--track" option and was
sure that that "git help branch" used the name "tracking branch" for the
branches created with "--track", but I was wrong. So:

"(remote) tracking branch" is a local branch stored under refs/remotes/
which is a "copy" of a branch on a remote repository.

That leaves open:

- What does "remote branch" mean, if it means anything at all? It could
be used for a branch in a remote repository, i.e. the other side of
fetch/push refspec (remote branch:tracking branch).

- How to name a local branch created with --track off of a (remote)
tracking branch? Local tracking branch? Downstream/work/modification branch?

I think that linguistically, the confusion comes from using the noun as
well as the adjective "remote". As an adjective: "remote something"
cleary is something residing remotely. As a noun it's the config added
by "git remote add".

Note that the glossary doesn't define remote at all. I'd volunteer
changing that once the discussion reaches a consensus.

My suggestion would be:

remote (noun): A configuration as created by "git remote add" which
points to a remote repository (using a URL) and sets up refspecs for
fetching and/or pulling. Note that the URL may point to a local
filesystem or even ".". See also remote (adjective).

remote (adjective): Anything residing in another repository. See also
remote (noun).

local: Sometimes used to emphasize things residing "locally", i.e. in
the repository at hand, as opposed to "remotely".

tracking branch: A branch tracking a (remote) branch on a remote
repository, i.e. a local branch stored under refs/remotes/ which is a
"copy" of a branch on a remote repository; typically created by "git
clone" or "git remote add/update".

remote tracking branch: Synonymous with tracking branch. [remote is a
noun, an object for track here]

[adjective to be found] branch: A local branch which is set up to pull
or rebase automatically from a tracking branch. Used for local
modifications to remote branches. [I'm tempted to use local tracking
branch here, but that would just add to the confusion.]

Come to think of it: "Off-track" branch would be the ideal name for a
local branch created off of a (remote) tracking branch, using --track to
specify the track where things go off. But I'm not sure everyone would
like the implied pun...

Cheers,
Michael

Re: Terminology question: "tracking" branches

[Marc Branchaud]

Michael J Gruber wrote:
> 
> remote tracking branch: Synonymous with tracking branch. [remote is a
> noun, an object for track here]

Er, "remote" is an adjective there...  :)

> [adjective to be found] branch: A local branch which is set up to pull
> or rebase automatically from a tracking branch. Used for local
> modifications to remote branches. [I'm tempted to use local tracking
> branch here, but that would just add to the confusion.]

I say there's no need for an adjective here, as this is just a plain old 
branch that git-branch creates by default (right?).

What's needed is an adjective for when git-branch is given the 
--no-track option (or when branch.automergesetup is false). 
"Non-tracking branch" perhaps?

		Marc

Re: Terminology question: "tracking" branches

[Björn Steinbrink]

On 2008.10.27 11:43:44 -0400, Marc Branchaud wrote:
> Michael J Gruber wrote:
>>
>> remote tracking branch: Synonymous with tracking branch. [remote is a
>> noun, an object for track here]
>
> Er, "remote" is an adjective there...  :)

Hm? It's a "branch that tracks (a branch of) a remote". Looks like a
noun to me.

>> [adjective to be found] branch: A local branch which is set up to pull
>> or rebase automatically from a tracking branch. Used for local
>> modifications to remote branches. [I'm tempted to use local tracking
>> branch here, but that would just add to the confusion.]
>
> I say there's no need for an adjective here, as this is just a plain old  
> branch that git-branch creates by default (right?).

By default, no. It depends on the branch.automergesetup configuration
and whether you create the branch from a remote tracking branch, a local
branch or some other committish.

autosetupmerge = false
----
git branch foo origin/foo   --> --no-track
git branch foo master       --> --no-track
git branch foo origin/foo^0 --> --no-track

autosetupmerge = true
----
git branch foo origin/foo   --> --track
git branch foo master       --> --no-track
git branch foo origin/foo^0 --> --no-track

autosetupmerge = always
----
git branch foo origin/foo   --> --track
git branch foo master       --> --track
git branch foo origin/foo^0 --> --no-track

> What's needed is an adjective for when git-branch is given the  
> --no-track option (or when branch.automergesetup is false).  
> "Non-tracking branch" perhaps?

Isn't that easily confused with "[remote] tracking branch"? Especially
since the "remote" is optional...

Björn

Re: Terminology question: "tracking" branches

[Björn Steinbrink]

On 2008.10.23 10:07:07 +0200, Michael J Gruber wrote:
> That leaves open:
> 
> - What does "remote branch" mean, if it means anything at all? It could
> be used for a branch in a remote repository, i.e. the other side of
> fetch/push refspec (remote branch:tracking branch).

I prefer to say "the branch on the remote" there, but that's just to
avoid confusion with "remote tracking branch".

> - How to name a local branch created with --track off of a (remote)
> tracking branch? Local tracking branch? Downstream/work/modification branch?

You can also have a local branch that "--track"s another local branch,
so those names look a bit suboptimal to me, but I don't have any better
ideas either :-/ For a second I thought about "pulling branch" but that
doesn't really describe it either, I guess...

> I think that linguistically, the confusion comes from using the noun as
> well as the adjective "remote". As an adjective: "remote something"
> cleary is something residing remotely. As a noun it's the config added
> by "git remote add".
> 
> Note that the glossary doesn't define remote at all. I'd volunteer
> changing that once the discussion reaches a consensus.
> 
> My suggestion would be:
> 
> remote (noun): A configuration as created by "git remote add" which
> points to a remote repository (using a URL) and sets up refspecs for
> fetching and/or pulling. Note that the URL may point to a local
> filesystem or even ".". See also remote (adjective).
> 
> remote (adjective): Anything residing in another repository. See also
> remote (noun).
> 
> local: Sometimes used to emphasize things residing "locally", i.e. in
> the repository at hand, as opposed to "remotely".
> 
> tracking branch: A branch tracking a (remote) branch on a remote
> repository, i.e. a local branch stored under refs/remotes/ which is a
> "copy" of a branch on a remote repository; typically created by "git
> clone" or "git remote add/update".
> 
> remote tracking branch: Synonymous with tracking branch. [remote is a
> noun, an object for track here]

Those look good, to me at least.

> [adjective to be found] branch: A local branch which is set up to pull
> or rebase automatically from a tracking branch.

"merge or rebase", and instead of "tracking" it should be "another" I
think. Again, because you can also --track a local branch.

> Used for local modifications to remote branches. [I'm tempted to use
> local tracking branch here, but that would just add to the confusion.]

Same here, it doesn't apply to "remote branches" only.

> Come to think of it: "Off-track" branch would be the ideal name for a
> local branch created off of a (remote) tracking branch, using --track to
> specify the track where things go off. But I'm not sure everyone would
> like the implied pun...

Haha :-)

thanks,
Björn

Re: Terminology question: "tracking" branches

[Johannes Schindelin]

Hi,

On Mon, 27 Oct 2008, Marc Branchaud wrote:

> Michael J Gruber wrote:
> > 
> > [adjective to be found] branch: A local branch which is set up to pull 
> > or rebase automatically from a tracking branch. Used for local 
> > modifications to remote branches. [I'm tempted to use local tracking 
> > branch here, but that would just add to the confusion.]
> 
> I say there's no need for an adjective here, as this is just a plain old
> branch that git-branch creates by default (right?).

I agree; I do not think that you need an adjective here.

I'd also recommend thinking about the branches as "local" and "remote".  
Not anything with "tracking" or "trailing" or somesuch.  When you really 
look at it from the non-technical view point, even the branches you have 
in refs/remotes/*/ are the remote branches; the fact that you have a local 
cached version of them is just a minor implementation detail.

And as to the "--track" thing?  I'd not make that big a deal out of it.  
At most, I'd say that a certain local branch happens to follow a certain 
remote branch, but that can change over time, right?  (Just change 
branch.<name>.merge...)

Ciao,
Dscho

Re: Working with remotes; cloning remote references

[Marc Branchaud]

Michael J Gruber wrote:
> 
> First you should decide whether it is worth for you.

I'm still trying to figure that out...

> Comments on the
> list tend to kick in once people see a proposed implementation. A viable
> strategy could be, mimicking git submodule behaviour in part:
> 
> - Implement "git remote export reponame" which takes an existing remote
> config from .git/config, writes it to .gitremotes and checks in (or
> better: stages for commit) the file .gitremotes [you would use this on main]
> 
> - Implement "git remote import" which reads the file .gitremotes and
> adds remote config to .git/config. [you would use this on clones]
> 
> - Change "git remote update" to take updates to .gitremotes into account
> before doing its usual routine (perhaps based on a config with default
> off, or a command line option, or better both)
> 
> Downside is that .gitremotes is tracked would show up as a file in the
> repo, but I can't come up with a better way which is as simple as the
> above. .gitremotes could be stored in a specially named branch, though.

That downside is a bit disappointing.  I might as well just make "git 
remote export" simply generate a script of "git remote add" commands 
based on the contents of .git/config, and check that script in.  Then I 
could run the script in a clone to recreate the origin's remotes.

It also seems awkward to have an export step in the origin repository. 
I don't really see a need for an export step (except as an artifact of 
the above implementation).

It seems to me that this would be more natural if our hypothetical "git 
remote import <X>" could just grab the remotes from repository <X> (or 
the origin, if <X> is unspecified).  I assume that would involve 
lower-level changes than what you described, but to me it seems like the 
more usable approach.  (But then I know nothing of Git's internals, so 
maybe this kind of change would be too much work?)

		Marc

Re: Terminology question: "tracking" branches

[Björn Steinbrink]

On 2008.10.27 17:28:40 +0100, Björn Steinbrink wrote:
> On 2008.10.23 10:07:07 +0200, Michael J Gruber wrote:
> > That leaves open:
> > 
> > - What does "remote branch" mean, if it means anything at all? It could
> > be used for a branch in a remote repository, i.e. the other side of
> > fetch/push refspec (remote branch:tracking branch).
> 
> I prefer to say "the branch on the remote" there, but that's just to
> avoid confusion with "remote tracking branch".

So I just happened to use "git remote show origin" and that uses "remote
branch" as you described it:

* remote origin
  URL: git://git.kernel.org/pub/scm/git/git.git
  Remote branch merged with 'git pull' while on branch master
    master
  Tracked remote branches
    html
    maint
    ...

JFYI
Björn

Re: Working with remotes; cloning remote references

[Michael J Gruber]

Marc Branchaud venit, vidit, dixit 27.10.2008 20:54:
> Michael J Gruber wrote:
>> Downside is that .gitremotes is tracked would show up as a file in the
>> repo, but I can't come up with a better way which is as simple as the
>> above. .gitremotes could be stored in a specially named branch, though.
> 
> That downside is a bit disappointing.  I might as well just make "git 
> remote export" simply generate a script of "git remote add" commands 
> based on the contents of .git/config, and check that script in.  Then I 
> could run the script in a clone to recreate the origin's remotes.

Yes, if you feel okay about running a script repeatedly which may change
due to being versioned; rather than running a script/command which uses
versioned input.

> It also seems awkward to have an export step in the origin repository. 
> I don't really see a need for an export step (except as an artifact of 
> the above implementation).

I had the impression that you want to configure as much as possible on
the "central", and have clones follow automatically. It's very likely
that you want your clones to see only a subset of central's remotes.
Both (individually) imply that there has to be a step on central which
determines which (if any) central remotes appear in clones automatically.

> It seems to me that this would be more natural if our hypothetical "git 
> remote import <X>" could just grab the remotes from repository <X> (or 
> the origin, if <X> is unspecified).  I assume that would involve 
> lower-level changes than what you described, but to me it seems like the 
> more usable approach.  (But then I know nothing of Git's internals, so 
> maybe this kind of change would be too much work?)

Feel free to hack the protocol and remote commands... Implementing
anything which exposes "server's" .git/config on the client without
interaction on the server side will most probably be rejected, though.
My suggestions were meant to be a minimal effort attempt within and
following (see submodules) current infrastructure. I guess now it's up
to you to pick the approach you deem most appropriate in your scenario
and follow through with it.

Michael

Re: Working with remotes; cloning remote references

[Marc Branchaud]

Michael J Gruber wrote:
>
>> That downside is a bit disappointing.  I might as well just make "git 
>> remote export" simply generate a script of "git remote add" commands 
>> based on the contents of .git/config, and check that script in.  Then I 
>> could run the script in a clone to recreate the origin's remotes.
> 
> Yes, if you feel okay about running a script repeatedly which may change
> due to being versioned; rather than running a script/command which uses
> versioned input.

Either way it's a two-step process: "git pull; git remote import" vs. 
"git pull; ./import-remotes.sh".

>> It also seems awkward to have an export step in the origin repository. 
>> I don't really see a need for an export step (except as an artifact of 
>> the above implementation).
> 
> I had the impression that you want to configure as much as possible on
> the "central", and have clones follow automatically. It's very likely
> that you want your clones to see only a subset of central's remotes.
> Both (individually) imply that there has to be a step on central which
> determines which (if any) central remotes appear in clones automatically.

I actually don't care to limit what the clones can see.  I understand 
the reasoning behind such control, but in my case it just doesn't apply.

>> It seems to me that this would be more natural if our hypothetical "git 
>> remote import <X>" could just grab the remotes from repository <X> (or 
>> the origin, if <X> is unspecified).  I assume that would involve 
>> lower-level changes than what you described, but to me it seems like the 
>> more usable approach.  (But then I know nothing of Git's internals, so 
>> maybe this kind of change would be too much work?)
> 
> Feel free to hack the protocol and remote commands... Implementing
> anything which exposes "server's" .git/config on the client without
> interaction on the server side will most probably be rejected, though.

Point taken, but I think I will start with no origin-side controls, 
since I have no need for them anyway.  If/when I cobble together a patch 
hopefully at that point we'll be able to have a wider discussion about 
such controls.

> My suggestions were meant to be a minimal effort attempt within and
> following (see submodules) current infrastructure. I guess now it's up
> to you to pick the approach you deem most appropriate in your scenario
> and follow through with it.

Many thanks for taking the time to discuss this.  Now I've got to roll 
up my sleeves!

(BTW, any pointers to descriptions or overviews of git's code base?)

		Marc