Question on git fetch to bare repo

Question on git fetch to bare repo

[Bill Lear]

If I have not yet made myself unwelcome, I now have another problem
using git 1.4.4.1.

I have a public bare repo I created yesterday:

% mkdir project && cd project
% git --bare init-db --shared
% git --bare fetch git://source/project
[All seems well]

I have a private repo:

% mkdir project && cd project
% git clone /repos/git/project
[All is well]

A co-worker checks something in to our company repo, so I go to my
public repo to fetch the changes:

% cd /repos/git/project
% git --bare fetch -v git://source/project
remote: Generating pack...
remote: Done counting 230 objects.
remote: Result has 152 objects.
remote: Deltifying 152 objects.
remote:  100% (152/152) done
Unpacking 152 objects
remote: Total 152, written 152 (delta 109), reused 90 (delta 51)
 100% (152/152) done
* fetched git://source/project
  commit: 5c2d43d

I then go to my private repo to pull from my public one:

% cd ~/project
% git branch
  topic
* master
% git pull
Already up-to-date.
% cat .git/remotes/origin
URL: /repos/git/project
Pull: refs/heads/master:refs/heads/origin
Pull: refs/heads/topic:refs/heads/topic
[All seems well with this repo ??]

I try to push from my private to my public:

% git push /repos/git/project
Everything up-to-date

I go back to my public repo, and poking around, can't see what is wrong.

The commit that the fetch says it pulled is there, if I do:

% cd /repos/git/project
% git --bare show -t 5c2d43d

the patch comes out exactly as it did in my email notification, so I
know it's there.

I tried then to be more forceful, and did another fetch like this,
which I sort of expected to fail:

% git --bare fetch -v git://source/project master:master
remote: Generating pack...
remote: Done counting 230 objects.
remote: Result has 152 objects.
remote: Deltifying 152 objects.
remote:  100% (152/152) done
Unpacking 152 objects
remote: Total 152, written 152 (delta 109), reused 90 (delta 51)
 100% (152/152) done
* refs/heads/master: fast forward to branch 'master' of git://source/project
  old..new: 37e2298..5c2d43d
Cannot fetch into the current branch.

Having enabled the logs, I can investigate a bit:

% cat logs/refs/heads/master
37e229835103a11365b1e081f9b9987a88437e62 5c2d43dc819fc1bc37ebae1696c3fbfd6a4401db Bill Lear <rael@zopyra.com> 1170973321 -0600     fetch git://source/project: fast-forward
5c2d43dc819fc1bc37ebae1696c3fbfd6a4401db 37e229835103a11365b1e081f9b9987a88437e62 Bill Lear <rael@zopyra.com> 1170973321 -0600     fetch git://source/project: Undoing incorrectly fetched HEAD.

And, my branches seem to be there, properly:

% git --bare branch
  topic
* master

So, I'm confused.  Why does my fetch seem to fetch things the first
time, yet I cannot pull these into my private repo?  I could swear I
just did this same sequence of operations on Monday and it worked.

Perhaps I just need a vacation ...


Bill

Re: Question on git fetch to bare repo

[Jakub Narebski]

Bill Lear wrote:

> I have a public bare repo I created yesterday:
> 
> % mkdir project && cd project
> % git --bare init-db --shared
> % git --bare fetch git://source/project
> [All seems well]

_Seems_ well

  % git clone --bare --shared git://source/project project

is a proper invocation.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: Question on git fetch to bare repo

[Johannes Schindelin]

Hi,

[Jakub, you broke the thread. I Cc' Bill, but I cannot join the thread]

On Thu, 8 Feb 2007, Jakub Narebski wrote:

> Bill Lear wrote:
> 
> > I have a public bare repo I created yesterday:
> > 
> > % mkdir project && cd project
> > % git --bare init-db --shared
> > % git --bare fetch git://source/project
> > [All seems well]
> 
> _Seems_ well
> 
>   % git clone --bare --shared git://source/project project
> 
> is a proper invocation.

No.

init --shared means that the repository is set up such that different 
users can update branches.

clone --shared means that if the original repository is on the local 
machine (which it is not in your example), then that is used as an 
alternate, i.e. the objects are not copied at all, but reused from the 
other location.

I complained about that ambiguity, but I am probably the guilty person: 
AFAIR it was me who introduced "init --shared".

Ciao,
Dscho

Re: Question on git fetch to bare repo

[Johannes Schindelin]

Hi,

On Fri, 9 Feb 2007, Johannes Schindelin wrote:

> [Jakub, you broke the thread. I Cc' Bill, but I cannot join the thread]

Sorry, PEBCAK. The thread is not broken, but you -- again -- did not reply 
to the original poster.

Ciao,
Dscho

Re: Question on git fetch to bare repo

[Jeff King]

On Thu, Feb 08, 2007 at 04:28:29PM -0600, Bill Lear wrote:

> % mkdir project && cd project
> % git --bare init-db --shared
> % git --bare fetch git://source/project
> [All seems well]

It's not. You have fetched the objects from git://source/project, and
FETCH_HEAD (an ephemeral pointer to the things you just fetched) points
to them. But look in project/refs/heads; you have no branches!

What you want instead is:
  git --bare fetch git://source/project master:master
which means
  "fetch project's master and store it as my master; if it's not a
  fast-forward, then complain loudly. Don't do any merges."

> I have a private repo:
> 
> % mkdir project && cd project
> % git clone /repos/git/project
> [All is well]

Is it? You shouldn't have any branches in /repos/git/project at this
point.

> A co-worker checks something in to our company repo, so I go to my
> public repo to fetch the changes:
> 
> % cd /repos/git/project
> % git --bare fetch -v git://source/project
> remote: Generating pack...
> remote: Done counting 230 objects.
> remote: Result has 152 objects.
> remote: Deltifying 152 objects.
> remote:  100% (152/152) done
> Unpacking 152 objects
> remote: Total 152, written 152 (delta 109), reused 90 (delta 51)
>  100% (152/152) done
> * fetched git://source/project
>   commit: 5c2d43d

Again, you're not _storing_ those changes in a branch, you're just
putting them in FETCH_HEAD.

> I then go to my private repo to pull from my public one:
> 
> % cd ~/project
> % git branch
>   topic
> * master
> % git pull
> Already up-to-date.

Not surprising, since you didn't actually update any branches in the
previous step.

> % cat .git/remotes/origin
> URL: /repos/git/project
> Pull: refs/heads/master:refs/heads/origin
> Pull: refs/heads/topic:refs/heads/topic
> [All seems well with this repo ??]

I'm confused as to how there are branches in the 'public' repo at this
step, since your initial fetch shouldn't have actually made any,
especially not a 'topic' branch.

> % cd /repos/git/project
> % git --bare show -t 5c2d43d
> 
> the patch comes out exactly as it did in my email notification, so I
> know it's there.

Sure, you have the object, but you don't have any _pointers_ to it.

> I tried then to be more forceful, and did another fetch like this,
> which I sort of expected to fail:
> 
> % git --bare fetch -v git://source/project master:master
> remote: Generating pack...
> remote: Done counting 230 objects.
> remote: Result has 152 objects.
> remote: Deltifying 152 objects.
> remote:  100% (152/152) done
> Unpacking 152 objects
> remote: Total 152, written 152 (delta 109), reused 90 (delta 51)
>  100% (152/152) done
> * refs/heads/master: fast forward to branch 'master' of git://source/project
>   old..new: 37e2298..5c2d43d
> Cannot fetch into the current branch.

You're more on the right track here, but not quite. If you want to save
things in branches, you either need to use a 'remotes' shorthand (which
defines pull lines with refspecs) or you need to specify the refspec on
the command line (like master:master). However, you _don't_ want to
fetch directly into your master branch. fetch is for copying refs and
objects, not for merging (and you've presumably made some commits on
master that the upstream doesn't have).

It seems like you're trying to merge in your "public" repository, which
is a mistake. I think you would be much better served to think of your
public repository as a place where you publish changes (_only_ from your
private repository), and do all of your external fetching and merging in
your private repository. Everyone has a public repo, so you always
"pull" from other people's public repos, and "push" into your public
repo. Thus you don't need --shared at all.

IOW, do this:

# set up public repository; initially empty, but we will push something
# useful into it soon.
mkdir /git/repo/project && cd /git/repo/project
git --bare init

# set up our private repository, which is a clone of the company repo
cd $HOME
git clone git://source/project

# at this point, our .git/remotes/origin file is set up to pull from
# the company repo. But we still want to publish our changes in our
# personal public repo. Let's set that up. We always want the public
# branches to match ours, even if we've reset or rebased, so we use '+'
# to always overwrite. This is safe, because data is never getting into
# the public repo in any way _except_ for us pushing it.
cat >.git/remotes/publish <<'EOF'
URL: /git/repo/project
Push: +master:master
Push: +topic:topic

# now we can make our first publication, which at this point is the same
# as the source repo
git push publish

# and now we make some changes
hack hack hack
git commit

# and we can publish more changes
git push publish

# but now we want to grab some changes from Bob's public repo. Let's set
# up a new remote for him. Again, we use '+' to overwrite, since we are
# just tracking what Bob is doing. Note that we have Bob's branch now,
# but we don't _publish_ it, since it's not in our publish remote.
cat >.git/remotes/bob <<'EOF'
URL: /path/to/bobs/repo
Pull: +master:bob-master
EOF

# and now we can fetch/pull from bob
git fetch bob
# what has bob done that we haven't?
gitk bob-master..master
# ok, let's merge
git pull . bob-master
# or do the fetch/pull in one step
git pull bob master

And Bob can of course do the same to us.

Does that make more sense? Or have I completely missed what you are
trying to accomplish? :)

-Peff

Re: Question on git fetch to bare repo

[Bill Lear]

On Thursday, February 8, 2007 at 21:39:41 (-0500) Jeff King writes:
>...
>Does that make more sense? Or have I completely missed what you are
>trying to accomplish? :)

No, you haven't misunderstood.  This gives me yet more options, so
thank you very much.


Bill

Re: Question on git fetch to bare repo

[Bill Lear]

On Thursday, February 8, 2007 at 21:39:41 (-0500) Jeff King writes:
>...
>You're more on the right track here, but not quite. If you want to save
>things in branches, you either need to use a 'remotes' shorthand (which
>defines pull lines with refspecs) or you need to specify the refspec on
>the command line (like master:master). However, you _don't_ want to
>fetch directly into your master branch. fetch is for copying refs and
>objects, not for merging (and you've presumably made some commits on
>master that the upstream doesn't have).
>
>It seems like you're trying to merge in your "public" repository, which
>is a mistake. I think you would be much better served to think of your
>public repository as a place where you publish changes (_only_ from your
>private repository), and do all of your external fetching and merging in
>your private repository. Everyone has a public repo, so you always
>"pull" from other people's public repos, and "push" into your public
>repo. Thus you don't need --shared at all.

Ok, I've gotten some time to read through this.  Kinda sad that
something published 4 days ago seems like the distant past ...

The problem I have with doing all of my fetching and merging in my
private repo is this: I have an update hook in my public repo that I
use to communicate my changes to my peers.  The problem is when I pull
from a peer's repo into my private repo, make some of my changes, and
then publish (push) my changes to the public repo, HIS changes are
pushed as well, and the update script naturally picks up on these and
broadcasts them.  My peer group ends up getting the same update
message about his commits that they have already received.  Multiply
this among 6 peers and it becomes a real headache.

On the other hand, if I fetch his changes into my public (bare) repo
first, and then pull from there into my private, make changes in my
private and then commit and publish to my public, the update script
will send out my changes --- and only MY changes --- to my peers.

So, what I have (just now) tried to do (using the latest 1.5 code),
is clone my public repo to create my private repo:

% git clone /repos/git/project

and my .git/config file is now:

% cat .git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = /repos/git/project
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master

and no .git/remotes/origin exists.

I notice that this did not "clone" all the branches in my public repo:

% git branch
* master

whereas in my public repo:

% cd /repos/git/project
% git --bare branch
  topic
* master

So, regardless if I can correct that (very confused as to why the
clone did not grab the branch), it seems that I did not make my intent
clear...

I'm still not sure if I'm obeying the "don't develop on anything
on the RHS of ':' dictum".

I'm going to puzzle over the rest of your advice to see if I can
grok it and fold it into the new (1.5) way of life.


Bill

Re: Question on git fetch to bare repo

[Jeff King]

On Mon, Feb 12, 2007 at 02:47:02PM -0600, Bill Lear wrote:

> The problem I have with doing all of my fetching and merging in my
> private repo is this: I have an update hook in my public repo that I
> use to communicate my changes to my peers.  The problem is when I pull
> from a peer's repo into my private repo, make some of my changes, and
> then publish (push) my changes to the public repo, HIS changes are
> pushed as well, and the update script naturally picks up on these and
> broadcasts them.  My peer group ends up getting the same update
> message about his commits that they have already received.  Multiply
> this among 6 peers and it becomes a real headache.

Interesting. Why not have your update hook know who you are, and send
out changes only for commits that are either authored by you, or
committed by you (depending on your workflow, these may have different
results)?

Something like this (mostly untested!) on top of the stock update hook:

diff --git a/templates/hooks--update b/templates/hooks--update
index d4253cb..1598f1b 100644
--- a/templates/hooks--update
+++ b/templates/hooks--update
@@ -55,6 +55,7 @@ projectdesc=$(cat $GIT_DIR/description)
 recipients=$(git-repo-config hooks.mailinglist)
 announcerecipients=$(git-repo-config hooks.announcelist)
 allowunannotated=$(git-repo-config --bool hooks.allowunannotated)
+authorfilter=$(git-repo-config hooks.authorfilter)
 
 # --- Check types
 newrev_type=$(git-cat-file -t "$newrev")
@@ -148,7 +149,7 @@ case "$refname_type" in
 			# This shows all log entries that are not already covered by
 			# another ref - i.e. commits that are now accessible from this
 			# ref that were previously not accessible
-			git-rev-list --pretty $newref $(git-rev-parse --not --all)
+			git-rev-list ${authorfilter:+--author="$authorfilter"} --pretty $newref $(git-rev-parse --not --all)
 			echo $LOGEND
 		else
 			# oldrev is valid
@@ -165,7 +166,7 @@ case "$refname_type" in
 			baserev=$(git-merge-base $oldrev $newrev)
 
 			# Commit with a parent
-			for rev in $(git-rev-list $newrev ^$baserev)
+			for rev in $(git-rev-list ${authorfilter:+--author="$authorfilter"} $newrev ^$baserev)
 			do
 				revtype=$(git-cat-file -t "$rev")
 				echo "       via  $rev ($revtype)"
@@ -190,7 +191,7 @@ case "$refname_type" in
 			fi
 			echo ""
 			echo $LOGBEGIN
-			git-rev-list --pretty $newrev ^$baserev
+			git-rev-list ${authorfilter:+--author="$authorfilter"} --pretty $newrev ^$baserev
 			echo $LOGEND
 			echo ""
 			echo "Diffstat:"


Just set hooks.authorfilter in your config to 'Bill Lear'.  Note that
this still gives the _full_ diffstat between the two endpoints. I would
think you would really want to show a diffstat for each filtered commit
individually. And you're probably not using this hook currently, but I
hope it should be obvious how to modify whatever you are using.

Of course, what you're doing now isn't _wrong_, once we fix the
"committing on tracking branches" problem (which it looks like you are
addressing below), so don't let me drag you too far from your workflow.
:)

> So, what I have (just now) tried to do (using the latest 1.5 code),

Good, 1.5 is much more pleasant to work with. :)

> [remote "origin"]
>         url = /repos/git/project
>         fetch = +refs/heads/*:refs/remotes/origin/*

So this means everything in your public repo's refs/heads/* is mirrored
in your private refs/remotes/origin/*.

> [branch "master"]
>         remote = origin
>         merge = refs/heads/master

And the local branch master will merge from origin's refs/heads/master.

> and no .git/remotes/origin exists.

Right, the config above supersedes it.

> I notice that this did not "clone" all the branches in my public repo:
> 
> % git branch
> * master
> 
> whereas in my public repo:
> 
> % cd /repos/git/project
> % git --bare branch
>   topic
> * master

It did clone it; git-branch just doesn't print remote branches by
default. Try 'git branch -a'. You can create a new topic branch from
your origin topic branch like this:

  git checkout -b topic origin/topic

If you want it to pull automagically from the upstream topic (when you
do a git-pull without any arguments), then you need a config similar to
what it automatically set up for master:

  [branch "topic"]
    remote = origin
    merge = refs/heads/topic

> I'm still not sure if I'm obeying the "don't develop on anything
> on the RHS of ':' dictum".

You are, because 1.5 makes it much harder to do so. Doing a
'git-checkout topic' won't work until you create a local topic branch.
With the detached head work in 1.5, you _can_ do this:

  git checkout origin/topic

but it will issue a warning.

-Peff

Re: Question on git fetch to bare repo

[Junio C Hamano]

Bill Lear <rael@zopyra.com> writes:

> The problem I have with doing all of my fetching and merging in my
> private repo is this: I have an update hook in my public repo that I
> use to communicate my changes to my peers.  The problem is when I pull
> from a peer's repo into my private repo, make some of my changes, and
> then publish (push) my changes to the public repo, HIS changes are
> pushed as well, and the update script naturally picks up on these and
> broadcasts them.  My peer group ends up getting the same update
> message about his commits that they have already received.  Multiply
> this among 6 peers and it becomes a real headache.

I suspect that is because your "email when pushed" hook lists
what commits are new on the branch.  I do not use any "email
when pushed" hook myself, so I do not know how yours is set up,
but I suspect it is doing a moral equivalent of:

	#!/bin/sh
	name=<name of the branch the email will talk about>
	old=<commit before this push updates the branch tip>
        new=<commit this push is trying to update the branch tip with>

	git shortlog $old..$new |
        mailx -s "branch $name updated" recipients

That behaviour might be desirable when branches in the
repositories are more or less independent, but in situations
like yours where commit are cross pushed into each other's
branches, you are not necessarily interested in the progress of
each branch.

One solution would be to list only the new commits introduced by
the push into the repository, regardless of the branch.  You can
replace the "git-shortlog $old..$new" with:

	git shortlog $new --not --all

to get that information, I think.

Re: Question on git fetch to bare repo

[Jeff King]

On Mon, Feb 12, 2007 at 09:28:10PM -0800, Junio C Hamano wrote:

> One solution would be to list only the new commits introduced by
> the push into the repository, regardless of the branch.  You can
> replace the "git-shortlog $old..$new" with:
> 
> 	git shortlog $new --not --all
> 
> to get that information, I think.

I don't think that addresses his concern. I had suggested to him that he
do all pulling/merging in his private repo, then simply keep the public
repo as a mirror. In that case, when pushing out to the public repo,
_all_ commits will likely be new to the public repo, including those
ones that he pulled from elsewhere.

-Peff