Git fast-import : Warning duplicated ref: refs/remotes/origin/master

Git fast-import : Warning duplicated ref: refs/remotes/origin/master

[Jérémie NIKAES]

Hi,
I am currently working on the git-mediawiki gateway project.
This project uses fast-import to get data from mediawiki. For each
revision from mediawiki, we generate a chunk of fast-export stream
looking like this :

commit refs/remotes/origin/master
mark :<int>
commiter <user> <address> <timestamp> +0000
data <sizeofcomment>
<comment>
M 644 inline <title>.wiki
data <sizeoffile>
<content>

At the end of a git clone mediawiki::http://ourwikitest, everything
looks perfectly fine, including the git log, but a "Warning duplicated
ref: refs/remotes/origin/master" is thrown. And indeed, it is
duplicated in .git/packedrefs

A git gc solves everything, but I wondered if there was a way to fix
this or if this was a known bug.

Thank you,
Regards,

--
Jérémie Nikaes

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

[Jonathan Nieder]

Hi Jérémie,

Jérémie NIKAES wrote:

> At the end of a git clone mediawiki::http://ourwikitest, everything
> looks perfectly fine

I just wanted to say, thank you for working on this[1].

This code answers some questions I had about the remote helper
protocol, among other things.

> a "Warning duplicated
> ref: refs/remotes/origin/master" is thrown.

To end the suspense: I assume this was fixed by 9055039c (Bug fix for
pulls with multiple revisions pulled, 2011-05-29).

Cheers,
Jonathan

[1] git://github.com/Bibzball/Git-Mediawiki.git

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

[Jérémie NIKAES]

Jonathan,

> I just wanted to say, thank you for working on this[1].
>
> This code answers some questions I had about the remote helper
> protocol, among other things.

Glad we can help !

>
>> a "Warning duplicated
>> ref: refs/remotes/origin/master" is thrown.
>
> To end the suspense: I assume this was fixed by 9055039c (Bug fix for
> pulls with multiple revisions pulled, 2011-05-29).

No, I actually still have the problem. The version commited on github
is pulling revisions straight to refs/heads/master which is gross
After discussing this matter with our teacher Matthieu Moy, I wanted
to change this to refs/remotes/origin/master but then this warning
gets thrown.

Does anyone have an idea ?

Thanks,
Cheers,
-- 
Jérémie Nikaes

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

[Jonathan Nieder]

Jérémie NIKAES wrote:
>> Jérémie NIKAES wrote:

>>> a "Warning duplicated
>>> ref: refs/remotes/origin/master" is thrown.
[...]
> No, I actually still have the problem. The version commited on github
> is pulling revisions straight to refs/heads/master which is gross
> After discussing this matter with our teacher Matthieu Moy, I wanted
> to change this to refs/remotes/origin/master but then this warning
> gets thrown.

Could you push a branch exhibiting the problem?  The reason I ask
is that searching with

	git grep -F -e 'duplicated ref' origin/master

in git.git yields no hits for me, so I wanted to reproduce this to get
the exact error message or figure out which program is printing it.

Thanks.

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

[Jérémie NIKAES]

>
> Could you push a branch exhibiting the problem?  The reason I ask
> is that searching with
>
>        git grep -F -e 'duplicated ref' origin/master
>
> in git.git yields no hits for me, so I wanted to reproduce this to get
> the exact error message or figure out which program is printing it.
>
> Thanks.
>

Ok I just committed my changes to the master branch, feel free to pull
and try it out.

Thanks for your help
-- 
Jérémie Nikaes

[PATCH] Documentation: do not misinterpret refspecs as bold text (Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master)

[Jonathan Nieder]

Jérémie NIKAES wrote:

> Ok I just committed my changes to the master branch, feel free to pull
> and try it out.

Nice.  Now I get

| $ git fetch --all
[...]
| Fetching & writing export data...
| 1/2: Revision n°7 of New_page
| 2/2: Revision n°8 of New_page
| 
| error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
| From mediawiki::http://localhost/mediawiki
|  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
| error: Could not fetch origin

which means that the transport machinery thought the helper was going
to be fetching directly to "master".  I suspect you will want a
'refspec' capability like

	refspec refs/heads/*:refs/mediawiki/${remotename}/*

to fix this.

Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map
private ref names into normal names, 2009-11-18).  What namespace
should a helper use when asked to fetch to FETCH_HEAD without a remote
name, like

	git fetch mediawiki::testwiki

?

-- >8 --
Subject: Documentation: do not misinterpret refspecs as bold text

In v1.7.3.3~2 (Documentation: do not misinterpret pull refspec as bold
text, 2010-12-03) many uses of asterisks in expressions like
"refs/heads/*:refs/svn/origin/branches/*" were escaped as {asterisk}
to avoid being treated as delimiters for bold text, but these two were
missed.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/git-remote-helpers.txt |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt
index 87cd11f..58f6ad4 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -181,11 +181,11 @@ CAPABILITIES
 	When using the import command, expect the source ref to have
 	been written to the destination ref. The earliest applicable
 	refspec takes precedence. For example
-	"refs/heads/*:refs/svn/origin/branches/*" means that, after an
-	"import refs/heads/name", the script has written to
+	"refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}" means
+	that, after an "import refs/heads/name", the script has written to
 	refs/svn/origin/branches/name. If this capability is used at
 	all, it must cover all refs reported by the list command; if
-	it is not used, it is effectively "*:*"
+	it is not used, it is effectively "{asterisk}:{asterisk}"
 
 REF LIST ATTRIBUTES
 -------------------
-- 
1.7.5.3

remote helpers: best practices for using the "refspec" capability

[Jonathan Nieder]

Jonathan Nieder wrote:

> | error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
> | From mediawiki::http://localhost/mediawiki
> |  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
> | error: Could not fetch origin
>
> which means that the transport machinery thought the helper was going
> to be fetching directly to "master".  I suspect you will want a
> 'refspec' capability like
>
>	refspec refs/heads/*:refs/mediawiki/${remotename}/*
>
> to fix this.
>
> Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map
> private ref names into normal names, 2009-11-18).  What namespace
> should a helper use when asked to fetch to FETCH_HEAD without a remote
> name, like
>
>	git fetch mediawiki::testwiki
>
> ?

Actually cc-ing this time.  Sorry for the noise.

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

[Jonathan Nieder]

Jérémie NIKAES wrote:

> No, I actually still have the problem. The version commited on github
> is pulling revisions straight to refs/heads/master which is gross
> After discussing this matter with our teacher Matthieu Moy, I wanted
> to change this to refs/remotes/origin/master but then this warning
> gets thrown.

The following fixes it for me.

-- >8 --
Subject: use refs/mediawiki/<remote> namespace for private refs

Unless the "refspec" capability is used, the remote helper machinery
expects remote helpers to write remote refs/heads/master ref to
refs/heads/master.  That's not very good since it scribbles on top of
whatever local changes the user might have.

So specify a refspec that puts the refs elsewhere.  For now this
only works when fetching from a named remote, like so:

	git remote add origin mediawiki::testwiki
	git fetch origin

Attempting to fetch directly from a URL with a command like
"git fetch mediawiki::testwiki" produces an "Invalid refspec"
error.
---
 git-remote-mediawiki |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/git-remote-mediawiki b/git-remote-mediawiki
index af66b58..7a3fdd3 100755
--- a/git-remote-mediawiki
+++ b/git-remote-mediawiki
@@ -7,6 +7,7 @@ use Storable qw(freeze thaw);
 use DateTime::Format::ISO8601;
 use Encode qw(encode_utf8);
 
+my $remotename = $ARGV[0];
 my $url = $ARGV[1];
 
 print STDERR "$url\n";
@@ -62,6 +63,7 @@ while ($loop) {
 
 sub mw_capabilities {
 #	print STDOUT "fetch\n";
+	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
 	print STDOUT "import\n";
 	print STDOUT "list\n";
 	print STDOUT "option\n";
@@ -91,7 +93,7 @@ sub mw_import {
 	
 	sub get_last_revision {
 		# Get last commit sha1
-		my $commit_sha1 = `git rev-parse refs/remotes/origin/master 2>/dev/null`;
+		my $commit_sha1 = `git rev-parse refs/mediawiki/$remotename/master 2>/dev/null`;
 
 		# Get note regarding that commit
 		chomp($commit_sha1);
@@ -202,15 +204,15 @@ sub mw_import {
 		
 		# If it's a clone and the first revision, initalize this ref
 		if ($fetch_from == 1 && $n == 1) {
-			#print "reset refs/remotes/origin/master\n";
+			#print "reset refs/mediawiki/$remotename/master\n";
 		}
-		print "commit refs/remotes/origin/master\n";
+		print "commit refs/mediawiki/$remotename/master\n";
 		print "mark :$n\n";
 		print "committer $user <$user\@$wiki_name> ", $dt->epoch, " +0000\n";
 		print "data ", bytes::length(encode_utf8($comment)), "\n", encode_utf8($comment);
 		# If it's not a clone, needs to know where to start from
 		if ($fetch_from != 1 && $n == 1) {
-			print "from refs/remotes/origin/master^0\n";
+			print "from refs/mediawiki/$remotename/master^0\n";
 		}
 		print "M 644 inline $title.wiki\n";
 		print "data ", bytes::length(encode_utf8($content)), "\n", encode_utf8($content);
-- 
1.7.5.3

Re: remote helpers: best practices for using the "refspec" capability

[Daniel Barkalow]

On Mon, 30 May 2011, Jonathan Nieder wrote:

> Jonathan Nieder wrote:
> 
> > | error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
> > | From mediawiki::http://localhost/mediawiki
> > |  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
> > | error: Could not fetch origin
> >
> > which means that the transport machinery thought the helper was going
> > to be fetching directly to "master".  I suspect you will want a
> > 'refspec' capability like
> >
> >	refspec refs/heads/*:refs/mediawiki/${remotename}/*
> >
> > to fix this.
> >
> > Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map
> > private ref names into normal names, 2009-11-18).  What namespace
> > should a helper use when asked to fetch to FETCH_HEAD without a remote
> > name, like
> >
> >	git fetch mediawiki::testwiki
> >
> > ?

The main purpose of having the private ref names is to support incremental 
imports, where you obviously need to identify the sha1 of the last thing 
you imported, and you may have included sufficient machine-readable 
information in your imported commit messages to know what the foreign 
state was that generated that sha1. (And you need a namespace because you 
may have multiple refs that you imported.) I'm not sure that it makes much 
sense to do incremental imports without a remote name, since that's 
generally an operation you're not planning do repeatedly.

But the purpose of the namespacing is to be able to continue the correct 
incremental import, so it would make sense to do some arbitrary 
transformation to make your url be a valid ref directory, and use that.

It would make sense to add support for a namespace where fast-import can 
write whatever it wants, and it'll get discarded after the fetch is done, 
if it's the case that anyone can stand not having incremental imports.

	-Daniel
*This .sig left intentionally blank*

Re: remote helpers: best practices for using the "refspec" capability

[Sverre Rabbelier]

Heya,

On Mon, May 30, 2011 at 12:43, Daniel Barkalow <barkalow@iabervon.org> wrote:
> But the purpose of the namespacing is to be able to continue the correct
> incremental import, so it would make sense to do some arbitrary
> transformation to make your url be a valid ref directory, and use that.
>
> It would make sense to add support for a namespace where fast-import can
> write whatever it wants, and it'll get discarded after the fetch is done,
> if it's the case that anyone can stand not having incremental imports.

I remember a patch series by someone else a lot more that _disabled_
creating of refs during fast-import, and instead allowed for
specifying what refs to update with which values at the end of the
import.

-- 
Cheers,

Sverre Rabbelier

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

[Matthieu Moy]

Jonathan Nieder <jrnieder@gmail.com> writes:

> Jérémie NIKAES wrote:
>
>> No, I actually still have the problem. The version commited on github
>> is pulling revisions straight to refs/heads/master which is gross
>> After discussing this matter with our teacher Matthieu Moy, I wanted
>> to change this to refs/remotes/origin/master but then this warning
>> gets thrown.
>
> The following fixes it for me.

It seems it does, thanks!

clone does not give the warning anymore, and incremental pull work.

>  sub mw_capabilities {
>  #	print STDOUT "fetch\n";
> +	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";

Is this "mediawiki" comming from the URL (mediawiki::...), or is it just
a convention you've set?

We've tried with refspec refs/heads/*:refs/remotes/origin/*, but without
success.

Do I understand correctly that the "pull" is done in 3 stages:

1) import into refs/mediawiki/origin

2) fetch the imported ref into refs/remotes/origin

3) and merge as usual into the current branch

?

If so, I don't understand why the distinction between 1) and 2) is
necessary, and why not to fetch directly into refs/remotes/origin/. IOW,
in which case is refs/remotes/origin/master different from
refs/mediawiki/origin/master for example?

Thanks,

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

[Jérémie NIKAES]

2011/5/30 Matthieu Moy <Matthieu.Moy@grenoble-inp.fr>:
> Jonathan Nieder <jrnieder@gmail.com> writes:
>
>> Jérémie NIKAES wrote:
>>
>>> No, I actually still have the problem. The version commited on github
>>> is pulling revisions straight to refs/heads/master which is gross
>>> After discussing this matter with our teacher Matthieu Moy, I wanted
>>> to change this to refs/remotes/origin/master but then this warning
>>> gets thrown.
>>
>> The following fixes it for me.
>
> It seems it does, thanks!
>
> clone does not give the warning anymore, and incremental pull work.
>

Yes ! It definitely does ! Works like a charm.

>>  sub mw_capabilities {
>>  #    print STDOUT "fetch\n";
>> +     print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
>
> Is this "mediawiki" comming from the URL (mediawiki::...), or is it just
> a convention you've set?
>
> We've tried with refspec refs/heads/*:refs/remotes/origin/*, but without
> success.
>
> Do I understand correctly that the "pull" is done in 3 stages:
>
> 1) import into refs/mediawiki/origin
>
> 2) fetch the imported ref into refs/remotes/origin
>
> 3) and merge as usual into the current branch
>
> ?
>
> If so, I don't understand why the distinction between 1) and 2) is
> necessary, and why not to fetch directly into refs/remotes/origin/. IOW,
> in which case is refs/remotes/origin/master different from
> refs/mediawiki/origin/master for example?
>
> Thanks,

We just committed this change to our master branch on github. We
actually have the same questions as Matthieu. We don't really
understand why we have to use another ref to make it work.

Huge thanks to you,

-- 
Jérémie Nikaes

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

[Jonathan Nieder]

Hi,

Matthieu Moy wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:

>>  sub mw_capabilities {
>>  #	print STDOUT "fetch\n";
>> +	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
>
> Is this "mediawiki" comming from the URL (mediawiki::...), or is it just
> a convention you've set?
>
> We've tried with refspec refs/heads/*:refs/remotes/origin/*, but without
> success.

Yes, I stole the convention for the namespace name from the commit message
to v1.7.0-rc0~62^2~19 (Allow helper to map private ref names into
normal names, 2009-11-18).

> Do I understand correctly that the "pull" is done in 3 stages:
>
> 1) import into refs/mediawiki/origin
>
> 2) fetch the imported ref into refs/remotes/origin
>
> 3) and merge as usual into the current branch
>
> ?
>
> If so, I don't understand why the distinction between 1) and 2) is
> necessary, and why not to fetch directly into refs/remotes/origin/. IOW,
> in which case is refs/remotes/origin/master different from
> refs/mediawiki/origin/master for example?

That's a good question.  One answer is that fetching to elsewhere
allows the transport machinery to take care of rejecting non-fast-forward
fetches and pretty-printing the ref update information:

 From mediawiki::http://localhost/mediawiki
    479ee21..accd2c1  master     -> origin/master

I think in the long term, importing to no ref at all as Sverre
suggested and teaching the transport-helper to do the ref update would
make more sense.

Regards,
Jonathan

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

[Jérémie NIKAES]

> Matthieu Moy wrote:
>> Jonathan Nieder <jrnieder@gmail.com> writes:

>> If so, I don't understand why the distinction between 1) and 2) is
>> necessary, and why not to fetch directly into refs/remotes/origin/. IOW,
>> in which case is refs/remotes/origin/master different from
>> refs/mediawiki/origin/master for example?
>
> That's a good question.  One answer is that fetching to elsewhere
> allows the transport machinery to take care of rejecting non-fast-forward
> fetches and pretty-printing the ref update information:
>
>  From mediawiki::http://localhost/mediawiki
>    479ee21..accd2c1  master     -> origin/master
>
> I think in the long term, importing to no ref at all as Sverre
> suggested and teaching the transport-helper to do the ref update would
> make more sense.

Well yes that would indeed make more sense. It feels weird to have to
do it like this.
Anyways, thanks a lot again for your help, a rfc patch should come
relatively quickly with the project in its current state.

Regards,
-- 
Jérémie Nikaes

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

[Sverre Rabbelier]

Heya,

On Mon, May 30, 2011 at 15:32, Jérémie NIKAES <jeremie.nikaes@gmail.com> wrote:
> Well yes that would indeed make more sense. It feels weird to have to
> do it like this.

The patch series I was referring to can be found at [0].

[0] http://thread.gmane.org/gmane.comp.version-control.git/157860/focus=157864

-- 
Cheers,

Sverre Rabbelier