[PATCH 0/8] git-remote-mediawiki: fixes, optimizations, and progress report

[PATCH 0/8] git-remote-mediawiki: fixes, optimizations, and progress report

[Matthieu Moy]

After the recent improvements to git-remote-mediawiki, I did a bit of
real-life testing, by importing https://ensiwiki.ensimag.fr/ and
https://git.wiki.kernel.org/. It turned out we had a few blocking bugs
and a grossly unoptimized algorithm on push. This patch serie makes
git-remote-mediawiki usable on these wikis (although it's still a bit
slow).

My initial patch serie also contained a patch "git-remote-mediawiki:
replace TODO-list in comment by appropriate link" that replaced the
TODO-list in the comments of the file with just 

 # Gateway between Git and MediaWiki.
 # Documentation & bugtracker: https://github.com/moy/Git-Mediawiki/

but I'm holding it for now as there seems to be interest in changing
the homepage, so the link may not be appropriate.

Matthieu Moy (8):
  git-remote-mediawiki: don't split namespaces with spaces
  git-remote-mediawiki: actually send empty comment when they're empty
  git-remote-mediawiki: make mediafiles export optional
  git-remote-mediawiki: get rid of O(N^2) loop
  git-remote-mediawiki: use --force when adding notes
  git-remote-mediawiki: show progress information when listing pages
  git-remote-mediawiki: show progress information when getting last
    remote revision
  git-remote-mediawiki: properly deal with invalid remote revisions

 contrib/mw-to-git/git-remote-mediawiki | 80 +++++++++++++++++++++++++++-------
 1 file changed, 64 insertions(+), 16 deletions(-)

-- 
1.7.11.1.30.g7e1baf9.dirty

[PATCH 1/8] git-remote-mediawiki: don't split namespaces with spaces

[Matthieu Moy]

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index accd70a..a6ad8cf 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1275,7 +1275,7 @@ sub get_mw_namespace_id {
 		# Look at configuration file, if the record for that namespace is
 		# already cached. Namespaces are stored in form:
 		# "Name_of_namespace:Id_namespace", ex.: "File:6".
-		my @temp = split(/[ \n]/, run_git("config --get-all remote."
+		my @temp = split(/[\n]/, run_git("config --get-all remote."
 						. $remotename .".namespaceCache"));
 		chomp(@temp);
 		foreach my $ns (@temp) {
-- 
1.7.11.1.30.g7e1baf9.dirty

Re: [PATCH 1/8] git-remote-mediawiki: don't split namespaces with spaces

[Junio C Hamano]

Looks sensible.

[PATCH 2/8] git-remote-mediawiki: actually send empty comment when they're empty

[Matthieu Moy]

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index a6ad8cf..a2da52f 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n";
 # used to reflect file creation or deletion in diff.
 use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
 
+# Used on Git's side to reflect empty edit messages on the wiki
+use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';
+
 my $remotename = $ARGV[0];
 my $url = $ARGV[1];
 
@@ -935,7 +938,7 @@ sub mw_import_revids {
 
 		my %commit;
 		$commit{author} = $rev->{user} || 'Anonymous';
-		$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
+		$commit{comment} = $rev->{comment} || EMPTY_MESSAGE;
 		$commit{title} = mediawiki_smudge_filename($page_title);
 		$commit{mw_revision} = $rev->{revid};
 		$commit{content} = mediawiki_smudge($rev->{'*'});
@@ -1050,6 +1053,10 @@ sub mw_push_file {
 	my $oldrevid = shift;
 	my $newrevid;
 
+	if ($summary eq EMPTY_MESSAGE) {
+		$summary = '';
+	}
+
 	my $new_sha1 = $diff_info_split[3];
 	my $old_sha1 = $diff_info_split[2];
 	my $page_created = ($old_sha1 eq NULL_SHA1);
-- 
1.7.11.1.30.g7e1baf9.dirty

Re: [PATCH 2/8] git-remote-mediawiki: actually send empty comment when they're empty

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@imag.fr> writes:

> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
> ---
>  contrib/mw-to-git/git-remote-mediawiki | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
> index a6ad8cf..a2da52f 100755
> --- a/contrib/mw-to-git/git-remote-mediawiki
> +++ b/contrib/mw-to-git/git-remote-mediawiki
> @@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n";
>  # used to reflect file creation or deletion in diff.
>  use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
>  
> +# Used on Git's side to reflect empty edit messages on the wiki
> +use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';
> +

Is there a reason why this sentinel value cannot be a real empty
string?

Not that I am complaining, as I find it highly unlikely for a
message stored in mw to be exactly "*Empty MediaWiki Message*", but
just wondering.

>  my $remotename = $ARGV[0];
>  my $url = $ARGV[1];
>  
> @@ -935,7 +938,7 @@ sub mw_import_revids {
>  
>  		my %commit;
>  		$commit{author} = $rev->{user} || 'Anonymous';
> -		$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
> +		$commit{comment} = $rev->{comment} || EMPTY_MESSAGE;
>  		$commit{title} = mediawiki_smudge_filename($page_title);
>  		$commit{mw_revision} = $rev->{revid};
>  		$commit{content} = mediawiki_smudge($rev->{'*'});
> @@ -1050,6 +1053,10 @@ sub mw_push_file {
>  	my $oldrevid = shift;
>  	my $newrevid;
>  
> +	if ($summary eq EMPTY_MESSAGE) {
> +		$summary = '';
> +	}
> +
>  	my $new_sha1 = $diff_info_split[3];
>  	my $old_sha1 = $diff_info_split[2];
>  	my $page_created = ($old_sha1 eq NULL_SHA1);

Re: [PATCH 2/8] git-remote-mediawiki: actually send empty comment when they're empty

[Matthieu Moy]

Junio C Hamano <gitster@pobox.com> writes:

> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>
>> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
>> ---
>>  contrib/mw-to-git/git-remote-mediawiki | 9 ++++++++-
>>  1 file changed, 8 insertions(+), 1 deletion(-)
>>
>> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
>> index a6ad8cf..a2da52f 100755
>> --- a/contrib/mw-to-git/git-remote-mediawiki
>> +++ b/contrib/mw-to-git/git-remote-mediawiki
>> @@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n";
>>  # used to reflect file creation or deletion in diff.
>>  use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
>>  
>> +# Used on Git's side to reflect empty edit messages on the wiki
>> +use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';
>> +
>
> Is there a reason why this sentinel value cannot be a real empty
> string?

That would mean having an empty commit message on the Git side, which is
against the use (although possible with fast-import). The import turns
empty mediawiki messages (which are very common in my experience) into
this string to provide a nicer-looking history.

It's not like EMPTY_CONTENT, which is explicitely forbidden on the
MediaWiki side, hence really cannot be the empty string.

Ideally, EMPTY_MESSAGE could be configurable, and the empty string could
be an acceptable value to make this conversion optional.

(note that the string was already there, my patch only makes it a
constant to be able to use it in two distinct places).

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH 2/8] git-remote-mediawiki: actually send empty comment when they're empty

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:

> Junio C Hamano <gitster@pobox.com> writes:
>
>> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>>
>>> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
>>> ---
>>>  contrib/mw-to-git/git-remote-mediawiki | 9 ++++++++-
>>>  1 file changed, 8 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
>>> index a6ad8cf..a2da52f 100755
>>> --- a/contrib/mw-to-git/git-remote-mediawiki
>>> +++ b/contrib/mw-to-git/git-remote-mediawiki
>>> @@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n";
>>>  # used to reflect file creation or deletion in diff.
>>>  use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
>>>  
>>> +# Used on Git's side to reflect empty edit messages on the wiki
>>> +use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';
>>> +
>>
>> Is there a reason why this sentinel value cannot be a real empty
>> string?
>
> That would mean having an empty commit message on the Git side, which is
> against the use (although possible with fast-import).

It also is possible with "commit --allow-empty-message", no?

> (note that the string was already there, my patch only makes it a
> constant to be able to use it in two distinct places).

It was clear from the patch text without looking beyond context, but
after all I thought this was meant to be a clean-up series, and I
simply wondered why mapping empty to some magic string was not
considered to be something to be cleaned up.  That's all.

I obviously do not care that deeply either way.

[PATCH 3/8] git-remote-mediawiki: make mediafiles export optional

[Matthieu Moy]

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index a2da52f..8e46e4e 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -66,11 +66,16 @@ chomp(@tracked_pages);
 my @tracked_categories = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".categories"));
 chomp(@tracked_categories);
 
-# Import media files too.
+# Import media files on pull
 my $import_media = run_git("config --get --bool remote.". $remotename .".mediaimport");
 chomp($import_media);
 $import_media = ($import_media eq "true");
 
+# Export media files on push
+my $export_media = run_git("config --get --bool remote.". $remotename .".mediaexport");
+chomp($export_media);
+$export_media = !($export_media eq "false");
+
 my $wiki_login = run_git("config --get remote.". $remotename .".mwLogin");
 # Note: mwPassword is discourraged. Use the credential system instead.
 my $wiki_passwd = run_git("config --get remote.". $remotename .".mwPassword");
@@ -1068,6 +1073,11 @@ sub mw_push_file {
 		$extension = "";
 	}
 	if ($extension eq "mw") {
+		my $ns = get_mw_namespace_id_for_page($complete_file_name);
+		if ($ns && $ns == get_mw_namespace_id("File") && (!$export_media)) {
+			print STDERR "Ignoring media file related page: $complete_file_name\n";
+			return ($oldrevid, "ok");
+		}
 		my $file_content;
 		if ($page_deleted) {
 			# Deleting a page usually requires
@@ -1107,10 +1117,12 @@ sub mw_push_file {
 		}
 		$newrevid = $result->{edit}->{newrevid};
 		print STDERR "Pushed file: $new_sha1 - $title\n";
-	} else {
+	} elsif ($export_media) {
 		$newrevid = mw_upload_file($complete_file_name, $new_sha1,
 					   $extension, $page_deleted,
 					   $summary);
+	} else {
+		print STDERR "Ignoring media file $title\n";
 	}
 	$newrevid = ($newrevid or $oldrevid);
 	return ($newrevid, "ok");
@@ -1328,3 +1340,11 @@ sub get_mw_namespace_id {
 		die "No such namespace $name on MediaWiki.";
 	}
 }
+
+sub get_mw_namespace_id_for_page {
+	if (my ($namespace) = $_[0] =~ /^([^:]*):/) {
+		return get_mw_namespace_id($namespace);
+	} else {
+		return;
+	}
+}
-- 
1.7.11.1.30.g7e1baf9.dirty

Re: [PATCH 3/8] git-remote-mediawiki: make mediafiles export optional

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@imag.fr> writes:

> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>

It would have been nicer to hint why people may want to omit
mediafiles from their export under what condition somewhere in the
documentation or at least in the proposed commit log message.

Re: [PATCH 3/8] git-remote-mediawiki: make mediafiles export optional

[Matthieu Moy]

Junio C Hamano <gitster@pobox.com> writes:

> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>
>> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
>
> It would have been nicer to hint why people may want to omit
> mediafiles from their export under what condition somewhere in the
> documentation or at least in the proposed commit log message.

The real use-case for which I implemented is a bit particular (described
below), but I'll resend with a more general explanation.

For the curious: I have a wiki with ~2Gb of mediafiles, and I want to
secure it without particular access to the server. I import the wiki on
a desktop machine with a large disk, and re-export it to another
webserver with a small disk quota. Importing mediafiles is good because
it gives me a backup, but exporting them is not an option as my quota on
the webserver is too small.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

[PATCH 4/8] git-remote-mediawiki: get rid of O(N^2) loop

[Matthieu Moy]

The algorithm to find a path from the local revision to the remote one
was calling "git rev-list" and parsing its output N times. Run rev-list
only once, and fill a hashtable with the result to optimize the body of
the loop.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 8e46e4e..f9c0cc6 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1196,16 +1196,27 @@ sub mw_push_revision {
 	if ($last_local_revid > 0) {
 		my $parsed_sha1 = $remoteorigin_sha1;
 		# Find a path from last MediaWiki commit to pushed commit
+		print STDERR "Computing path from local to remote ...\n";
+		my @local_ancestry = split(/\n/, run_git("rev-list --boundary --children $local ^$parsed_sha1"));
+		my %local_ancestry;
+		foreach my $line (@local_ancestry) {
+			if (my ($parent, $child) = $line =~ m/^-?([a-f0-9]+) ([a-f0-9]+)/) {
+				$local_ancestry{$parent} = $child;
+				if ($parent eq $parsed_sha1 || $child eq $parsed_sha1) {
+					print STDERR "$parent -> $child\n";
+				}
+			} elsif (!$line =~ m/^([a-f0-9]+)/) {
+				die "Unexpected output from git rev-list: $line";
+			}
+		}
 		while ($parsed_sha1 ne $HEAD_sha1) {
-			my @commit_info =  grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $local")));
-			if (!@commit_info) {
+			my $child = $local_ancestry{$parsed_sha1};
+			if (!$child) {
+				printf STDERR "Cannot find a path in history from remote commit to last commit\n";
 				return error_non_fast_forward($remote);
 			}
-			my @commit_info_split = split(/ |\n/, $commit_info[0]);
-			# $commit_info_split[1] is the sha1 of the commit to export
-			# $commit_info_split[0] is the sha1 of its direct child
-			push(@commit_pairs, \@commit_info_split);
-			$parsed_sha1 = $commit_info_split[1];
+			push(@commit_pairs, [$parsed_sha1, $child]);
+			$parsed_sha1 = $child;
 		}
 	} else {
 		# No remote mediawiki revision. Export the whole
-- 
1.7.11.1.30.g7e1baf9.dirty

Re: [PATCH 4/8] git-remote-mediawiki: get rid of O(N^2) loop

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@imag.fr> writes:

> The algorithm to find a path from the local revision to the remote one
> was calling "git rev-list" and parsing its output N times. Run rev-list
> only once, and fill a hashtable with the result to optimize the body of
> the loop.

Good thinking.  I wonder if it would further reduce the overhead if
you stop using --children and do this using --parents instead, as
you will be reading the parsed_sha1..local range either way yourself
anyway.

> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
> ---
>  contrib/mw-to-git/git-remote-mediawiki | 25 ++++++++++++++++++-------
>  1 file changed, 18 insertions(+), 7 deletions(-)
>
> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
> index 8e46e4e..f9c0cc6 100755
> --- a/contrib/mw-to-git/git-remote-mediawiki
> +++ b/contrib/mw-to-git/git-remote-mediawiki
> @@ -1196,16 +1196,27 @@ sub mw_push_revision {
>  	if ($last_local_revid > 0) {
>  		my $parsed_sha1 = $remoteorigin_sha1;
>  		# Find a path from last MediaWiki commit to pushed commit
> +		print STDERR "Computing path from local to remote ...\n";
> +		my @local_ancestry = split(/\n/, run_git("rev-list --boundary --children $local ^$parsed_sha1"));
> +		my %local_ancestry;
> +		foreach my $line (@local_ancestry) {
> +			if (my ($parent, $child) = $line =~ m/^-?([a-f0-9]+) ([a-f0-9]+)/) {
> +				$local_ancestry{$parent} = $child;
> +				if ($parent eq $parsed_sha1 || $child eq $parsed_sha1) {
> +					print STDERR "$parent -> $child\n";
> +				}
> +			} elsif (!$line =~ m/^([a-f0-9]+)/) {
> +				die "Unexpected output from git rev-list: $line";
> +			}
> +		}
>  		while ($parsed_sha1 ne $HEAD_sha1) {
> -			my @commit_info =  grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $local")));
> -			if (!@commit_info) {
> +			my $child = $local_ancestry{$parsed_sha1};
> +			if (!$child) {
> +				printf STDERR "Cannot find a path in history from remote commit to last commit\n";
>  				return error_non_fast_forward($remote);
>  			}
> -			my @commit_info_split = split(/ |\n/, $commit_info[0]);
> -			# $commit_info_split[1] is the sha1 of the commit to export
> -			# $commit_info_split[0] is the sha1 of its direct child
> -			push(@commit_pairs, \@commit_info_split);
> -			$parsed_sha1 = $commit_info_split[1];
> +			push(@commit_pairs, [$parsed_sha1, $child]);
> +			$parsed_sha1 = $child;
>  		}
>  	} else {
>  		# No remote mediawiki revision. Export the whole

Re: [PATCH 4/8] git-remote-mediawiki: get rid of O(N^2) loop

[Matthieu Moy]

Junio C Hamano <gitster@pobox.com> writes:

> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>
>> The algorithm to find a path from the local revision to the remote one
>> was calling "git rev-list" and parsing its output N times. Run rev-list
>> only once, and fill a hashtable with the result to optimize the body of
>> the loop.
>
> Good thinking.  I wonder if it would further reduce the overhead if
> you stop using --children and do this using --parents instead, as
> you will be reading the parsed_sha1..local range either way yourself
> anyway.

It is possible, yes. I'll resend a version with --parents, but this
probably doesn't change the performance much: what we really need is for
Git to prune dead-ends in the subgraph, to make sure we find a path
without having to backtrack (i.e. we need parent rewriting history
simplification), so Git has to do something a bit clever anyway.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

[PATCH 5/8] git-remote-mediawiki: use --force when adding notes

[Matthieu Moy]

When notes are created to record a push, it normally doesn't exist yet.
However, when a push is interrupted and then restarted, it may happen
that a commit already has notes attached, and we want to reflect the newly
created remote revision, hence use 'git notes add -f' to override the
existing one

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index f9c0cc6..f56f9b0 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1264,7 +1264,7 @@ sub mw_push_revision {
 			}
 		}
 		unless ($dumb_push) {
-			run_git("notes --ref=$remotename/mediawiki add -m \"mediawiki_revision: $mw_revision\" $sha1_commit");
+			run_git("notes --ref=$remotename/mediawiki add -f -m \"mediawiki_revision: $mw_revision\" $sha1_commit");
 			run_git("update-ref -m \"Git-MediaWiki push\" refs/mediawiki/$remotename/master $sha1_commit $sha1_child");
 		}
 	}
-- 
1.7.11.1.30.g7e1baf9.dirty

[PATCH 6/8] git-remote-mediawiki: show progress information when listing pages

[Matthieu Moy]

Initial phases of push and pull with git-remote-mediawiki can be long on
a large wiki. Let the user know what's going on.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index f56f9b0..755c9d0 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -371,6 +371,8 @@ sub get_mw_first_pages {
 sub get_mw_pages {
 	mw_connect_maybe();
 
+	print STDERR "Listing pages on remote wiki...\n";
+
 	my %pages; # hash on page titles to avoid duplicates
 	my $user_defined;
 	if (@tracked_pages) {
@@ -394,6 +396,7 @@ sub get_mw_pages {
 			get_all_mediafiles(\%pages);
 		}
 	}
+	print STDERR (scalar keys %pages) . " pages found.\n";
 	return %pages;
 }
 
-- 
1.7.11.1.30.g7e1baf9.dirty

[PATCH 7/8] git-remote-mediawiki: show progress information when getting last remote revision

[Matthieu Moy]

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 755c9d0..cf467ac 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -579,6 +579,8 @@ sub get_last_remote_revision {
 
 	my $max_rev_num = 0;
 
+	print STDERR "Getting last revision id on tracked pages...\n";
+
 	foreach my $page (@pages) {
 		my $id = $page->{pageid};
 
-- 
1.7.11.1.30.g7e1baf9.dirty

[PATCH 8/8] git-remote-mediawiki: properly deal with invalid remote revisions

[Matthieu Moy]

Some wiki, including https://git.wiki.kernel.org/ have invalid revision
numbers (i.e. the actual revision numbers are non-contiguous). Don't die
when encountering one.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index cf467ac..80a07cc 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -910,6 +910,10 @@ sub mw_import_revids {
 	my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined
 
 	foreach my $pagerevid (@$revision_ids) {
+	        # Count page even if we skip it, since we display
+		# $n/$total and $total includes skipped pages.
+		$n++;
+
 		# fetch the content of the pages
 		my $query = {
 			action => 'query',
@@ -924,6 +928,11 @@ sub mw_import_revids {
 			die "Failed to retrieve modified page for revision $pagerevid";
 		}
 
+		if (defined($result->{query}->{badrevids}->{$pagerevid})) {
+			# The revision id does not exist on the remote wiki.
+			next;
+		}
+
 		if (!defined($result->{query}->{pages})) {
 			die "Invalid revision $pagerevid.";
 		}
@@ -932,10 +941,6 @@ sub mw_import_revids {
 		my $result_page = $result_pages[0];
 		my $rev = $result_pages[0]->{revisions}->[0];
 
-	        # Count page even if we skip it, since we display
-		# $n/$total and $total includes skipped pages.
-		$n++;
-
 		my $page_title = $result_page->{title};
 
 		if (!exists($pages->{$page_title})) {
-- 
1.7.11.1.30.g7e1baf9.dirty

Re: [PATCH 0/8] git-remote-mediawiki: fixes, optimizations, and progress report

[Junio C Hamano]

Thanks for updates. Will queue on 'pu' as-is.

[PATCH 0/8 v2] git-remote-mediawiki: fixes, optimizations, and progress report

[Matthieu Moy]

Changes since v1: more explanations in commit message for "make
mediafiles export optional", and use --parents instead of --children
in "get rid of O(N^2) loop".

Matthieu Moy (8):
  git-remote-mediawiki: don't split namespaces with spaces
  git-remote-mediawiki: actually send empty comment when they're empty
  git-remote-mediawiki: make mediafiles export optional
  git-remote-mediawiki: get rid of O(N^2) loop
  git-remote-mediawiki: use --force when adding notes
  git-remote-mediawiki: show progress information when listing pages
  git-remote-mediawiki: show progress information when getting last
    remote revision
  git-remote-mediawiki: properly deal with invalid remote revisions

 contrib/mw-to-git/git-remote-mediawiki | 79 +++++++++++++++++++++++++++-------
 1 file changed, 63 insertions(+), 16 deletions(-)

-- 
1.7.11.2.258.g5ff3cdf.dirty

[PATCH 1/8] git-remote-mediawiki: don't split namespaces with spaces

[Matthieu Moy]

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index accd70a..a6ad8cf 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1275,7 +1275,7 @@ sub get_mw_namespace_id {
 		# Look at configuration file, if the record for that namespace is
 		# already cached. Namespaces are stored in form:
 		# "Name_of_namespace:Id_namespace", ex.: "File:6".
-		my @temp = split(/[ \n]/, run_git("config --get-all remote."
+		my @temp = split(/[\n]/, run_git("config --get-all remote."
 						. $remotename .".namespaceCache"));
 		chomp(@temp);
 		foreach my $ns (@temp) {
-- 
1.7.11.2.258.g5ff3cdf.dirty

[PATCH 2/8] git-remote-mediawiki: actually send empty comment when they're empty

[Matthieu Moy]

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index a6ad8cf..a2da52f 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n";
 # used to reflect file creation or deletion in diff.
 use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
 
+# Used on Git's side to reflect empty edit messages on the wiki
+use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';
+
 my $remotename = $ARGV[0];
 my $url = $ARGV[1];
 
@@ -935,7 +938,7 @@ sub mw_import_revids {
 
 		my %commit;
 		$commit{author} = $rev->{user} || 'Anonymous';
-		$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
+		$commit{comment} = $rev->{comment} || EMPTY_MESSAGE;
 		$commit{title} = mediawiki_smudge_filename($page_title);
 		$commit{mw_revision} = $rev->{revid};
 		$commit{content} = mediawiki_smudge($rev->{'*'});
@@ -1050,6 +1053,10 @@ sub mw_push_file {
 	my $oldrevid = shift;
 	my $newrevid;
 
+	if ($summary eq EMPTY_MESSAGE) {
+		$summary = '';
+	}
+
 	my $new_sha1 = $diff_info_split[3];
 	my $old_sha1 = $diff_info_split[2];
 	my $page_created = ($old_sha1 eq NULL_SHA1);
-- 
1.7.11.2.258.g5ff3cdf.dirty

[PATCH 3/8] git-remote-mediawiki: make mediafiles export optional

[Matthieu Moy]

It is possible to use git-remote-mediawiki on a tree with both .mw files
and other files. Before git-remote-mediawiki learnt how to export
mediafiles, such mixed trees allowed the user to maintain both the wiki
and other files for the same project in the same repository. With the
newly added support for exporting mediafiles, pushing such mixed trees
would upload unrelated files as mediafiles, which may not be desired.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index a2da52f..8e46e4e 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -66,11 +66,16 @@ chomp(@tracked_pages);
 my @tracked_categories = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".categories"));
 chomp(@tracked_categories);
 
-# Import media files too.
+# Import media files on pull
 my $import_media = run_git("config --get --bool remote.". $remotename .".mediaimport");
 chomp($import_media);
 $import_media = ($import_media eq "true");
 
+# Export media files on push
+my $export_media = run_git("config --get --bool remote.". $remotename .".mediaexport");
+chomp($export_media);
+$export_media = !($export_media eq "false");
+
 my $wiki_login = run_git("config --get remote.". $remotename .".mwLogin");
 # Note: mwPassword is discourraged. Use the credential system instead.
 my $wiki_passwd = run_git("config --get remote.". $remotename .".mwPassword");
@@ -1068,6 +1073,11 @@ sub mw_push_file {
 		$extension = "";
 	}
 	if ($extension eq "mw") {
+		my $ns = get_mw_namespace_id_for_page($complete_file_name);
+		if ($ns && $ns == get_mw_namespace_id("File") && (!$export_media)) {
+			print STDERR "Ignoring media file related page: $complete_file_name\n";
+			return ($oldrevid, "ok");
+		}
 		my $file_content;
 		if ($page_deleted) {
 			# Deleting a page usually requires
@@ -1107,10 +1117,12 @@ sub mw_push_file {
 		}
 		$newrevid = $result->{edit}->{newrevid};
 		print STDERR "Pushed file: $new_sha1 - $title\n";
-	} else {
+	} elsif ($export_media) {
 		$newrevid = mw_upload_file($complete_file_name, $new_sha1,
 					   $extension, $page_deleted,
 					   $summary);
+	} else {
+		print STDERR "Ignoring media file $title\n";
 	}
 	$newrevid = ($newrevid or $oldrevid);
 	return ($newrevid, "ok");
@@ -1328,3 +1340,11 @@ sub get_mw_namespace_id {
 		die "No such namespace $name on MediaWiki.";
 	}
 }
+
+sub get_mw_namespace_id_for_page {
+	if (my ($namespace) = $_[0] =~ /^([^:]*):/) {
+		return get_mw_namespace_id($namespace);
+	} else {
+		return;
+	}
+}
-- 
1.7.11.2.258.g5ff3cdf.dirty

[PATCH 4/8] git-remote-mediawiki: get rid of O(N^2) loop

[Matthieu Moy]

The algorithm to find a path from the local revision to the remote one
was calling "git rev-list" and parsing its output N times. Run rev-list
only once, and fill a hashtable with the result to optimize the body of
the loop.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 8e46e4e..fb1e9e0 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1196,16 +1196,26 @@ sub mw_push_revision {
 	if ($last_local_revid > 0) {
 		my $parsed_sha1 = $remoteorigin_sha1;
 		# Find a path from last MediaWiki commit to pushed commit
+		print STDERR "Computing path from local to remote ...\n";
+		my @local_ancestry = split(/\n/, run_git("rev-list --boundary --parents $local ^$parsed_sha1"));
+		my %local_ancestry;
+		foreach my $line (@local_ancestry) {
+			if (my ($child, $parents) = $line =~ m/^-?([a-f0-9]+) ([a-f0-9 ]+)/) {
+				foreach my $parent (split(' ', $parents)) {
+					$local_ancestry{$parent} = $child;
+				}
+			} elsif (!$line =~ m/^([a-f0-9]+)/) {
+				die "Unexpected output from git rev-list: $line";
+			}
+		}
 		while ($parsed_sha1 ne $HEAD_sha1) {
-			my @commit_info =  grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $local")));
-			if (!@commit_info) {
+			my $child = $local_ancestry{$parsed_sha1};
+			if (!$child) {
+				printf STDERR "Cannot find a path in history from remote commit to last commit\n";
 				return error_non_fast_forward($remote);
 			}
-			my @commit_info_split = split(/ |\n/, $commit_info[0]);
-			# $commit_info_split[1] is the sha1 of the commit to export
-			# $commit_info_split[0] is the sha1 of its direct child
-			push(@commit_pairs, \@commit_info_split);
-			$parsed_sha1 = $commit_info_split[1];
+			push(@commit_pairs, [$parsed_sha1, $child]);
+			$parsed_sha1 = $child;
 		}
 	} else {
 		# No remote mediawiki revision. Export the whole
-- 
1.7.11.2.258.g5ff3cdf.dirty

[PATCH 5/8] git-remote-mediawiki: use --force when adding notes

[Matthieu Moy]

When notes are created to record a push, it normally doesn't exist yet.
However, when a push is interrupted and then restarted, it may happen
that a commit already has notes attached, and we want to reflect the newly
created remote revision, hence use 'git notes add -f' to override the
existing one

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index fb1e9e0..517a4db 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -1263,7 +1263,7 @@ sub mw_push_revision {
 			}
 		}
 		unless ($dumb_push) {
-			run_git("notes --ref=$remotename/mediawiki add -m \"mediawiki_revision: $mw_revision\" $sha1_commit");
+			run_git("notes --ref=$remotename/mediawiki add -f -m \"mediawiki_revision: $mw_revision\" $sha1_commit");
 			run_git("update-ref -m \"Git-MediaWiki push\" refs/mediawiki/$remotename/master $sha1_commit $sha1_child");
 		}
 	}
-- 
1.7.11.2.258.g5ff3cdf.dirty

[PATCH 6/8] git-remote-mediawiki: show progress information when listing pages

[Matthieu Moy]

Initial phases of push and pull with git-remote-mediawiki can be long on
a large wiki. Let the user know what's going on.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 517a4db..729a0bc 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -371,6 +371,8 @@ sub get_mw_first_pages {
 sub get_mw_pages {
 	mw_connect_maybe();
 
+	print STDERR "Listing pages on remote wiki...\n";
+
 	my %pages; # hash on page titles to avoid duplicates
 	my $user_defined;
 	if (@tracked_pages) {
@@ -394,6 +396,7 @@ sub get_mw_pages {
 			get_all_mediafiles(\%pages);
 		}
 	}
+	print STDERR (scalar keys %pages) . " pages found.\n";
 	return %pages;
 }
 
-- 
1.7.11.2.258.g5ff3cdf.dirty

[PATCH 7/8] git-remote-mediawiki: show progress information when getting last remote revision

[Matthieu Moy]

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 729a0bc..8badff6 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -579,6 +579,8 @@ sub get_last_remote_revision {
 
 	my $max_rev_num = 0;
 
+	print STDERR "Getting last revision id on tracked pages...\n";
+
 	foreach my $page (@pages) {
 		my $id = $page->{pageid};
 
-- 
1.7.11.2.258.g5ff3cdf.dirty

[PATCH 8/8] git-remote-mediawiki: properly deal with invalid remote revisions

[Matthieu Moy]

Some wiki, including https://git.wiki.kernel.org/ have invalid revision
numbers (i.e. the actual revision numbers are non-contiguous). Don't die
when encountering one.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index 8badff6..5eab96b 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -910,6 +910,10 @@ sub mw_import_revids {
 	my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined
 
 	foreach my $pagerevid (@$revision_ids) {
+	        # Count page even if we skip it, since we display
+		# $n/$total and $total includes skipped pages.
+		$n++;
+
 		# fetch the content of the pages
 		my $query = {
 			action => 'query',
@@ -924,6 +928,11 @@ sub mw_import_revids {
 			die "Failed to retrieve modified page for revision $pagerevid";
 		}
 
+		if (defined($result->{query}->{badrevids}->{$pagerevid})) {
+			# The revision id does not exist on the remote wiki.
+			next;
+		}
+
 		if (!defined($result->{query}->{pages})) {
 			die "Invalid revision $pagerevid.";
 		}
@@ -932,10 +941,6 @@ sub mw_import_revids {
 		my $result_page = $result_pages[0];
 		my $rev = $result_pages[0]->{revisions}->[0];
 
-	        # Count page even if we skip it, since we display
-		# $n/$total and $total includes skipped pages.
-		$n++;
-
 		my $page_title = $result_page->{title};
 
 		if (!exists($pages->{$page_title})) {
-- 
1.7.11.2.258.g5ff3cdf.dirty

Re: [PATCH 0/8 v2] git-remote-mediawiki: fixes, optimizations, and progress report

[Junio C Hamano]

Thanks; will replace and requeue.

[PATCH 0/2] git-remote-mediawiki: two more fixes

[Matthieu Moy]

These patches can be added to branch mm/mediawiki-usability in pu.

For the curious: I originally had patch 2/2 in my serie, but
mistakenly discarded it during a rebase. While recovering it and
testing it, I found a typo fixed by patch 1/2.

Matthieu Moy (2):
  git-remote-mediawiki: fix incorrect test usage in test
  git-remote-mediawiki: allow page names with a ':'

 contrib/mw-to-git/git-remote-mediawiki      | 49 +++++++++++++++++++----------
 contrib/mw-to-git/t/push-pull-tests.sh      |  2 +-
 contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh | 20 ++++++++++++
 3 files changed, 54 insertions(+), 17 deletions(-)

-- 
1.7.11.2.258.g5ff3cdf.dirty

[PATCH 1/2] git-remote-mediawiki: fix incorrect test usage in test

[Matthieu Moy]

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/t/push-pull-tests.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/mw-to-git/t/push-pull-tests.sh b/contrib/mw-to-git/t/push-pull-tests.sh
index 6692a0f..9da2dc5 100644
--- a/contrib/mw-to-git/t/push-pull-tests.sh
+++ b/contrib/mw-to-git/t/push-pull-tests.sh
@@ -104,7 +104,7 @@ test_push_pull () {
 			git push
 		) &&
 
-		test ! wiki_page_exist Foo
+		test_must_fail wiki_page_exist Foo
 	'
 
 	test_expect_success 'Merge conflict expected and solving it' '
-- 
1.7.11.2.258.g5ff3cdf.dirty

[PATCH 2/2] git-remote-mediawiki: allow page names with a ':'

[Matthieu Moy]

Traditionnally, pages named Foo:Bar are page 'Bar' in namespace 'Foo'.
However, it is also possible to call a page Foo:Bar if 'Foo' is not a
namespace. In this case, the actual name of the page is 'Foo:Bar', in the
main namespace. Since we can't tell with only the filename, query the
wiki for a namespace 'Foo' in these cases, but deal with the case where
no such namespace is found.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki      | 49 +++++++++++++++++++----------
 contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh | 20 ++++++++++++
 2 files changed, 53 insertions(+), 16 deletions(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index d6be188..47d4878 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -955,8 +955,11 @@ sub mw_import_revids {
 		# Differentiates classic pages and media files.
 		my ($namespace, $filename) = $page_title =~ /^([^:]*):(.*)$/;
 		my %mediafile;
-		if ($namespace && get_mw_namespace_id($namespace) == get_mw_namespace_id("File")) {
-			%mediafile = get_mw_mediafile_for_page_revision($filename, $rev->{timestamp});
+		if ($namespace) {
+			my $id = get_mw_namespace_id($namespace);
+			if ($id && $id == get_mw_namespace_id("File")) {
+				%mediafile = get_mw_mediafile_for_page_revision($filename, $rev->{timestamp});
+			}
 		}
 		# If this is a revision of the media page for new version
 		# of a file do one common commit for both file and media page.
@@ -1306,7 +1309,11 @@ sub get_mw_namespace_id {
 		chomp(@temp);
 		foreach my $ns (@temp) {
 			my ($n, $id) = split(/:/, $ns);
-			$namespace_id{$n} = $id;
+			if ($id eq 'notANameSpace') {
+				$namespace_id{$n} = {is_namespace => 0};
+			} else {
+				$namespace_id{$n} = {is_namespace => 1, id => $id};
+			}
 			$cached_mw_namespace_id{$n} = 1;
 		}
 	}
@@ -1324,28 +1331,38 @@ sub get_mw_namespace_id {
 
 	        while (my ($id, $ns) = each(%{$result->{query}->{namespaces}})) {
 	                if (defined($ns->{id}) && defined($ns->{canonical})) {
-				$namespace_id{$ns->{canonical}} = $ns->{id};
+				$namespace_id{$ns->{canonical}} = {is_namespace => 1, id => $ns->{id}};
 				if ($ns->{'*'}) {
 					# alias (e.g. french Fichier: as alias for canonical File:)
-					$namespace_id{$ns->{'*'}} = $ns->{id};
+					$namespace_id{$ns->{'*'}} = {is_namespace => 1, id => $ns->{id}};
 				}
 			}
 	        }
 	}
 
-	my $id = $namespace_id{$name};
+	my $ns = $namespace_id{$name};
+	my $id;
 
-	if (defined $id) {
-		# Store explicitely requested namespaces on disk
-		if (!exists $cached_mw_namespace_id{$name}) {
-			run_git("config --add remote.". $remotename
-				.".namespaceCache \"". $name .":". $id ."\"");
-			$cached_mw_namespace_id{$name} = 1;
-		}
-		return $id;
-	} else {
-		die "No such namespace $name on MediaWiki.";
+	unless (defined $ns) {
+		print STDERR "No such namespace $name on MediaWiki.\n";
+		$ns = {is_namespace => 0};
+		$namespace_id{$name} = $ns;
+	}
+
+	if ($ns->{is_namespace}) {
+		$id = $ns->{id};
+	}
+
+	# Store "notANameSpace" as special value for inexisting namespaces
+	my $store_id = ($id || 'notANameSpace');
+
+	# Store explicitely requested namespaces on disk
+	if (!exists $cached_mw_namespace_id{$name}) {
+		run_git("config --add remote.". $remotename
+			.".namespaceCache \"". $name .":". $store_id ."\"");
+		$cached_mw_namespace_id{$name} = 1;
 	}
+	return $id;
 }
 
 sub get_mw_namespace_id_for_page {
diff --git a/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh b/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh
index 8635878..246d47d 100755
--- a/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh
+++ b/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh
@@ -169,6 +169,26 @@ test_expect_failure 'special character at the begining of file name from mw to g
 	test_path_is_file mw_dir_11/[char_2
 '
 
+test_expect_success 'Pull page with title containing ":" other than namespace separator' '
+	wiki_editpage Foo:Bar content false &&
+	(
+		cd mw_dir_11 &&
+		git pull
+	) &&
+	test_path_is_file mw_dir_11/Foo:Bar.mw
+'
+
+test_expect_success 'Push page with title containing ":" other than namespace separator' '
+	(
+		cd mw_dir_11 &&
+		echo content >NotANameSpace:Page.mw &&
+		git add NotANameSpace:Page.mw &&
+		git commit -m "add page with colon" &&
+		git push
+	) &&
+	wiki_page_exist NotANameSpace:Page
+'
+
 test_expect_success 'test of correct formating for file name from mw to git' '
 	wiki_reset &&
 	git clone mediawiki::'"$WIKI_URL"' mw_dir_12 &&
-- 
1.7.11.2.258.g5ff3cdf.dirty

Re: [PATCH 2/2] git-remote-mediawiki: allow page names with a ':'

[Dan Johnson]

On Tue, Jul 17, 2012 at 10:06 AM, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
> Traditionnally, pages named Foo:Bar are page 'Bar' in namespace 'Foo'.
> However, it is also possible to call a page Foo:Bar if 'Foo' is not a
> namespace. In this case, the actual name of the page is 'Foo:Bar', in the
> main namespace. Since we can't tell with only the filename, query the
> wiki for a namespace 'Foo' in these cases, but deal with the case where
> no such namespace is found.

Might not be worth fixing, and it's just a typo in the commit message, but:
s/Traditionnally/Traditionally/
?

-- 
-Dan

Re: [PATCH 2/2] git-remote-mediawiki: allow page names with a ':'

[Matthieu Moy]

Dan Johnson <computerdruid@gmail.com> writes:

> On Tue, Jul 17, 2012 at 10:06 AM, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
>> Traditionnally, pages named Foo:Bar are page 'Bar' in namespace 'Foo'.
>> However, it is also possible to call a page Foo:Bar if 'Foo' is not a
>> namespace. In this case, the actual name of the page is 'Foo:Bar', in the
>> main namespace. Since we can't tell with only the filename, query the
>> wiki for a namespace 'Foo' in these cases, but deal with the case where
>> no such namespace is found.
>
> Might not be worth fixing, and it's just a typo in the commit message, but:
> s/Traditionnally/Traditionally/
> ?

Thanks, but the topic is in next already, so it's too late to fix the
commit message.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/