[PATCH 1/2] fast-import: initialize variable require_explicit_termination

[PATCH 1/2] fast-import: initialize variable require_explicit_termination

[Matthieu Moy]

The uninitialized variable seems harmless in practice, but let's still be clean.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
For some reason, remote helpers seem to be forced to use the "done"
command now. Investing why, I found this, but that wasn't what I was
looking for.

 fast-import.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/fast-import.c b/fast-import.c
index 7cc2262..ed8f3cd 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -355,7 +355,7 @@ static unsigned int cmd_save = 100;
 static uintmax_t next_mark;
 static struct strbuf new_data = STRBUF_INIT;
 static int seen_data_command;
-static int require_explicit_termination;
+static int require_explicit_termination = 0;
 
 /* Signal handling */
 static volatile sig_atomic_t checkpoint_requested;
-- 
1.7.6.585.g5929f.dirty

[PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)

[Matthieu Moy]

From: Jeremie Nikaes <jeremie.nikaes@ensimag.imag.fr>

Implement a gate between git and mediawiki, allowing git users to push
and pull objects from mediawiki just as one would do with a classic git
repository thanks to remote-helpers.

The following packages need to be installed (available on common
repositories):

     libmediawiki-api-perl
     libdatetime-format-iso8601-perl

Use remote helpers in order to be as transparent as possible to the git
user.

Download Mediawiki revisions through the Mediawiki API and then
fast-import into git.

Mediawiki revision number and git commits are linked thanks to notes
bound to commits.

The import part is done on a refs/mediawiki/<remote> branch before
coming to refs/remote/origin/master (Huge thanks to Jonathan Nieder
for his help)

We use UTF-8 everywhere: use encoding 'utf8'; does most of the job, but
we also read the output of Git commands in UTF-8 with the small helper
run_git, and write to the console (STDERR) in UTF-8. This allows a
seamless use of non-ascii characters in page titles, but hasn't been
tested on non-UTF-8 systems. In particular, UTF-8 encoding for filenames
could raise problems if different file systems handle UTF-8 filenames
differently. A uri_escape of mediawiki filenames could be imaginable, and
is still to be discussed further.

Partial cloning is supported using one of:

git clone -c remote.origin.pages='A_Page  Another_Page' mediawiki::http://wikiurl

git clone -c remote.origin.categories='Some_Category' mediawiki::http://wikiurl

git clone -c remote.origin.shallow='True' mediawiki::http://wikiurl

Thanks to notes metadata, it is possible to compare remote and local last
mediawiki revision to warn non-fast forward pushes and "everything
up-to-date" case.

When allowed, push looks for each commit between remotes/origin/master
and HEAD, catches every blob related to these commit and push them in
chronological order. To do so, it uses git rev-list --children HEAD and
travels the tree from remotes/origin/master to HEAD through children. In
other words :

	* Shortest path from remotes/origin/master to HEAD
	* For each commit encountered, push blobs related to this commit

Signed-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
Signed-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
Signed-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>
Signed-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>
Signed-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
Signed-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
This is a resurection of an old patch serie.

The previous round was here:

  http://thread.gmane.org/gmane.comp.version-control.git/175523

Maintaining two separate patches for import/export was getting
painful, so I've merged them into a single patch.

Git-Mediawiki is originally a student project, and since the students
didn't have time to finish the work this summer, I've taken time to
do some cleanup and testing.

In short, the changes since v3 are:

* Adapt to newer Git, which seem to require a "done" command at the
  end of the fast-import stream. I don't understand why this is
  needed, since fast-import is called without the --done flag by
  remote-helpers, but if I don't do this, "git fetch" doesn't
  terminate and keeps waiting ...

* Allow importing just a category

* Support shallow clone and fetch (for fetch, this means fetch just
  the last revision each time).

* Basic support for authentication (with password cleartext in
  .git/config :-( ).

* Support pushing to a new wiki (i.e. not the one you've cloned to).

* Accented characters in filenames now displayed correctly on the
  terminal.

* Support page deletion (by replacing the page content with
  [[Category:Deleted]]).

* Support creation of empty files

* Support forbidden characters {}[]| in filenames

* Support partial clone with more than 50 pages

I'd like to get this merged in contrib/, so that the code be in a
safe place, where other people can take care of it too. I'll probably
offer a student project "improve Git-Mediawiki" to my students next
June.

 contrib/mw-to-git/git-remote-mediawiki     |  722 ++++++++++++++++++++++++++++
 contrib/mw-to-git/git-remote-mediawiki.txt |    7 +
 2 files changed, 729 insertions(+), 0 deletions(-)
 create mode 100755 contrib/mw-to-git/git-remote-mediawiki
 create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
new file mode 100755
index 0000000..62c6794
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -0,0 +1,722 @@
+#! /usr/bin/perl
+
+# Copyright (C) 2011
+#     Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
+#     Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
+#     Claire Fousse <claire.fousse@ensimag.imag.fr>
+#     David Amouyal <david.amouyal@ensimag.imag.fr>
+#     Matthieu Moy <matthieu.moy@grenoble-inp.fr>
+# License: GPL v2 or later
+
+# Gateway between Git and MediaWiki.
+#   https://github.com/Bibzball/Git-Mediawiki/wiki
+#
+# Known limitations:
+#
+# - Only wiki pages are managed, no support for [[File:...]]
+#   attachments.
+#
+# - Poor performance in the best case: it takes forever to check
+#   whether we're up-to-date (on fetch or push) or to fetch a few
+#   revisions from a large wiki, because we use exclusively a
+#   page-based synchronization. We could switch to a wiki-wide
+#   synchronization when the synchronization involves few revisions
+#   but the wiki is large.
+#
+# - Git renames could be turned into MediaWiki renames (see TODO
+#   below)
+#
+# - login/password support requires the user to write the password
+#   cleartext in a file (see TODO below).
+#
+# - No way to import "one page, and all pages included in it"
+#
+# - Multiple remote MediaWikis have not been very well tested.
+
+use strict;
+use MediaWiki::API;
+use DateTime::Format::ISO8601;
+use encoding 'utf8';
+
+# use encoding 'utf8' doesn't change STDERROR
+# but we're going to output UTF-8 filenames to STDERR
+binmode STDERR, ":utf8";
+
+use URI::Escape;
+use warnings;
+
+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced
+use constant SLASH_REPLACEMENT => "%2F";
+
+# It's not always possible to delete pages (may require some
+# priviledges). Deleted pages are replaced with this content.
+use constant DELETED_CONTENT => "[[Category:Deleted]]\n";
+
+# It's not possible to create empty pages. New empty files in Git are
+# sent with this content instead.
+use constant EMPTY_CONTENT => "<!-- empty page -->\n";
+
+# used to reflect file creation or deletion in diff.
+use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
+
+my $remotename = $ARGV[0];
+my $url = $ARGV[1];
+
+# Accept both space-separated and multiple keys in config file.
+# Spaces should be written as _ anyway because we'll use chomp.
+my @tracked_pages = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".pages"));
+chomp(@tracked_pages);
+
+# Just like @tracked_pages, but for MediaWiki categories.
+my @tracked_categories = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".categories"));
+chomp(@tracked_categories);
+
+my $wiki_login = run_git("config --get remote.". $remotename .".mwLogin");
+# TODO: ideally, this should be able to read from keyboard, but we're
+# inside a remote helper, so our stdin is connect to git, not to a
+# terminal.
+my $wiki_passwd = run_git("config --get remote.". $remotename .".mwPassword");
+chomp ($wiki_login);
+chomp ($wiki_passwd);
+
+# Import only last revisions (both for clone and fetch)
+my $shallow_import = run_git("config --get --bool remote.". $remotename .".shallow");
+chomp($shallow_import);
+$shallow_import = ($shallow_import eq "true");
+
+my $wiki_name = $url;
+$wiki_name =~ s/[^\/]*:\/\///;
+
+my $import_started;
+
+# Commands parser
+my $entry;
+my @cmd;
+while (<STDIN>) {
+	chomp;
+	@cmd = split(/ /);
+	if (defined($cmd[0])) {
+		# Line not blank
+		if ($cmd[0] eq "capabilities") {
+			die("Too many arguments for capabilities") unless (!defined($cmd[1]));
+			mw_capabilities();
+		} elsif ($cmd[0] eq "list") {
+			die("Too many arguments for list") unless (!defined($cmd[2]));
+			mw_list($cmd[1]);
+		} elsif ($cmd[0] eq "import") {
+			die("Invalid arguments for import") unless ($cmd[1] ne "" && !defined($cmd[2]));
+			mw_import($cmd[1]);
+		} elsif ($cmd[0] eq "option") {
+			die("Too many arguments for option") unless ($cmd[1] ne "" && $cmd[2] ne "" && !defined($cmd[3]));
+			mw_option($cmd[1],$cmd[2]);
+		} elsif ($cmd[0] eq "push") {
+			# Check the pattern <src>:<dst>
+			my @pushargs = split(/:/,$cmd[1]);
+			die("Invalid arguments for push") unless ($pushargs[1] ne "" && !defined($pushargs[2]));
+			mw_push($pushargs[0],$pushargs[1]);
+		} else {
+			print STDERR "Unknown command. Aborting...\n";
+			last;
+		}
+	} else {
+		# blank line: we should terminate
+		last;
+	}
+
+	BEGIN { $| = 1 } # flush STDOUT, to make sure the previous
+			 # command is fully processed.
+}
+# End of input
+if ($import_started) {
+	# Terminate the fast-import stream properly.
+	# Git requires one "done" command, and only
+	# one This is OK since we only have one
+	# branch, so import will be called only once
+	# (plus once for HEAD, for which we won't
+	# reach this point).
+	print STDOUT "done\n";
+}
+BEGIN { $| = 1 };
+if (!eof(STDIN)) {
+	# Wait for Git to terminate. If we don't, git fetch
+	# (transport-helper.c's sendline function) will try to write
+	# to our stdin, which will be closed, and git fetch will be
+	# killed. That's probably a bug in transport-helper.c, but in
+	# the meantime ...
+	sleep .1;
+};
+
+########################## Functions ##############################
+
+# MediaWiki API instance, created lazily.
+my $mediawiki;
+
+sub mw_connect_maybe {
+	if ($mediawiki) {
+	    return;
+	}
+	$mediawiki = MediaWiki::API->new;
+	$mediawiki->{config}->{api_url} = "$url/api.php";
+	if ($wiki_login) {
+		if (!$mediawiki->login({
+			lgname => $wiki_login,
+			lgpassword => $wiki_passwd,
+		})) {
+			print STDERR "Failed to log in mediawiki user \"$wiki_login\" on $url\n";
+			print STDERR "(error " .
+			    $mediawiki->{error}->{code} . ': ' .
+			    $mediawiki->{error}->{details} . ")\n";
+			exit 1;
+		} else {
+			print STDERR "Logged in with user \"$wiki_login\".\n";
+		}
+	}
+}
+
+sub get_mw_first_pages {
+	my $some_pages = shift;
+	my @some_pages = @{$some_pages};
+
+	my $pages = shift;
+
+	# pattern 'page1|page2|...' required by the API
+	my $titles = join('|', @some_pages);
+
+	my $mw_pages = $mediawiki->api({
+		action => 'query',
+		titles => $titles,
+	});
+	if (!defined($mw_pages)) {
+		print STDERR "fatal: could not query the list of wiki pages.\n";
+		print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+		print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+		exit 1;
+	}
+	while (my ($id, $page) = each (%{$mw_pages->{query}->{pages}})) {
+		if ($id < 0) {
+			print STDERR "Warning: page $page->{title} not found on wiki\n";
+		} else {
+			$pages->{$page->{title}} = $page;
+		}
+	}
+}
+
+sub get_mw_pages {
+	mw_connect_maybe();
+
+	my %pages; # hash on page titles to avoid duplicates
+	my $user_defined;
+	if (@tracked_pages) {
+		$user_defined = 1;
+		# The user provided a list of pages titles, but we
+		# still need to query the API to get the page IDs.
+
+		my @some_pages = @tracked_pages;
+		while (@some_pages) {
+			my $last = 50;
+			if ($#some_pages < $last) {
+				$last = $#some_pages;
+			}
+			my @slice = @some_pages[0..$last];
+			get_mw_first_pages(\@slice, \%pages);
+			@some_pages = @some_pages[51..$#some_pages];
+		}
+	}
+	if (@tracked_categories) {
+		$user_defined = 1;
+		foreach my $category (@tracked_categories) {
+			if (index($category, ':') < 0) {
+				# Mediawiki requires the Category
+				# prefix, but let's not force the user
+				# to specify it.
+				$category = "Category:" . $category;
+			}
+			my $mw_pages = $mediawiki->list ( {
+				action => 'query',
+				list => 'categorymembers',
+				cmtitle => $category,
+				cmlimit => 'max' } )
+			    || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};
+			foreach my $page (@{$mw_pages}) {
+				$pages{$page->{title}} = $page;
+			}
+		}
+	}
+	if (!$user_defined) {
+		# No user-provided list, get the list of pages from
+		# the API.
+		my $mw_pages = $mediawiki->list({
+			action => 'query',
+			list => 'allpages',
+			aplimit => 500,
+		});
+		if (!defined($mw_pages)) {
+			print STDERR "fatal: could not get the list of wiki pages.\n";
+			print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+			print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+			exit 1;
+		}
+		foreach my $page (@{$mw_pages}) {
+			$pages{$page->{title}} = $page;
+		}
+	}
+	return values(%pages);
+}
+
+sub run_git {
+	open(my $git, "-|:encoding(UTF-8)", "git " . $_[0]);
+	my $res = do { local $/; <$git> };
+	close($git);
+
+	return $res;
+}
+
+
+sub get_last_local_revision {
+	# Get note regarding last mediawiki revision
+	my $note = run_git("notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null");
+	my @note_info = split(/ /, $note);
+
+	my $lastrevision_number;
+	if (!(defined($note_info[0]) && $note_info[0] eq "mediawiki_revision:")) {
+		print STDERR "No previous mediawiki revision found";
+		$lastrevision_number = 0;
+	} else {
+		# Notes are formatted : mediawiki_revision: #number
+		$lastrevision_number = $note_info[1];
+		chomp($lastrevision_number);
+		print STDERR "Last local mediawiki revision found is $lastrevision_number";
+	}
+	return $lastrevision_number;
+}
+
+sub get_last_remote_revision {
+	mw_connect_maybe();
+
+	my @pages = get_mw_pages();
+
+	my $max_rev_num = 0;
+
+	foreach my $page (@pages) {
+		my $id = $page->{pageid};
+
+		my $query = {
+			action => 'query',
+			prop => 'revisions',
+			rvprop => 'ids',
+			pageids => $id,
+		};
+
+		my $result = $mediawiki->api($query);
+
+		my $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});
+
+		$max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);
+	}
+
+	print STDERR "Last remote revision found is $max_rev_num.\n";
+	return $max_rev_num;
+}
+
+# Clean content before sending it to MediaWiki
+sub mediawiki_clean {
+	my $string = shift;
+	my $page_created = shift;
+	# Mediawiki does not allow blank space at the end of a page and ends with a single \n.
+	# This function right trims a string and adds a \n at the end to follow this rule
+	$string =~ s/\s+$//;
+	if ($string eq "" && $page_created) {
+		# Creating empty pages is forbidden.
+		$string = EMPTY_CONTENT;
+	}
+	return $string."\n";
+}
+
+# Filter applied on MediaWiki data before adding them to Git
+sub mediawiki_smudge {
+	my $string = shift;
+	if ($string eq EMPTY_CONTENT) {
+		$string = "";
+	}
+	# This \n is important. This is due to mediawiki's way to handle end of files.
+	return $string."\n";
+}
+
+sub mediawiki_clean_filename {
+	my $filename = shift;
+	$filename =~ s/@{[SLASH_REPLACEMENT]}/\//g;
+	# [, ], |, {, and } are forbidden by MediaWiki, even URL-encoded.
+	# Do a variant of URL-encoding, i.e. looks like URL-encoding,
+	# but with _ added to prevent MediaWiki from thinking this is
+	# an actual special character.
+	$filename =~ s/[\[\]\{\}\|]/sprintf("_%%_%x", ord($&))/ge;
+	# If we use the uri escape before
+	# we should unescape here, before anything
+
+	return $filename;
+}
+
+sub mediawiki_smudge_filename {
+	my $filename = shift;
+	$filename =~ s/\//@{[SLASH_REPLACEMENT]}/g;
+	$filename =~ s/ /_/g;
+	# Decode forbidden characters encoded in mediawiki_clean_filename
+	$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf("%c", hex($1))/ge;
+	return $filename;
+}
+
+sub literal_data {
+	my ($content) = @_;
+	print STDOUT "data ", bytes::length($content), "\n", $content;
+}
+
+sub mw_capabilities {
+	# Revisions are imported to the private namespace
+	# refs/mediawiki/$remotename/ by the helper and fetched into
+	# refs/remotes/$remotename later by fetch.
+	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
+	print STDOUT "import\n";
+	print STDOUT "list\n";
+	print STDOUT "push\n";
+	print STDOUT "\n";
+}
+
+sub mw_list {
+	# MediaWiki do not have branches, we consider one branch arbitrarily
+	# called master, and HEAD pointing to it.
+	print STDOUT "? refs/heads/master\n";
+	print STDOUT "\@refs/heads/master HEAD\n";
+	print STDOUT "\n";
+}
+
+sub mw_option {
+	print STDERR "remote-helper command 'option $_[0]' not yet implemented\n";
+	print STDOUT "unsupported\n";
+}
+
+sub fetch_mw_revisions_for_page {
+	my $page = shift;
+	my $id = shift;
+	my $fetch_from = shift;
+	my @page_revs = ();
+	my $query = {
+		action => 'query',
+		prop => 'revisions',
+		rvprop => 'ids',
+		rvdir => 'newer',
+		rvstartid => $fetch_from,
+		rvlimit => 500,
+		pageids => $id,
+	};
+
+	my $revnum = 0;
+	# Get 500 revisions at a time due to the mediawiki api limit
+	while (1) {
+		my $result = $mediawiki->api($query);
+
+		# Parse each of those 500 revisions
+		foreach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {
+			my $page_rev_ids;
+			$page_rev_ids->{pageid} = $page->{pageid};
+			$page_rev_ids->{revid} = $revision->{revid};
+			push (@page_revs, $page_rev_ids);
+			$revnum++;
+		}
+		last unless $result->{'query-continue'};
+		$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};
+	}
+	if ($shallow_import && @page_revs) {
+		print STDERR "  Found 1 revision (shallow import).\n";
+		@page_revs = sort {$b->{revid} <=> $a->{revid}} (@page_revs);
+		return $page_revs[0];
+	}
+	print STDERR "  Found ", $revnum, " revision(s).\n";
+	return @page_revs;
+}
+
+sub fetch_mw_revisions {
+	my $pages = shift; my @pages = @{$pages};
+	my $fetch_from = shift;
+
+	my @revisions = ();
+	my $n = 1;
+	foreach my $page (@pages) {
+		my $id = $page->{pageid};
+
+		print STDERR "page $n/", scalar(@pages), ": ". $page->{title} ."\n";
+		$n++;
+		my @page_revs = fetch_mw_revisions_for_page($page, $id, $fetch_from);
+		@revisions = (@page_revs, @revisions);
+	}
+
+	return ($n, @revisions);
+}
+
+sub import_file_revision {
+	my $commit = shift;
+	my %commit = %{$commit};
+	my $full_import = shift;
+	my $n = shift;
+
+	my $title = $commit{title};
+	my $comment = $commit{comment};
+	my $content = $commit{content};
+	my $author = $commit{author};
+	my $date = $commit{date};
+
+	print STDOUT "commit refs/mediawiki/$remotename/master\n";
+	print STDOUT "mark :$n\n";
+	print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+	literal_data($comment);
+
+	# If it's not a clone, we need to know where to start from
+	if (!$full_import && $n == 1) {
+		print STDOUT "from refs/mediawiki/$remotename/master^0\n";
+	}
+	if ($content ne DELETED_CONTENT) {
+		print STDOUT "M 644 inline $title.mw\n";
+		literal_data($content);
+		print STDOUT "\n\n";
+	} else {
+		print STDOUT "D $title.mw\n";
+	}
+
+	# mediawiki revision number in the git note
+	if ($full_import && $n == 1) {
+		print STDOUT "reset refs/notes/$remotename/mediawiki\n";
+	}
+	print STDOUT "commit refs/notes/$remotename/mediawiki\n";
+	print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+	literal_data("note added by git-mediawiki");
+	if (!$full_import && $n == 1) {
+		print STDOUT "from refs/notes/$remotename/mediawiki^0\n";
+	}
+	print STDOUT "N inline :$n\n";
+	literal_data("mediawiki_revision: " . $commit{mw_revision});
+	print STDOUT "\n\n";
+}
+
+sub mw_import {
+	$import_started = 1;
+	my $ref = shift;
+	# the remote helper will call "import HEAD" and
+	# "import refs/heads/master"
+	# Since HEAD is a symbolic ref to master (by convention,
+	# followed by the output of the command "list" that we gave),
+	# we don't need to do anything in this case.
+	if ($ref eq "HEAD") {
+		return;
+	}
+
+	mw_connect_maybe();
+
+	my @pages = get_mw_pages();
+
+	print STDERR "Searching revisions...\n";
+	my $last_local = get_last_local_revision();
+	my $fetch_from = $last_local + 1;
+	if ($fetch_from == 1) {
+		print STDERR ", fetching from beginning.\n";
+	} else {
+		print STDERR ", fetching from here.\n";
+	}
+	my ($n, @revisions) = fetch_mw_revisions(\@pages, $fetch_from);
+
+	# Creation of the fast-import stream
+	print STDERR "Fetching & writing export data...\n";
+
+	$n = 0;
+	my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined
+
+	foreach my $pagerevid (sort {$a->{revid} <=> $b->{revid}} @revisions) {
+		# fetch the content of the pages
+		my $query = {
+			action => 'query',
+			prop => 'revisions',
+			rvprop => 'content|timestamp|comment|user|ids',
+			revids => $pagerevid->{revid},
+		};
+
+		my $result = $mediawiki->api($query);
+
+		my $rev = pop(@{$result->{query}->{pages}->{$pagerevid->{pageid}}->{revisions}});
+
+		$n++;
+
+		my %commit;
+		$commit{author} = $rev->{user} || 'Anonymous';
+		$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
+		$commit{title} = mediawiki_smudge_filename(
+			$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}
+		    );
+		$commit{mw_revision} = $pagerevid->{revid};
+		$commit{content} = mediawiki_smudge($rev->{'*'});
+
+		if (!defined($rev->{timestamp})) {
+			$last_timestamp++;
+		} else {
+			$last_timestamp = $rev->{timestamp};
+		}
+		$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);
+
+		print STDERR "$n/", scalar(@revisions), ": Revision #$pagerevid->{revid} of $commit{title}\n";
+
+		import_file_revision(\%commit, ($fetch_from == 1), $n);
+	}
+
+	if ($fetch_from == 1) {
+		if ($n != 0) {
+			print STDOUT "reset $ref\n";
+			print STDOUT "from :$n\n";
+		} else {
+			print STDERR "You appear to have cloned an empty mediawiki\n";
+			# Something has to be done remote-helper side. If nothing is done, an error is
+			# thrown saying that HEAD is refering to unknown object 0000000000000000000
+		}
+	}
+}
+
+sub error_non_fast_forward {
+	# Native git-push would show this after the summary.
+	# We can't ask it to display it cleanly, so print it
+	# ourselves before.
+	print STDERR "To prevent you from losing history, non-fast-forward updates were rejected\n";
+	print STDERR "Merge the remote changes (e.g. 'git pull') before pushing again. See the\n";
+	print STDERR "'Note about fast-forwards' section of 'git push --help' for details.\n";
+
+	print STDOUT "error $_[0] \"non-fast-forward\"\n";
+	print STDOUT "\n";
+}
+
+sub mw_push_file {
+	my $diff_info = shift;
+	# $diff_info contains a string in this format:
+	# 100644 100644 <sha1_of_blob_before_commit> <sha1_of_blob_now> <status>
+	my @diff_info_split = split(/[ \t]/, $diff_info);
+
+	# Filename, including .mw extension
+	my $complete_file_name = shift;
+	# Commit message
+	my $summary = shift;
+
+	my $new_sha1 = $diff_info_split[3];
+	my $old_sha1 = $diff_info_split[2];
+	my $page_created = ($old_sha1 eq NULL_SHA1);
+	my $page_deleted = ($new_sha1 eq NULL_SHA1);
+	$complete_file_name = mediawiki_clean_filename($complete_file_name);
+
+	if (substr($complete_file_name,-3) eq ".mw"){
+		my $title = substr($complete_file_name,0,-3);
+
+		my $file_content;
+		if ($page_deleted) {
+			# Deleting a page usually requires
+			# special priviledges. A common
+			# convention is to replace the page
+			# with this content instead:
+			$file_content = DELETED_CONTENT;
+		} else {
+			$file_content = run_git("cat-file -p $new_sha1");
+		}
+
+		mw_connect_maybe();
+
+		my $result = $mediawiki->edit( {
+			action => 'edit',
+			summary => $summary,
+			title => $title,
+			text => mediawiki_clean($file_content, $page_created),
+				  }, {
+					  skip_encoding => 1 # Helps with names with accentuated characters
+				  }) || die 'Fatal: Error ' .
+				  $mediawiki->{error}->{code} .
+				  ' from mediwiki: ' . $mediawiki->{error}->{details};
+		print STDERR "Pushed file : $new_sha1 - $title\n";
+	} else {
+		print STDERR "$complete_file_name not a mediawiki file (Not pushable on this version).\n"
+	}
+}
+
+sub mw_push {
+	my $last_local_revid = get_last_local_revision();
+	print STDERR ".\n"; # Finish sentence started by get_last_local_revision()
+	my $last_remote_revid = get_last_remote_revision();
+
+	# Get sha1 of commit pointed by local HEAD
+	my $HEAD_sha1 = run_git("rev-parse $_[0] 2>/dev/null"); chomp($HEAD_sha1);
+	# Get sha1 of commit pointed by remotes/$remotename/master
+	my $remoteorigin_sha1 = run_git("rev-parse refs/remotes/$remotename/master 2>/dev/null");
+	chomp($remoteorigin_sha1);
+
+	if ($last_local_revid > 0 &&
+	    $last_local_revid < $last_remote_revid){
+		return error_non_fast_forward($_[0]);
+	}
+
+	if ($HEAD_sha1 eq $remoteorigin_sha1) {
+		print STDOUT "\n";
+		return;
+	}
+
+	# Get every commit in between HEAD and refs/remotes/origin/master,
+	# including HEAD and refs/remotes/origin/master
+	my @commit_pairs = ();
+	if ($last_local_revid > 0) {
+		my $parsed_sha1 = $remoteorigin_sha1;
+		# Find a path from last MediaWiki commit to pushed commit
+		while ($parsed_sha1 ne $HEAD_sha1) {
+			my @commit_info =  grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $_[0]")));
+			if (!@commit_info) {
+				return error_non_fast_forward($_[0]);
+			}
+			my @commit_info_split = split(/ |\n/, $commit_info[0]);
+			# $commit_info_split[1] is the sha1 of the commit to export
+			# $commit_info_split[0] is the sha1 of its direct child
+			push (@commit_pairs, \@commit_info_split);
+			$parsed_sha1 = $commit_info_split[1];
+		}
+	} else {
+		# No remote mediawiki revision. Export the whole
+		# history (linearized with --first-parent)
+		print STDERR "Warning: no common ancestor, pushing complete history\n";
+		my $history = run_git("rev-list --first-parent --children $_[0]");
+		my @history = split('\n', $history);
+		@history = @history[1..$#history];
+		foreach my $line (reverse @history) {
+			my @commit_info_split = split(/ |\n/, $line);
+			push (@commit_pairs, \@commit_info_split);
+		}
+	}
+
+	foreach my $commit_info_split (@commit_pairs) {
+		my $sha1_child = @{$commit_info_split}[0];
+		my $sha1_commit = @{$commit_info_split}[1];
+		my $diff_infos = run_git("diff-tree -r --raw -z $sha1_child $sha1_commit");
+		# TODO: we could detect rename, and encode them with a #redirect on the wiki.
+		# TODO: for now, it's just a delete+add
+		my @diff_info_list = split(/\0/, $diff_infos);
+		# Keep the first line of the commit message as mediawiki comment for the revision
+		my $commit_msg = (split(/\n/, run_git("show --pretty=format:\"%s\" $sha1_commit")))[0];
+		chomp($commit_msg);
+		# Push every blob
+		while (@diff_info_list) {
+			# git diff-tree -z gives an output like
+			# <metadata>\0<filename1>\0
+			# <metadata>\0<filename2>\0
+			# and we've split on \0.
+			my $info = shift(@diff_info_list);
+			my $file = shift(@diff_info_list);
+			mw_push_file($info, $file, $commit_msg);
+		}
+	}
+
+	print STDOUT "ok $_[1]\n";
+	print STDOUT "\n";
+
+	print STDERR "Just pushed some revisions to MediaWiki.\n";
+	print STDERR "The pushed revisions now have to be re-imported, and your current branch\n";
+	print STDERR "needs to be updated with these re-imported commits. You can do this with\n";
+	print STDERR "\n";
+	print STDERR "  git pull --rebase\n";
+	print STDERR "\n";
+}
diff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt
new file mode 100644
index 0000000..4d211f5
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki.txt
@@ -0,0 +1,7 @@
+Git-Mediawiki is a project which aims the creation of a gate
+between git and mediawiki, allowing git users to push and pull
+objects from mediawiki just as one would do with a classic git
+repository thanks to remote-helpers.
+
+For more information, visit the wiki at
+https://github.com/Bibzball/Git-Mediawiki/wiki
-- 
1.7.6.585.g5929f.dirty

Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@imag.fr> writes:

> In short, the changes since v3 are:
>
> * Adapt to newer Git, which seem to require a "done" command at the
>   end of the fast-import stream. I don't understand why this is
>   needed, since fast-import is called without the --done flag by
>   remote-helpers, but if I don't do this, "git fetch" doesn't
>   terminate and keeps waiting ...

Hmmmm, is this a regression in fast-import? Can this be bisected if so?

Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)

[Sverre Rabbelier]

Heya,

2011/8/26 Junio C Hamano <gitster@pobox.com>:
> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>> In short, the changes since v3 are:
>>
>> * Adapt to newer Git, which seem to require a "done" command at the
>>   end of the fast-import stream. I don't understand why this is
>>   needed, since fast-import is called without the --done flag by
>>   remote-helpers, but if I don't do this, "git fetch" doesn't
>>   terminate and keeps waiting ...
>
> Hmmmm, is this a regression in fast-import? Can this be bisected if so?

We agreed that making this change is the best way to go forward, since
there's so few (read: none) remote-helpers yet, the fix is trivial,
and it solves a design mistake while the api is not yet set in stone.

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)

[Junio C Hamano]

Sverre Rabbelier <srabbelier@gmail.com> writes:

> 2011/8/26 Junio C Hamano <gitster@pobox.com>:
>> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>>> In short, the changes since v3 are:
>>>
>>> * Adapt to newer Git, which seem to require a "done" command at the
>>>   end of the fast-import stream. I don't understand why this is
>>>   needed, since fast-import is called without the --done flag by
>>>   remote-helpers, but if I don't do this, "git fetch" doesn't
>>>   terminate and keeps waiting ...
>>
>> Hmmmm, is this a regression in fast-import? Can this be bisected if so?
>
> We agreed that making this change is the best way to go forward, since
> there's so few (read: none) remote-helpers yet, the fix is trivial,
> and it solves a design mistake while the api is not yet set in stone.

Does this exchange suggest that at least we would need an update to
documentation around "done", as Matthieu's "why is 'done' needed even
though I am not calling with --done?" sounds like a very fair question.

Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)

[Sverre Rabbelier]

Heya,

On Mon, Aug 29, 2011 at 08:05, Junio C Hamano <gitster@pobox.com> wrote:
> Does this exchange suggest that at least we would need an update to
> documentation around "done", as Matthieu's "why is 'done' needed even
> though I am not calling with --done?" sounds like a very fair question.

No I think the documentation for fast-import is correct. If you pass
--use-done-feature or print 'feature done' in the stream the use of
'done' is required, otherwise it isn't. We did recently changed git to
pass '--use-done-feature' to the fast-import process though :).

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)

[Jonathan Nieder]

Sverre Rabbelier wrote:
> On Mon, Aug 29, 2011 at 08:05, Junio C Hamano <gitster@pobox.com> wrote:

>> Does this exchange suggest that at least we would need an update to
>> documentation around "done", as Matthieu's "why is 'done' needed even
>> though I am not calling with --done?" sounds like a very fair question.
>
> No I think the documentation for fast-import is correct.

However, the documentation for git-remote-helpers is hopelessly outdated.

Here's an old attempt to make the documentation a little easier to read,
and hopefully also to add to.  The text from "Capabilities for Pushing"
on needs help to avoid seeming redundant in light of the list of
capabilities before, but I'm sending it now to get comments and
hopefully help from other interested people.

It does _not_ document recent changes; patches for that would
presumably come on top.  Last time I sent this to list[1], it brought
to life some inconsistencies in how the various "list" commands are
used.  Another response was to flesh out what a capability for
fetching a single tree ("git archive --remote") would look like[2],
for which I was very grateful (and I'd happy to see it come to life).

[1] http://thread.gmane.org/gmane.comp.version-control.git/169341
[2] http://colabti.org/irclogger/irclogger_log/git-devel?date=2011-03-19#l434

-- >8 --
Subject: Documentation/remote-helpers: explain capabilities first

The current remote helper documentation is from the perspective of
git, so to speak: it presents a full menu of commands for a person
invoking a remote helper to choose from.  In practice, that's less
useful than it could be, since the daunted novice remote-helper author
probably just wanted a list of commands needs to implement to get
started.  So preface the command list with an overview of each
capability, its purpose, and what commands it requires.

As a side effect, this makes it a little clearer that git doesn't
choose arbitrary commands to run, even if the remote helper advertises
all capabilities --- instead, there are well defined command sequences
for various tasks.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 Documentation/git-remote-helpers.txt |  159 +++++++++++++++++++++++++++-------
 1 files changed, 129 insertions(+), 30 deletions(-)

diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt
index 58f6ad49..f3fd63a4 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -23,22 +23,141 @@ output. Because a remote helper runs as an independent process from
 git, there is no need to re-link git to add a new helper, nor any
 need to link the helper with the implementation of git.
 
-Every helper must support the "capabilities" command, which git will
-use to determine what other commands the helper will accept.  Other
-commands generally concern facilities like discovering and updating
-remote refs, transporting objects between the object database and
-the remote repository, and updating the local object store.
-
-Helpers supporting the 'fetch' capability can discover refs from the
-remote repository and transfer objects reachable from those refs to
-the local object store. Helpers supporting the 'push' capability can
-transfer local objects to the remote repository and update remote refs.
+Every helper must support the "capabilities" command, which git
+uses to determine what other commands the helper will accept.  Those
+other commands can be used to discover and update remote refs,
+transport objects between the object database and the remote repository,
+and update the local object store.
 
 Git comes with a "curl" family of remote helpers, that handle various
 transport protocols, such as 'git-remote-http', 'git-remote-https',
 'git-remote-ftp' and 'git-remote-ftps'. They implement the capabilities
 'fetch', 'option', and 'push'.
 
+INPUT FORMAT
+------------
+
+Git sends the remote helper a list of commands on standard input, one
+per line.  The first command is always the 'capabilities' command, in
+response to which the remote helper must print a list of the
+capabilities it supports (see below) followed by a blank line.  The
+response to the capabilities command determines what commands Git uses
+in the remainder of the command stream.
+
+The command stream is terminated by a blank line.  In some cases
+(indicated in the documentation of the relevant commands), this blank
+line is followed by a payload in some other protocol (e.g., the pack
+protocol), while in others it indicates the end of input.
+
+Capabilities
+~~~~~~~~~~~~
+
+Each remote helper is expected to support only a subset of commands.
+The operations a helper supports are declared to git in the response
+to the `capabilities` command (see COMMANDS, below).
+
+'option'::
+	For specifying settings like `verbosity` (how much output to
+	write to stderr) and `depth` (how much history is wanted in the
+	case of a shallow clone) that affect how other commands are
+	carried out.
+
+'connect'::
+	For fetching and pushing using git's native packfile protocol
+	that requires a bidirectional, full-duplex connection.
+
+'push'::
+	For listing remote refs and pushing specified objects from the
+	local object store to remote refs.
+
+'fetch'::
+	For listing remote refs and fetching the associated history to
+	the local object store.
+
+'import'::
+	For listing remote refs and fetching the associated history as
+	a fast-import stream.
+
+'refspec' <refspec>::
+	This modifies the 'import' capability, allowing the produced
+	fast-import stream to modify refs in a private namespace
+	instead of writing to refs/heads or refs/remotes directly.
+	It is recommended that all importers providing the 'import'
+	capability use this.
++
+A helper advertising the capability
+`refspec refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}`
+is saying that, when it is asked to `import refs/heads/topic`, the
+stream it outputs will update the `refs/svn/origin/branches/topic`
+ref.
++
+This capability can be advertised multiple times.  The first
+applicable refspec takes precedence.  The left-hand of refspecs
+advertised with this capability must cover all refs reported by
+the list command.  If no 'refspec' capability is advertised,
+there is an implied `refspec {asterisk}:{asterisk}`.
+
+Capabilities for Pushing
+~~~~~~~~~~~~~~~~~~~~~~~~
+'connect'::
+	Can attempt to connect to 'git receive-pack' (for pushing),
+	'git upload-pack', etc for communication using the
+	packfile protocol.
++
+Supported commands: 'connect'.
+
+'push'::
+	Can discover remote refs and push local commits and the
+	history leading up to them to new or existing remote refs.
++
+Supported commands: 'list for-push', 'push'.
+
+If a helper advertises both 'connect' and 'push', git will use
+'connect' if possible and fall back to 'push' if the helper requests
+so when connecting (see the 'connect' command under COMMANDS).
+
+Capabilities for Fetching
+~~~~~~~~~~~~~~~~~~~~~~~~~
+'connect'::
+	Can try to connect to 'git upload-pack' (for fetching),
+	'git receive-pack', etc for communication using the
+	packfile protocol.
++
+Supported commands: 'connect'.
+
+'fetch'::
+	Can discover remote refs and transfer objects reachable from
+	them to the local object store.
++
+Supported commands: 'list', 'fetch'.
+
+'import'::
+	Can discover remote refs and output objects reachable from
+	them as a stream in fast-import format.
++
+Supported commands: 'list', 'import'.
+
+If a helper advertises 'connect', git will use it if possible and
+fall back to another capability if the helper requests so when
+connecting (see the 'connect' command under COMMANDS).
+When choosing between 'fetch' and 'import', git prefers 'fetch'.
+Other frontends may have some other order of preference.
+
+'refspec' <refspec>::
+	This modifies the 'import' capability.
++
+A helper advertising
+`refspec refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}`
+in its capabilities is saying that, when it handles
+`import refs/heads/topic`, the stream it outputs will update the
+`refs/svn/origin/branches/topic` ref.
++
+This capability can be advertised multiple times.  The first
+applicable refspec takes precedence.  The left-hand of refspecs
+advertised with this capability must cover all refs reported by
+the list command.  If no 'refspec' capability is advertised,
+there is an implied `refspec {asterisk}:{asterisk}`.
+
 INVOCATION
 ----------
 
@@ -167,26 +286,6 @@ completing a valid response for the current command.
 Additional commands may be supported, as may be determined from
 capabilities reported by the helper.
 
-CAPABILITIES
-------------
-
-'fetch'::
-'option'::
-'push'::
-'import'::
-'connect'::
-	This helper supports the corresponding command with the same name.
-
-'refspec' 'spec'::
-	When using the import command, expect the source ref to have
-	been written to the destination ref. The earliest applicable
-	refspec takes precedence. For example
-	"refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}" means
-	that, after an "import refs/heads/name", the script has written to
-	refs/svn/origin/branches/name. If this capability is used at
-	all, it must cover all refs reported by the list command; if
-	it is not used, it is effectively "{asterisk}:{asterisk}"
-
 REF LIST ATTRIBUTES
 -------------------
 
-- 
1.7.6

Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> Sverre Rabbelier wrote:
>> On Mon, Aug 29, 2011 at 08:05, Junio C Hamano <gitster@pobox.com> wrote:
>
>>> Does this exchange suggest that at least we would need an update to
>>> documentation around "done", as Matthieu's "why is 'done' needed even
>>> though I am not calling with --done?" sounds like a very fair question.
>>
>> No I think the documentation for fast-import is correct.
>
> However, the documentation for git-remote-helpers is hopelessly outdated.

Good point. If somebody writes a working remote-helper using "import" to
trigger fast-import as the back-end, and if it breaks because we changed
the way fast-import is invoked and the helper needs to be updated, we at
least need to give them a useful documentation that clearly states how the
helper is expected to behave.

> It does _not_ document recent changes; patches for that would
> presumably come on top.

Will queue this version for now, if only as a reminder that we would need
more updates.

Thanks.

Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)

[Matthieu Moy]

Jonathan Nieder <jrnieder@gmail.com> writes:

> Here's an old attempt to make the documentation a little easier to read,
> and hopefully also to add to.

Thanks, that helps. I wish we had all this earlier ;-).

> +Git sends the remote helper a list of commands on standard input, one
> +per line.  The first command is always the 'capabilities' command, 

Do we want to set this in stone? Wouldn't a Git implementation calling
"option" before "capabilities" be correct?

> +Capabilities
> +~~~~~~~~~~~~

(perhaps name the section "Overview of Capabilities"?)

[...]
> +'refspec' <refspec>::
> +	This modifies the 'import' capability, allowing the produced
> +	fast-import stream to modify refs in a private namespace
> +	instead of writing to refs/heads or refs/remotes directly.
> +	It is recommended that all importers providing the 'import'
> +	capability use this.
> ++
> +A helper advertising the capability
> +`refspec refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}`
> +is saying that, when it is asked to `import refs/heads/topic`, the
> +stream it outputs will update the `refs/svn/origin/branches/topic`
> +ref.
> ++
> +This capability can be advertised multiple times.  The first
> +applicable refspec takes precedence.  The left-hand of refspecs
> +advertised with this capability must cover all refs reported by
> +the list command.  If no 'refspec' capability is advertised,
> +there is an implied `refspec {asterisk}:{asterisk}`.

Since this "Capabilities" section is meant to be an overview, I'd
shorten this to

+'refspec' <refspec>::
+	This modifies the 'import' capability, allowing the produced
+	fast-import stream to modify refs in a private namespace
+	instead of writing to refs/heads or refs/remotes directly.

and drop the detailed explanation here.

> +Capabilities for Fetching
> +~~~~~~~~~~~~~~~~~~~~~~~~~
[...]
> +'refspec' <refspec>::
> +	This modifies the 'import' capability.

Since this would be the "detailed explanation" part, this is the one
readers will read more carefully, so I'd put the recommandation right
here:

+	It is recommended that all importers providing the 'import'
+	capability use this.

and of course, keep this:

> ++
> +A helper advertising
> +`refspec refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}`
> +in its capabilities is saying that, when it handles
> +`import refs/heads/topic`, the stream it outputs will update the
> +`refs/svn/origin/branches/topic` ref.
> ++
> +This capability can be advertised multiple times.  The first
> +applicable refspec takes precedence.  The left-hand of refspecs
> +advertised with this capability must cover all refs reported by
> +the list command.  If no 'refspec' capability is advertised,
> +there is an implied `refspec {asterisk}:{asterisk}`.
> +

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)

[Jonathan Nieder]

Hi Matthieu,

Matthieu Moy wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:

>> Here's an old attempt to make the documentation a little easier to read,
>> and hopefully also to add to.
>
> Thanks, that helps. I wish we had all this earlier ;-).

Glad you like it.

>> +Git sends the remote helper a list of commands on standard input, one
>> +per line.  The first command is always the 'capabilities' command, 
>
> Do we want to set this in stone?

Yes, I think we should set it in stone.  Helpers can use that as a
sanity check to know they are actually being used as a remote
helper.  And specifying it makes the protocol more concrete.

> Wouldn't a Git implementation calling
> "option" before "capabilities" be correct?

At that point git doesn't know if the helper implements the "option"
capability.

>> +Capabilities
>> +~~~~~~~~~~~~
>
> (perhaps name the section "Overview of Capabilities"?)

Yes, good idea.

[...]
> Since this "Capabilities" section is meant to be an overview, I'd
> shorten this to
>
> +'refspec' <refspec>::
> +	This modifies the 'import' capability, allowing the produced
> +	fast-import stream to modify refs in a private namespace
> +	instead of writing to refs/heads or refs/remotes directly.
>
> and drop the detailed explanation here.

Another good idea.

>> +Capabilities for Fetching
>> +~~~~~~~~~~~~~~~~~~~~~~~~~
> [...]
>> +'refspec' <refspec>::
>> +	This modifies the 'import' capability.
>
> Since this would be the "detailed explanation" part, this is the one
> readers will read more carefully, so I'd put the recommandation right
> here:
>
> +	It is recommended that all importers providing the 'import'
> +	capability use this.

Yet another. :)

> and of course, keep this:
>
>> ++
>> +A helper advertising
>> +`refspec refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}`
>> +in its capabilities is saying that, when it handles
>> +`import refs/heads/topic`, the stream it outputs will update the
>> +`refs/svn/origin/branches/topic` ref.
>> ++
>> +This capability can be advertised multiple times.  The first
>> +applicable refspec takes precedence.  The left-hand of refspecs
>> +advertised with this capability must cover all refs reported by
>> +the list command.  If no 'refspec' capability is advertised,
>> +there is an implied `refspec {asterisk}:{asterisk}`.
>> +

Thanks much!  Will try to find time to resend later today.

done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))

[Matthieu Moy]

Sverre Rabbelier <srabbelier@gmail.com> writes:

> Heya,
>
> On Mon, Aug 29, 2011 at 08:05, Junio C Hamano <gitster@pobox.com> wrote:
>> Does this exchange suggest that at least we would need an update to
>> documentation around "done", as Matthieu's "why is 'done' needed even
>> though I am not calling with --done?" sounds like a very fair question.
>
> No I think the documentation for fast-import is correct. If you pass
> --use-done-feature or print 'feature done' in the stream the use of
> 'done' is required, otherwise it isn't. We did recently changed git to
> pass '--use-done-feature' to the fast-import process though :).

That doesn't help much someone writting a remote helper.

The documentation for remote-helpers neither talks about "done" nor
about "--use-done-feature" or whatever way Git uses this feature when
using remote-helpers.

The current state is particularly confusing: git seems to expect one and
only one "done" feature, even when multiple "import" commands are
issued. That's very strange, and I'm not sure whether it's the expected
behavior (I can try a documentation patch, but I need to understand
better what's expected and what's not).

It would be natural to write remote-helpers like

while ($cmd = <read command>) {
    if ($cmd eq "import") {
       <write fast-import stream>
       print "done\n";
    } ...
}

but in the current state, it doesn't work since we'll get a first
"import HEAD", issue a "done", then get a "import refs/heads/master" and
write to a dead pipe.

Then, it would be very tempting to write it like

while ($cmd = <read command>) {
    if ($cmd eq "import") {
       <write fast-import stream>
    } ...
}
print "done\n";

but this doesn't work either, because when calling "git push", no
"import" command is involved, no fast-import is started, and the "done"
breaks everything.

That's why I had to make it like

my $import_started;
while ($cmd = <read command>) {
    if ($cmd eq "import") {
       $import_started = 1;
       <write fast-import stream>
    } ...
}
if ($import_started) {
    print "done\n";
}

and I really had the feeling I was working around a mis-feature of Git
here.

So, is this the expected behavior? Wouldn't it be more sensible to allow
the remote-helper to issue a "done" after each "import" command? Right
now, my understanding is that after an "import" command is issued, it's
no longer possible to output anything other than fast-import stream on
stdout, and I guess it'd be more future-proof to allow closing the
fast-import with a "done", and allow any dialog between git and the
remote helper afterwards.

Thanks,

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))

[Sverre Rabbelier]

Heya,

On Wed, Aug 31, 2011 at 14:05, Matthieu Moy
<Matthieu.Moy@grenoble-inp.fr> wrote:
> So, is this the expected behavior? Wouldn't it be more sensible to allow
> the remote-helper to issue a "done" after each "import" command? Right
> now, my understanding is that after an "import" command is issued, it's
> no longer possible to output anything other than fast-import stream on
> stdout, and I guess it'd be more future-proof to allow closing the
> fast-import with a "done", and allow any dialog between git and the
> remote helper afterwards.

Wow, no that's not the intended behavior. We meant to make it exactly
as you describe, after each import command you end with a done. This
is (should) also be what the testgit implementation does currently,
and it's what my remote-hg helper does as well. I'm not sure why it's
not working for you, but if it's not then that is definitely a bug,
and not intended behavior.

-- 
Cheers,

Sverre Rabbelier

Re: done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))

[Matthieu Moy]

Sverre Rabbelier <srabbelier@gmail.com> writes:

> Heya,
>
> On Wed, Aug 31, 2011 at 14:05, Matthieu Moy
> <Matthieu.Moy@grenoble-inp.fr> wrote:
>> So, is this the expected behavior? Wouldn't it be more sensible to allow
>> the remote-helper to issue a "done" after each "import" command? Right
>> now, my understanding is that after an "import" command is issued, it's
>> no longer possible to output anything other than fast-import stream on
>> stdout, and I guess it'd be more future-proof to allow closing the
>> fast-import with a "done", and allow any dialog between git and the
>> remote helper afterwards.
>
> Wow, no that's not the intended behavior. We meant to make it exactly
> as you describe, after each import command you end with a done. This
> is (should) also be what the testgit implementation does currently,

Err, no, it isn't. From git-remote-testgit.py:

def do_import(repo, args):
[...]
    while True:
        line = sys.stdin.readline()
[...]
        # strip of leading 'import '
        ref = line[7:].strip()
        refs.append(ref)

    repo = update_local_repo(repo)
    repo.exporter.export_repo(repo.gitdir, refs)

    print "done"

What it does is that it reads multiple "import" commands, and process
them all at once, with a single "print done" at the end. Actually,
testgit would die("Expected import line.") if Git sent another command
after "import".

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))

[Sverre Rabbelier]

Heya,

On Wed, Aug 31, 2011 at 14:55, Matthieu Moy
<Matthieu.Moy@grenoble-inp.fr> wrote:
> Err, no, it isn't. From git-remote-testgit.py:
>
> def do_import(repo, args):
> [...]
>    while True:
>        line = sys.stdin.readline()
> [...]
>        # strip of leading 'import '
>        ref = line[7:].strip()
>        refs.append(ref)
>
>    repo = update_local_repo(repo)
>    repo.exporter.export_repo(repo.gitdir, refs)
>
>    print "done"
>
> What it does is that it reads multiple "import" commands, and process
> them all at once, with a single "print done" at the end. Actually,
> testgit would die("Expected import line.") if Git sent another command
> after "import".

Ah, see, this is why in my original version the syntax was:

import
refs/heads/master
refs/heads/next
\n

Instead of the current:

import refs/heads/master
import refs/heads/next
\n

You can have multiple imports in the current system, you just need to
remember to add the trailing newline.

import refs/heads/master
\n
import refs/heads/next
\n

In the above case you'll have to have two done commands.

-- 
Cheers,

Sverre Rabbelier

Re: done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))

[Matthieu Moy]

Sverre Rabbelier <srabbelier@gmail.com> writes:

> You can have multiple imports in the current system, you just need to
> remember to add the trailing newline.
>
> import refs/heads/master
> \n
> import refs/heads/next
> \n

Then I'm lost. Isn't \n supposed to mean that the list of commands is
over, and that the remote-helper should terminate?

Also, who is "you" in your sentence? It can't be the remote helper
(which reads this sequence), so it has to be Git's transport-helper. Are
you saying that the transport-helper should be modified to add \n after
sending an import command?

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: done feature in remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))

[Sverre Rabbelier]

Heya,

On Wed, Aug 31, 2011 at 15:12, Matthieu Moy
<Matthieu.Moy@grenoble-inp.fr> wrote:
> Sverre Rabbelier <srabbelier@gmail.com> writes:
>
>> You can have multiple imports in the current system, you just need to
>> remember to add the trailing newline.
>>
>> import refs/heads/master
>> \n
>> import refs/heads/next
>> \n
>
> Then I'm lost. Isn't \n supposed to mean that the list of commands is
> over, and that the remote-helper should terminate?

No, a list of 'import' statements has to be followed by a \n, similar
to the push command. You can have multiple 'blocks' of import/push
commands, but each block has to be followed by a newline. Again, you
should read:

import A
import B
\n
import C
import D
\n

as:
import
A
B
\n
import
C
D
\n

> Also, who is "you" in your sentence? It can't be the remote helper
> (which reads this sequence), so it has to be Git's transport-helper.

Correct.

> Are you saying that the transport-helper should be modified to add \n
> after sending an import command?

As said above, it already does that. Each group of import statements
is terminated with a \n.

-- 
Cheers,

Sverre Rabbelier

[PATCH] git-remote-helpers.txt: explain how import works with multiple refs

[Matthieu Moy]

This is important for two reasons:

* when two "import" lines follow each other, only one "done" command
  should be issued in the fast-import stream, not one per "import".

* The blank line terminating an import command should not be confused
  with the one terminating the sequence of commands.

While we're there, illustrate the corresponding explanation for push
batches with an example.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
Sverre Rabbelier <srabbelier@gmail.com> writes:

> No, a list of 'import' statements has to be followed by a \n, similar
> to the push command. You can have multiple 'blocks' of import/push
> commands, but each block has to be followed by a newline. Again, you
> should read:
>
> import A
> import B
> \n
> import C
> import D
> \n

Thanks for the explanation. This was documented for 'push', but your
illustration made it easier for me to understand, so I've added the
example for push.

The documentation of "batch import" also fixes the "it's not
documented that remote-helpers should use 'done'" as a side-effect.

While investigating all this, I found in git-remote-testgit.py that
the remote-helpers had an "export" functionality, which seems totally
undocumented. Any volunteer to document it?

 Documentation/git-remote-helpers.txt |   22 +++++++++++++++++++++-
 1 files changed, 21 insertions(+), 1 deletions(-)

diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt
index 760b164..526fc6a 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -241,7 +241,22 @@ Supported if the helper has the "fetch" capability.
 'push' +<src>:<dst>::
 	Pushes the given local <src> commit or branch to the
 	remote branch described by <dst>.  A batch sequence of
-	one or more push commands is terminated with a blank line.
+	one or more 'push' commands is terminated with a blank line
+	(if there is only one reference to push, a single 'push' command
+	is followed by a blank line). For example, the following would
+	be two batches of 'push', the first asking the remote-helper
+	to push the local ref 'master' to the remote ref 'master' and
+	the local 'HEAD' to the remote 'branch', and the second
+	asking to push ref 'foo' to ref 'bar' (forced update requested
+	by the '+').
++
+------------
+push refs/heads/master:refs/heads/master
+push HEAD:refs/heads/branch
+\n
+push +refs/heads/foo:refs/heads/bar
+\n
+------------
 +
 Zero or more protocol options may be entered after the last 'push'
 command, before the batch's terminating blank line.
@@ -266,6 +281,11 @@ Supported if the helper has the "push" capability.
 Especially useful for interoperability with a foreign versioning
 system.
 +
+Just like 'push', a batch sequence of one or more 'import' is
+terminated with a blank line. For each batch of 'import', the remote
+helper should produce a fast-import stream terminated by a 'done'
+command.
++
 Supported if the helper has the "import" capability.
 
 'connect' <service>::
-- 
1.7.7.rc0.78.ge4b5a.dirty

[PATCH] (short) documentation for the testgit remote helper

[Matthieu Moy]

While it's not a command meant to be used by actual users (hence, not
mentionned in git(1)), this command is a very precious help for
remote-helpers authors.

The best place for such technical doc is the source code, but users may
not find it without a link in a manpage.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---

While we're improving the docs, this is one more thing that would have
saved me some time ...

 Documentation/git-remote-helpers.txt |    2 ++
 Documentation/git-remote-testgit.txt |   30 ++++++++++++++++++++++++++++++
 git-remote-testgit.py                |   14 ++++++++++++++
 3 files changed, 46 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/git-remote-testgit.txt

diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt
index 526fc6a..674797c 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -362,6 +362,8 @@ SEE ALSO
 --------
 linkgit:git-remote[1]
 
+linkgit:git-remote-testgit[1]
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/Documentation/git-remote-testgit.txt b/Documentation/git-remote-testgit.txt
new file mode 100644
index 0000000..2a67d45
--- /dev/null
+++ b/Documentation/git-remote-testgit.txt
@@ -0,0 +1,30 @@
+git-remote-testgit(1)
+=====================
+
+NAME
+----
+git-remote-testgit - Example remote-helper
+
+
+SYNOPSIS
+--------
+[verse]
+git clone testgit::<source-repo> [<destination>]
+
+DESCRIPTION
+-----------
+
+This command is a simple remote-helper, that is used both as a
+testcase for the remote-helper functionality, and as an example to
+show remote-helper authors one possible implementation.
+
+The best way to learn more is to read the comments and source code in
+'git-remote-testgit.py'.
+
+SEE ALSO
+--------
+linkgit:git-remote-helpers[1]
+
+GIT
+---
+Part of the linkgit:git[1] suite
diff --git a/git-remote-testgit.py b/git-remote-testgit.py
index e9c832b..91d4409 100644
--- a/git-remote-testgit.py
+++ b/git-remote-testgit.py
@@ -1,5 +1,19 @@
 #!/usr/bin/env python
 
+# This command is a simple remote-helper, that is used both as a
+# testcase for the remote-helper functionality, and as an example to
+# show remote-helper authors one possible implementation.
+#
+# This is a Git <-> Git importer/exporter, that simply uses git
+# fast-import and git fast-export to consume and produce fast-import
+# streams.
+#
+# To understand better the way things work, one can set the variable
+# "static int debug" in transport-helper.c to 1, and/or the "DEBUG"
+# variable in git_remote_helpers/util.py to True, and try various
+# commands.
+
+
 # hashlib is only available in python >= 2.5
 try:
     import hashlib
-- 
1.7.7.rc0.75.g56f27

Re: [PATCH] (short) documentation for the testgit remote helper

[Sverre Rabbelier]

Heya,

On Wed, Aug 31, 2011 at 20:14, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
> +# To understand better the way things work, one can set the variable
> +# "static int debug" in transport-helper.c to 1, and/or the "DEBUG"
> +# variable in git_remote_helpers/util.py to True, and try various
> +# commands.

Both are controlled by an environmental variable, it would be better
to mention these directly.

I think that for remote-testgit it is GIT_DEBUG_TESTGIT=1.

Other than that:

Acked-by: Sverre Rabbelier <srabbelier@gmail.com>

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH] (short) documentation for the testgit remote helper

[Matthieu Moy]

Sverre Rabbelier <srabbelier@gmail.com> writes:

> Heya,
>
> On Wed, Aug 31, 2011 at 20:14, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
>> +# To understand better the way things work, one can set the variable
>> +# "static int debug" in transport-helper.c to 1, and/or the "DEBUG"
>> +# variable in git_remote_helpers/util.py to True, and try various
>> +# commands.
>
> Both are controlled by an environmental variable, it would be better
> to mention these directly.
>
> I think that for remote-testgit it is GIT_DEBUG_TESTGIT=1.

Oops, I had missed it. Will resend.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

[PATCH 1/2 v2] Documentation/git-remote-helpers: explain how import works with multiple refs

[Matthieu Moy]

This is important for two reasons:

* when two "import" lines follow each other, only one "done" command
  should be issued in the fast-import stream, not one per "import".

* The blank line terminating an import command should not be confused
  with the one terminating the sequence of commands.

While we're there, illustrate the corresponding explanation for push
batches with an example.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
No real change since v1, but resending together with the other to make
it a "patch serie" ;-)

 Documentation/git-remote-helpers.txt |   22 +++++++++++++++++++++-
 1 files changed, 21 insertions(+), 1 deletions(-)

diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt
index 760b164..526fc6a 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -241,7 +241,22 @@ Supported if the helper has the "fetch" capability.
 'push' +<src>:<dst>::
 	Pushes the given local <src> commit or branch to the
 	remote branch described by <dst>.  A batch sequence of
-	one or more push commands is terminated with a blank line.
+	one or more 'push' commands is terminated with a blank line
+	(if there is only one reference to push, a single 'push' command
+	is followed by a blank line). For example, the following would
+	be two batches of 'push', the first asking the remote-helper
+	to push the local ref 'master' to the remote ref 'master' and
+	the local 'HEAD' to the remote 'branch', and the second
+	asking to push ref 'foo' to ref 'bar' (forced update requested
+	by the '+').
++
+------------
+push refs/heads/master:refs/heads/master
+push HEAD:refs/heads/branch
+\n
+push +refs/heads/foo:refs/heads/bar
+\n
+------------
 +
 Zero or more protocol options may be entered after the last 'push'
 command, before the batch's terminating blank line.
@@ -266,6 +281,11 @@ Supported if the helper has the "push" capability.
 Especially useful for interoperability with a foreign versioning
 system.
 +
+Just like 'push', a batch sequence of one or more 'import' is
+terminated with a blank line. For each batch of 'import', the remote
+helper should produce a fast-import stream terminated by a 'done'
+command.
++
 Supported if the helper has the "import" capability.
 
 'connect' <service>::
-- 
1.7.7.rc0.75.g56f27

[PATCH 2/2 v2] (short) documentation for the testgit remote helper

[Matthieu Moy]

While it's not a command meant to be used by actual users (hence, not
mentionned in git(1)), this command is a very precious help for
remote-helpers authors.

The best place for such technical doc is the source code, but users may
not find it without a link in a manpage.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
change since v1: document environment variables for debugging, not
hacking the source code.

 Documentation/git-remote-helpers.txt |    2 ++
 Documentation/git-remote-testgit.txt |   30 ++++++++++++++++++++++++++++++
 git-remote-testgit.py                |   13 +++++++++++++
 3 files changed, 45 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/git-remote-testgit.txt

diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt
index 526fc6a..674797c 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -362,6 +362,8 @@ SEE ALSO
 --------
 linkgit:git-remote[1]
 
+linkgit:git-remote-testgit[1]
+
 GIT
 ---
 Part of the linkgit:git[1] suite
diff --git a/Documentation/git-remote-testgit.txt b/Documentation/git-remote-testgit.txt
new file mode 100644
index 0000000..2a67d45
--- /dev/null
+++ b/Documentation/git-remote-testgit.txt
@@ -0,0 +1,30 @@
+git-remote-testgit(1)
+=====================
+
+NAME
+----
+git-remote-testgit - Example remote-helper
+
+
+SYNOPSIS
+--------
+[verse]
+git clone testgit::<source-repo> [<destination>]
+
+DESCRIPTION
+-----------
+
+This command is a simple remote-helper, that is used both as a
+testcase for the remote-helper functionality, and as an example to
+show remote-helper authors one possible implementation.
+
+The best way to learn more is to read the comments and source code in
+'git-remote-testgit.py'.
+
+SEE ALSO
+--------
+linkgit:git-remote-helpers[1]
+
+GIT
+---
+Part of the linkgit:git[1] suite
diff --git a/git-remote-testgit.py b/git-remote-testgit.py
index e9c832b..3dc4851 100644
--- a/git-remote-testgit.py
+++ b/git-remote-testgit.py
@@ -1,5 +1,18 @@
 #!/usr/bin/env python
 
+# This command is a simple remote-helper, that is used both as a
+# testcase for the remote-helper functionality, and as an example to
+# show remote-helper authors one possible implementation.
+#
+# This is a Git <-> Git importer/exporter, that simply uses git
+# fast-import and git fast-export to consume and produce fast-import
+# streams.
+#
+# To understand better the way things work, one can activate debug
+# traces by setting (to any value) the environment variables
+# GIT_TRANSPORT_HELPER_DEBUG and GIT_DEBUG_TESTGIT, to see messages
+# from the transport-helper side, or from this example remote-helper.
+
 # hashlib is only available in python >= 2.5
 try:
     import hashlib
-- 
1.7.7.rc0.75.g56f27

Re: [PATCH 2/2 v2] (short) documentation for the testgit remote helper

[Sverre Rabbelier]

Heya,

On Thu, Sep 1, 2011 at 18:49, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
> While it's not a command meant to be used by actual users (hence, not
> mentionned in git(1)), this command is a very precious help for
> remote-helpers authors.
>
> The best place for such technical doc is the source code, but users may
> not find it without a link in a manpage.
>
> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>

Acked-by: Sverre Rabbelier <srabbelier@gmail.com>

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH 1/2 v2] Documentation/git-remote-helpers: explain how import works with multiple refs

[Sverre Rabbelier]

Heya,

On Thu, Sep 1, 2011 at 18:49, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
> This is important for two reasons:
>
> * when two "import" lines follow each other, only one "done" command
>  should be issued in the fast-import stream, not one per "import".
>
> * The blank line terminating an import command should not be confused
>  with the one terminating the sequence of commands.
>
> While we're there, illustrate the corresponding explanation for push
> batches with an example.
>
> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>

Acked-by: Sverre Rabbelier <srabbelier@gmail.com>

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH] git-remote-helpers.txt: explain how import works with multiple refs

[Sverre Rabbelier]

Heya,

[+Jonathan Nieder]

On Wed, Aug 31, 2011 at 18:47, Matthieu Moy <Matthieu.Moy@imag.fr> wrote:
> This is important for two reasons:
>
> * when two "import" lines follow each other, only one "done" command
>  should be issued in the fast-import stream, not one per "import".
>
> * The blank line terminating an import command should not be confused
>  with the one terminating the sequence of commands.
>
> While we're there, illustrate the corresponding explanation for push
> batches with an example.

Thank you!

Acked-by: Sverre Rabbelier <srabbelier@gmail.com

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH] git-remote-helpers.txt: explain how import works with multiple refs

[Jonathan Nieder]

Hi Matthieu,

Matthieu Moy wrote:

> --- a/Documentation/git-remote-helpers.txt
> +++ b/Documentation/git-remote-helpers.txt
> @@ -241,7 +241,22 @@ Supported if the helper has the "fetch" capability.
>  'push' +<src>:<dst>::
>  	Pushes the given local <src> commit or branch to the
>  	remote branch described by <dst>.  A batch sequence of
> -	one or more push commands is terminated with a blank line.
> +	one or more 'push' commands is terminated with a blank line
> +	(if there is only one reference to push, a single 'push' command
> +	is followed by a blank line). For example, the following would
> +	be two batches of 'push', the first asking the remote-helper
> +	to push the local ref 'master' to the remote ref 'master' and
> +	the local 'HEAD' to the remote 'branch', and the second
> +	asking to push ref 'foo' to ref 'bar' (forced update requested
> +	by the '+').
> ++
> +------------
> +push refs/heads/master:refs/heads/master
> +push HEAD:refs/heads/branch
> +\n
> +push +refs/heads/foo:refs/heads/bar
> +\n
> +------------

Probably examples like this could go in a later EXAMPLES section.

At first I was worried about this not actually working, thinking
"push" might have traditionally had the newline-ends-command-stream
semantics that "connect" has.  But the push codepath does not set the
no_disconnect_req flag, so I was worrying in vain. :)  The stream
passed to the helper ends with two newlines when git pushes.

>  +
>  Zero or more protocol options may be entered after the last 'push'
>  command, before the batch's terminating blank line.
> @@ -266,6 +281,11 @@ Supported if the helper has the "push" capability.
>  Especially useful for interoperability with a foreign versioning
>  system.
>  +
> +Just like 'push', a batch sequence of one or more 'import' is
> +terminated with a blank line. For each batch of 'import', the remote
> +helper should produce a fast-import stream terminated by a 'done'
> +command.
> ++
>  Supported if the helper has the "import" capability.

This documents v1.7.7-rc0~61^2~3 (transport-helper: change import
semantics, 2011-07-16).  Such a documentation fix was desperately
needed; thank you!

For what it's worth, with or without a change to prevent the example
from interrupting the flow of description,
Acked-by: Jonathan Nieder <jrnieder@gmail.com>

Re: [PATCH] git-remote-helpers.txt: explain how import works with multiple refs

[Matthieu Moy]

Jonathan Nieder <jrnieder@gmail.com> writes:

>> +	is followed by a blank line). For example, the following would
>> +	be two batches of 'push', the first asking the remote-helper
>> +	to push the local ref 'master' to the remote ref 'master' and
>> +	the local 'HEAD' to the remote 'branch', and the second
>> +	asking to push ref 'foo' to ref 'bar' (forced update requested
>> +	by the '+').
>> ++
>> +------------
>> +push refs/heads/master:refs/heads/master
>> +push HEAD:refs/heads/branch
>> +\n
>> +push +refs/heads/foo:refs/heads/bar
>> +\n
>> +------------
>
> Probably examples like this could go in a later EXAMPLES section.

I think having a few examples within the explanations helps the reader
visualize what commands look like, and understand better further
explanations.

Plus, the example explains quickly the + which isn't documented
otherwise (but people reading this probably already know what a refspec
is and what the + normally means in this context so it's probably
sufficient to explain it as a side remark in an example)

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

[PATCH v5] Add a remote helper to interact with mediawiki (fetch & push)

[Matthieu Moy]

From: Jeremie Nikaes <jeremie.nikaes@ensimag.imag.fr>

Implement a gate between git and mediawiki, allowing git users to push
and pull objects from mediawiki just as one would do with a classic git
repository thanks to remote-helpers.

The following packages need to be installed (available on common
repositories):

     libmediawiki-api-perl
     libdatetime-format-iso8601-perl

Use remote helpers in order to be as transparent as possible to the git
user.

Download Mediawiki revisions through the Mediawiki API and then
fast-import into git.

Mediawiki revision number and git commits are linked thanks to notes
bound to commits.

The import part is done on a refs/mediawiki/<remote> branch before
coming to refs/remote/origin/master (Huge thanks to Jonathan Nieder
for his help)

We use UTF-8 everywhere: use encoding 'utf8'; does most of the job, but
we also read the output of Git commands in UTF-8 with the small helper
run_git, and write to the console (STDERR) in UTF-8. This allows a
seamless use of non-ascii characters in page titles, but hasn't been
tested on non-UTF-8 systems. In particular, UTF-8 encoding for filenames
could raise problems if different file systems handle UTF-8 filenames
differently. A uri_escape of mediawiki filenames could be imaginable, and
is still to be discussed further.

Partial cloning is supported using one of:

git clone -c remote.origin.pages='A_Page  Another_Page' mediawiki::http://wikiurl

git clone -c remote.origin.categories='Some_Category' mediawiki::http://wikiurl

git clone -c remote.origin.shallow='True' mediawiki::http://wikiurl

Thanks to notes metadata, it is possible to compare remote and local last
mediawiki revision to warn non-fast forward pushes and "everything
up-to-date" case.

When allowed, push looks for each commit between remotes/origin/master
and HEAD, catches every blob related to these commit and push them in
chronological order. To do so, it uses git rev-list --children HEAD and
travels the tree from remotes/origin/master to HEAD through children. In
other words :

	* Shortest path from remotes/origin/master to HEAD
	* For each commit encountered, push blobs related to this commit

Signed-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
Signed-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
Signed-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>
Signed-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>
Signed-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
Signed-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
Oops, I had an uncommited fix when I sent the previous patch. Change
since v4 is just this:

-       if ($fetch_from == 1) {
-               if ($n != 0) {
-                       print STDOUT "reset $ref\n";
-                       print STDOUT "from :$n\n";
-               } else {
-                       print STDERR "You appear to have cloned an empty mediawiki\n";
-                       # Something has to be done remote-helper side. If nothing is done, an error is
-                       # thrown saying that HEAD is refering to unknown object 0000000000000000000
-               }
+       if ($fetch_from == 1 && $n == 0) {
+               print STDERR "You appear to have cloned an empty MediaWiki.\n";
+               # Something has to be done remote-helper side. If nothing is done, an error is
+               # thrown saying that HEAD is refering to unknown object 0000000000000000000
+               # and the clone fails.

Sorry for the noise.

 contrib/mw-to-git/git-remote-mediawiki     |  718 ++++++++++++++++++++++++++++
 contrib/mw-to-git/git-remote-mediawiki.txt |    7 +
 2 files changed, 725 insertions(+), 0 deletions(-)
 create mode 100755 contrib/mw-to-git/git-remote-mediawiki
 create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
new file mode 100755
index 0000000..e24a640
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -0,0 +1,718 @@
+#! /usr/bin/perl
+
+# Copyright (C) 2011
+#     Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
+#     Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
+#     Claire Fousse <claire.fousse@ensimag.imag.fr>
+#     David Amouyal <david.amouyal@ensimag.imag.fr>
+#     Matthieu Moy <matthieu.moy@grenoble-inp.fr>
+# License: GPL v2 or later
+
+# Gateway between Git and MediaWiki.
+#   https://github.com/Bibzball/Git-Mediawiki/wiki
+#
+# Known limitations:
+#
+# - Only wiki pages are managed, no support for [[File:...]]
+#   attachments.
+#
+# - Poor performance in the best case: it takes forever to check
+#   whether we're up-to-date (on fetch or push) or to fetch a few
+#   revisions from a large wiki, because we use exclusively a
+#   page-based synchronization. We could switch to a wiki-wide
+#   synchronization when the synchronization involves few revisions
+#   but the wiki is large.
+#
+# - Git renames could be turned into MediaWiki renames (see TODO
+#   below)
+#
+# - login/password support requires the user to write the password
+#   cleartext in a file (see TODO below).
+#
+# - No way to import "one page, and all pages included in it"
+#
+# - Multiple remote MediaWikis have not been very well tested.
+
+use strict;
+use MediaWiki::API;
+use DateTime::Format::ISO8601;
+use encoding 'utf8';
+
+# use encoding 'utf8' doesn't change STDERROR
+# but we're going to output UTF-8 filenames to STDERR
+binmode STDERR, ":utf8";
+
+use URI::Escape;
+use warnings;
+
+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced
+use constant SLASH_REPLACEMENT => "%2F";
+
+# It's not always possible to delete pages (may require some
+# priviledges). Deleted pages are replaced with this content.
+use constant DELETED_CONTENT => "[[Category:Deleted]]\n";
+
+# It's not possible to create empty pages. New empty files in Git are
+# sent with this content instead.
+use constant EMPTY_CONTENT => "<!-- empty page -->\n";
+
+# used to reflect file creation or deletion in diff.
+use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
+
+my $remotename = $ARGV[0];
+my $url = $ARGV[1];
+
+# Accept both space-separated and multiple keys in config file.
+# Spaces should be written as _ anyway because we'll use chomp.
+my @tracked_pages = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".pages"));
+chomp(@tracked_pages);
+
+# Just like @tracked_pages, but for MediaWiki categories.
+my @tracked_categories = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".categories"));
+chomp(@tracked_categories);
+
+my $wiki_login = run_git("config --get remote.". $remotename .".mwLogin");
+# TODO: ideally, this should be able to read from keyboard, but we're
+# inside a remote helper, so our stdin is connect to git, not to a
+# terminal.
+my $wiki_passwd = run_git("config --get remote.". $remotename .".mwPassword");
+chomp ($wiki_login);
+chomp ($wiki_passwd);
+
+# Import only last revisions (both for clone and fetch)
+my $shallow_import = run_git("config --get --bool remote.". $remotename .".shallow");
+chomp($shallow_import);
+$shallow_import = ($shallow_import eq "true");
+
+my $wiki_name = $url;
+$wiki_name =~ s/[^\/]*:\/\///;
+
+my $import_started;
+
+# Commands parser
+my $entry;
+my @cmd;
+while (<STDIN>) {
+	chomp;
+	@cmd = split(/ /);
+	if (defined($cmd[0])) {
+		# Line not blank
+		if ($cmd[0] eq "capabilities") {
+			die("Too many arguments for capabilities") unless (!defined($cmd[1]));
+			mw_capabilities();
+		} elsif ($cmd[0] eq "list") {
+			die("Too many arguments for list") unless (!defined($cmd[2]));
+			mw_list($cmd[1]);
+		} elsif ($cmd[0] eq "import") {
+			die("Invalid arguments for import") unless ($cmd[1] ne "" && !defined($cmd[2]));
+			mw_import($cmd[1]);
+		} elsif ($cmd[0] eq "option") {
+			die("Too many arguments for option") unless ($cmd[1] ne "" && $cmd[2] ne "" && !defined($cmd[3]));
+			mw_option($cmd[1],$cmd[2]);
+		} elsif ($cmd[0] eq "push") {
+			# Check the pattern <src>:<dst>
+			my @pushargs = split(/:/,$cmd[1]);
+			die("Invalid arguments for push") unless ($pushargs[1] ne "" && !defined($pushargs[2]));
+			mw_push($pushargs[0],$pushargs[1]);
+		} else {
+			print STDERR "Unknown command. Aborting...\n";
+			last;
+		}
+	} else {
+		# blank line: we should terminate
+		last;
+	}
+
+	BEGIN { $| = 1 } # flush STDOUT, to make sure the previous
+			 # command is fully processed.
+}
+# End of input
+if ($import_started) {
+	# Terminate the fast-import stream properly.
+	# Git requires one "done" command, and only
+	# one This is OK since we only have one
+	# branch, so import will be called only once
+	# (plus once for HEAD, for which we won't
+	# reach this point).
+	print STDOUT "done\n";
+}
+BEGIN { $| = 1 };
+if (!eof(STDIN)) {
+	# Wait for Git to terminate. If we don't, git fetch
+	# (transport-helper.c's sendline function) will try to write
+	# to our stdin, which will be closed, and git fetch will be
+	# killed. That's probably a bug in transport-helper.c, but in
+	# the meantime ...
+	sleep .1;
+};
+
+########################## Functions ##############################
+
+# MediaWiki API instance, created lazily.
+my $mediawiki;
+
+sub mw_connect_maybe {
+	if ($mediawiki) {
+	    return;
+	}
+	$mediawiki = MediaWiki::API->new;
+	$mediawiki->{config}->{api_url} = "$url/api.php";
+	if ($wiki_login) {
+		if (!$mediawiki->login({
+			lgname => $wiki_login,
+			lgpassword => $wiki_passwd,
+		})) {
+			print STDERR "Failed to log in mediawiki user \"$wiki_login\" on $url\n";
+			print STDERR "(error " .
+			    $mediawiki->{error}->{code} . ': ' .
+			    $mediawiki->{error}->{details} . ")\n";
+			exit 1;
+		} else {
+			print STDERR "Logged in with user \"$wiki_login\".\n";
+		}
+	}
+}
+
+sub get_mw_first_pages {
+	my $some_pages = shift;
+	my @some_pages = @{$some_pages};
+
+	my $pages = shift;
+
+	# pattern 'page1|page2|...' required by the API
+	my $titles = join('|', @some_pages);
+
+	my $mw_pages = $mediawiki->api({
+		action => 'query',
+		titles => $titles,
+	});
+	if (!defined($mw_pages)) {
+		print STDERR "fatal: could not query the list of wiki pages.\n";
+		print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+		print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+		exit 1;
+	}
+	while (my ($id, $page) = each (%{$mw_pages->{query}->{pages}})) {
+		if ($id < 0) {
+			print STDERR "Warning: page $page->{title} not found on wiki\n";
+		} else {
+			$pages->{$page->{title}} = $page;
+		}
+	}
+}
+
+sub get_mw_pages {
+	mw_connect_maybe();
+
+	my %pages; # hash on page titles to avoid duplicates
+	my $user_defined;
+	if (@tracked_pages) {
+		$user_defined = 1;
+		# The user provided a list of pages titles, but we
+		# still need to query the API to get the page IDs.
+
+		my @some_pages = @tracked_pages;
+		while (@some_pages) {
+			my $last = 50;
+			if ($#some_pages < $last) {
+				$last = $#some_pages;
+			}
+			my @slice = @some_pages[0..$last];
+			get_mw_first_pages(\@slice, \%pages);
+			@some_pages = @some_pages[51..$#some_pages];
+		}
+	}
+	if (@tracked_categories) {
+		$user_defined = 1;
+		foreach my $category (@tracked_categories) {
+			if (index($category, ':') < 0) {
+				# Mediawiki requires the Category
+				# prefix, but let's not force the user
+				# to specify it.
+				$category = "Category:" . $category;
+			}
+			my $mw_pages = $mediawiki->list ( {
+				action => 'query',
+				list => 'categorymembers',
+				cmtitle => $category,
+				cmlimit => 'max' } )
+			    || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};
+			foreach my $page (@{$mw_pages}) {
+				$pages{$page->{title}} = $page;
+			}
+		}
+	}
+	if (!$user_defined) {
+		# No user-provided list, get the list of pages from
+		# the API.
+		my $mw_pages = $mediawiki->list({
+			action => 'query',
+			list => 'allpages',
+			aplimit => 500,
+		});
+		if (!defined($mw_pages)) {
+			print STDERR "fatal: could not get the list of wiki pages.\n";
+			print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+			print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+			exit 1;
+		}
+		foreach my $page (@{$mw_pages}) {
+			$pages{$page->{title}} = $page;
+		}
+	}
+	return values(%pages);
+}
+
+sub run_git {
+	open(my $git, "-|:encoding(UTF-8)", "git " . $_[0]);
+	my $res = do { local $/; <$git> };
+	close($git);
+
+	return $res;
+}
+
+
+sub get_last_local_revision {
+	# Get note regarding last mediawiki revision
+	my $note = run_git("notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null");
+	my @note_info = split(/ /, $note);
+
+	my $lastrevision_number;
+	if (!(defined($note_info[0]) && $note_info[0] eq "mediawiki_revision:")) {
+		print STDERR "No previous mediawiki revision found";
+		$lastrevision_number = 0;
+	} else {
+		# Notes are formatted : mediawiki_revision: #number
+		$lastrevision_number = $note_info[1];
+		chomp($lastrevision_number);
+		print STDERR "Last local mediawiki revision found is $lastrevision_number";
+	}
+	return $lastrevision_number;
+}
+
+sub get_last_remote_revision {
+	mw_connect_maybe();
+
+	my @pages = get_mw_pages();
+
+	my $max_rev_num = 0;
+
+	foreach my $page (@pages) {
+		my $id = $page->{pageid};
+
+		my $query = {
+			action => 'query',
+			prop => 'revisions',
+			rvprop => 'ids',
+			pageids => $id,
+		};
+
+		my $result = $mediawiki->api($query);
+
+		my $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});
+
+		$max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);
+	}
+
+	print STDERR "Last remote revision found is $max_rev_num.\n";
+	return $max_rev_num;
+}
+
+# Clean content before sending it to MediaWiki
+sub mediawiki_clean {
+	my $string = shift;
+	my $page_created = shift;
+	# Mediawiki does not allow blank space at the end of a page and ends with a single \n.
+	# This function right trims a string and adds a \n at the end to follow this rule
+	$string =~ s/\s+$//;
+	if ($string eq "" && $page_created) {
+		# Creating empty pages is forbidden.
+		$string = EMPTY_CONTENT;
+	}
+	return $string."\n";
+}
+
+# Filter applied on MediaWiki data before adding them to Git
+sub mediawiki_smudge {
+	my $string = shift;
+	if ($string eq EMPTY_CONTENT) {
+		$string = "";
+	}
+	# This \n is important. This is due to mediawiki's way to handle end of files.
+	return $string."\n";
+}
+
+sub mediawiki_clean_filename {
+	my $filename = shift;
+	$filename =~ s/@{[SLASH_REPLACEMENT]}/\//g;
+	# [, ], |, {, and } are forbidden by MediaWiki, even URL-encoded.
+	# Do a variant of URL-encoding, i.e. looks like URL-encoding,
+	# but with _ added to prevent MediaWiki from thinking this is
+	# an actual special character.
+	$filename =~ s/[\[\]\{\}\|]/sprintf("_%%_%x", ord($&))/ge;
+	# If we use the uri escape before
+	# we should unescape here, before anything
+
+	return $filename;
+}
+
+sub mediawiki_smudge_filename {
+	my $filename = shift;
+	$filename =~ s/\//@{[SLASH_REPLACEMENT]}/g;
+	$filename =~ s/ /_/g;
+	# Decode forbidden characters encoded in mediawiki_clean_filename
+	$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf("%c", hex($1))/ge;
+	return $filename;
+}
+
+sub literal_data {
+	my ($content) = @_;
+	print STDOUT "data ", bytes::length($content), "\n", $content;
+}
+
+sub mw_capabilities {
+	# Revisions are imported to the private namespace
+	# refs/mediawiki/$remotename/ by the helper and fetched into
+	# refs/remotes/$remotename later by fetch.
+	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
+	print STDOUT "import\n";
+	print STDOUT "list\n";
+	print STDOUT "push\n";
+	print STDOUT "\n";
+}
+
+sub mw_list {
+	# MediaWiki do not have branches, we consider one branch arbitrarily
+	# called master, and HEAD pointing to it.
+	print STDOUT "? refs/heads/master\n";
+	print STDOUT "\@refs/heads/master HEAD\n";
+	print STDOUT "\n";
+}
+
+sub mw_option {
+	print STDERR "remote-helper command 'option $_[0]' not yet implemented\n";
+	print STDOUT "unsupported\n";
+}
+
+sub fetch_mw_revisions_for_page {
+	my $page = shift;
+	my $id = shift;
+	my $fetch_from = shift;
+	my @page_revs = ();
+	my $query = {
+		action => 'query',
+		prop => 'revisions',
+		rvprop => 'ids',
+		rvdir => 'newer',
+		rvstartid => $fetch_from,
+		rvlimit => 500,
+		pageids => $id,
+	};
+
+	my $revnum = 0;
+	# Get 500 revisions at a time due to the mediawiki api limit
+	while (1) {
+		my $result = $mediawiki->api($query);
+
+		# Parse each of those 500 revisions
+		foreach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {
+			my $page_rev_ids;
+			$page_rev_ids->{pageid} = $page->{pageid};
+			$page_rev_ids->{revid} = $revision->{revid};
+			push (@page_revs, $page_rev_ids);
+			$revnum++;
+		}
+		last unless $result->{'query-continue'};
+		$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};
+	}
+	if ($shallow_import && @page_revs) {
+		print STDERR "  Found 1 revision (shallow import).\n";
+		@page_revs = sort {$b->{revid} <=> $a->{revid}} (@page_revs);
+		return $page_revs[0];
+	}
+	print STDERR "  Found ", $revnum, " revision(s).\n";
+	return @page_revs;
+}
+
+sub fetch_mw_revisions {
+	my $pages = shift; my @pages = @{$pages};
+	my $fetch_from = shift;
+
+	my @revisions = ();
+	my $n = 1;
+	foreach my $page (@pages) {
+		my $id = $page->{pageid};
+
+		print STDERR "page $n/", scalar(@pages), ": ". $page->{title} ."\n";
+		$n++;
+		my @page_revs = fetch_mw_revisions_for_page($page, $id, $fetch_from);
+		@revisions = (@page_revs, @revisions);
+	}
+
+	return ($n, @revisions);
+}
+
+sub import_file_revision {
+	my $commit = shift;
+	my %commit = %{$commit};
+	my $full_import = shift;
+	my $n = shift;
+
+	my $title = $commit{title};
+	my $comment = $commit{comment};
+	my $content = $commit{content};
+	my $author = $commit{author};
+	my $date = $commit{date};
+
+	print STDOUT "commit refs/mediawiki/$remotename/master\n";
+	print STDOUT "mark :$n\n";
+	print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+	literal_data($comment);
+
+	# If it's not a clone, we need to know where to start from
+	if (!$full_import && $n == 1) {
+		print STDOUT "from refs/mediawiki/$remotename/master^0\n";
+	}
+	if ($content ne DELETED_CONTENT) {
+		print STDOUT "M 644 inline $title.mw\n";
+		literal_data($content);
+		print STDOUT "\n\n";
+	} else {
+		print STDOUT "D $title.mw\n";
+	}
+
+	# mediawiki revision number in the git note
+	if ($full_import && $n == 1) {
+		print STDOUT "reset refs/notes/$remotename/mediawiki\n";
+	}
+	print STDOUT "commit refs/notes/$remotename/mediawiki\n";
+	print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+	literal_data("note added by git-mediawiki");
+	if (!$full_import && $n == 1) {
+		print STDOUT "from refs/notes/$remotename/mediawiki^0\n";
+	}
+	print STDOUT "N inline :$n\n";
+	literal_data("mediawiki_revision: " . $commit{mw_revision});
+	print STDOUT "\n\n";
+}
+
+sub mw_import {
+	$import_started = 1;
+	my $ref = shift;
+	# the remote helper will call "import HEAD" and
+	# "import refs/heads/master"
+	# Since HEAD is a symbolic ref to master (by convention,
+	# followed by the output of the command "list" that we gave),
+	# we don't need to do anything in this case.
+	if ($ref eq "HEAD") {
+		return;
+	}
+
+	mw_connect_maybe();
+
+	my @pages = get_mw_pages();
+
+	print STDERR "Searching revisions...\n";
+	my $last_local = get_last_local_revision();
+	my $fetch_from = $last_local + 1;
+	if ($fetch_from == 1) {
+		print STDERR ", fetching from beginning.\n";
+	} else {
+		print STDERR ", fetching from here.\n";
+	}
+	my ($n, @revisions) = fetch_mw_revisions(\@pages, $fetch_from);
+
+	# Creation of the fast-import stream
+	print STDERR "Fetching & writing export data...\n";
+
+	$n = 0;
+	my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined
+
+	foreach my $pagerevid (sort {$a->{revid} <=> $b->{revid}} @revisions) {
+		# fetch the content of the pages
+		my $query = {
+			action => 'query',
+			prop => 'revisions',
+			rvprop => 'content|timestamp|comment|user|ids',
+			revids => $pagerevid->{revid},
+		};
+
+		my $result = $mediawiki->api($query);
+
+		my $rev = pop(@{$result->{query}->{pages}->{$pagerevid->{pageid}}->{revisions}});
+
+		$n++;
+
+		my %commit;
+		$commit{author} = $rev->{user} || 'Anonymous';
+		$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
+		$commit{title} = mediawiki_smudge_filename(
+			$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}
+		    );
+		$commit{mw_revision} = $pagerevid->{revid};
+		$commit{content} = mediawiki_smudge($rev->{'*'});
+
+		if (!defined($rev->{timestamp})) {
+			$last_timestamp++;
+		} else {
+			$last_timestamp = $rev->{timestamp};
+		}
+		$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);
+
+		print STDERR "$n/", scalar(@revisions), ": Revision #$pagerevid->{revid} of $commit{title}\n";
+
+		import_file_revision(\%commit, ($fetch_from == 1), $n);
+	}
+
+	if ($fetch_from == 1 && $n == 0) {
+		print STDERR "You appear to have cloned an empty MediaWiki.\n";
+		# Something has to be done remote-helper side. If nothing is done, an error is
+		# thrown saying that HEAD is refering to unknown object 0000000000000000000
+		# and the clone fails.
+	}
+}
+
+sub error_non_fast_forward {
+	# Native git-push would show this after the summary.
+	# We can't ask it to display it cleanly, so print it
+	# ourselves before.
+	print STDERR "To prevent you from losing history, non-fast-forward updates were rejected\n";
+	print STDERR "Merge the remote changes (e.g. 'git pull') before pushing again. See the\n";
+	print STDERR "'Note about fast-forwards' section of 'git push --help' for details.\n";
+
+	print STDOUT "error $_[0] \"non-fast-forward\"\n";
+	print STDOUT "\n";
+}
+
+sub mw_push_file {
+	my $diff_info = shift;
+	# $diff_info contains a string in this format:
+	# 100644 100644 <sha1_of_blob_before_commit> <sha1_of_blob_now> <status>
+	my @diff_info_split = split(/[ \t]/, $diff_info);
+
+	# Filename, including .mw extension
+	my $complete_file_name = shift;
+	# Commit message
+	my $summary = shift;
+
+	my $new_sha1 = $diff_info_split[3];
+	my $old_sha1 = $diff_info_split[2];
+	my $page_created = ($old_sha1 eq NULL_SHA1);
+	my $page_deleted = ($new_sha1 eq NULL_SHA1);
+	$complete_file_name = mediawiki_clean_filename($complete_file_name);
+
+	if (substr($complete_file_name,-3) eq ".mw"){
+		my $title = substr($complete_file_name,0,-3);
+
+		my $file_content;
+		if ($page_deleted) {
+			# Deleting a page usually requires
+			# special priviledges. A common
+			# convention is to replace the page
+			# with this content instead:
+			$file_content = DELETED_CONTENT;
+		} else {
+			$file_content = run_git("cat-file -p $new_sha1");
+		}
+
+		mw_connect_maybe();
+
+		my $result = $mediawiki->edit( {
+			action => 'edit',
+			summary => $summary,
+			title => $title,
+			text => mediawiki_clean($file_content, $page_created),
+				  }, {
+					  skip_encoding => 1 # Helps with names with accentuated characters
+				  }) || die 'Fatal: Error ' .
+				  $mediawiki->{error}->{code} .
+				  ' from mediwiki: ' . $mediawiki->{error}->{details};
+		print STDERR "Pushed file : $new_sha1 - $title\n";
+	} else {
+		print STDERR "$complete_file_name not a mediawiki file (Not pushable on this version).\n"
+	}
+}
+
+sub mw_push {
+	my $last_local_revid = get_last_local_revision();
+	print STDERR ".\n"; # Finish sentence started by get_last_local_revision()
+	my $last_remote_revid = get_last_remote_revision();
+
+	# Get sha1 of commit pointed by local HEAD
+	my $HEAD_sha1 = run_git("rev-parse $_[0] 2>/dev/null"); chomp($HEAD_sha1);
+	# Get sha1 of commit pointed by remotes/$remotename/master
+	my $remoteorigin_sha1 = run_git("rev-parse refs/remotes/$remotename/master 2>/dev/null");
+	chomp($remoteorigin_sha1);
+
+	if ($last_local_revid > 0 &&
+	    $last_local_revid < $last_remote_revid){
+		return error_non_fast_forward($_[0]);
+	}
+
+	if ($HEAD_sha1 eq $remoteorigin_sha1) {
+		print STDOUT "\n";
+		return;
+	}
+
+	# Get every commit in between HEAD and refs/remotes/origin/master,
+	# including HEAD and refs/remotes/origin/master
+	my @commit_pairs = ();
+	if ($last_local_revid > 0) {
+		my $parsed_sha1 = $remoteorigin_sha1;
+		# Find a path from last MediaWiki commit to pushed commit
+		while ($parsed_sha1 ne $HEAD_sha1) {
+			my @commit_info =  grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $_[0]")));
+			if (!@commit_info) {
+				return error_non_fast_forward($_[0]);
+			}
+			my @commit_info_split = split(/ |\n/, $commit_info[0]);
+			# $commit_info_split[1] is the sha1 of the commit to export
+			# $commit_info_split[0] is the sha1 of its direct child
+			push (@commit_pairs, \@commit_info_split);
+			$parsed_sha1 = $commit_info_split[1];
+		}
+	} else {
+		# No remote mediawiki revision. Export the whole
+		# history (linearized with --first-parent)
+		print STDERR "Warning: no common ancestor, pushing complete history\n";
+		my $history = run_git("rev-list --first-parent --children $_[0]");
+		my @history = split('\n', $history);
+		@history = @history[1..$#history];
+		foreach my $line (reverse @history) {
+			my @commit_info_split = split(/ |\n/, $line);
+			push (@commit_pairs, \@commit_info_split);
+		}
+	}
+
+	foreach my $commit_info_split (@commit_pairs) {
+		my $sha1_child = @{$commit_info_split}[0];
+		my $sha1_commit = @{$commit_info_split}[1];
+		my $diff_infos = run_git("diff-tree -r --raw -z $sha1_child $sha1_commit");
+		# TODO: we could detect rename, and encode them with a #redirect on the wiki.
+		# TODO: for now, it's just a delete+add
+		my @diff_info_list = split(/\0/, $diff_infos);
+		# Keep the first line of the commit message as mediawiki comment for the revision
+		my $commit_msg = (split(/\n/, run_git("show --pretty=format:\"%s\" $sha1_commit")))[0];
+		chomp($commit_msg);
+		# Push every blob
+		while (@diff_info_list) {
+			# git diff-tree -z gives an output like
+			# <metadata>\0<filename1>\0
+			# <metadata>\0<filename2>\0
+			# and we've split on \0.
+			my $info = shift(@diff_info_list);
+			my $file = shift(@diff_info_list);
+			mw_push_file($info, $file, $commit_msg);
+		}
+	}
+
+	print STDOUT "ok $_[1]\n";
+	print STDOUT "\n";
+
+	print STDERR "Just pushed some revisions to MediaWiki.\n";
+	print STDERR "The pushed revisions now have to be re-imported, and your current branch\n";
+	print STDERR "needs to be updated with these re-imported commits. You can do this with\n";
+	print STDERR "\n";
+	print STDERR "  git pull --rebase\n";
+	print STDERR "\n";
+}
diff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt
new file mode 100644
index 0000000..4d211f5
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki.txt
@@ -0,0 +1,7 @@
+Git-Mediawiki is a project which aims the creation of a gate
+between git and mediawiki, allowing git users to push and pull
+objects from mediawiki just as one would do with a classic git
+repository thanks to remote-helpers.
+
+For more information, visit the wiki at
+https://github.com/Bibzball/Git-Mediawiki/wiki
-- 
1.7.6.585.g5929f.dirty

[PATCH v6] Add a remote helper to interact with mediawiki (fetch & push)

[Matthieu Moy]

From: Jeremie Nikaes <jeremie.nikaes@ensimag.imag.fr>

Implement a gate between git and mediawiki, allowing git users to push
and pull objects from mediawiki just as one would do with a classic git
repository thanks to remote-helpers.

The following packages need to be installed (available on common
repositories):

     libmediawiki-api-perl
     libdatetime-format-iso8601-perl

Use remote helpers in order to be as transparent as possible to the git
user.

Download Mediawiki revisions through the Mediawiki API and then
fast-import into git.

Mediawiki revision number and git commits are linked thanks to notes
bound to commits.

The import part is done on a refs/mediawiki/<remote> branch before
coming to refs/remote/origin/master (Huge thanks to Jonathan Nieder
for his help)

We use UTF-8 everywhere: use encoding 'utf8'; does most of the job, but
we also read the output of Git commands in UTF-8 with the small helper
run_git, and write to the console (STDERR) in UTF-8. This allows a
seamless use of non-ascii characters in page titles, but hasn't been
tested on non-UTF-8 systems. In particular, UTF-8 encoding for filenames
could raise problems if different file systems handle UTF-8 filenames
differently. A uri_escape of mediawiki filenames could be imaginable, and
is still to be discussed further.

Partial cloning is supported using one of:

git clone -c remote.origin.pages='A_Page  Another_Page' mediawiki::http://wikiurl

git clone -c remote.origin.categories='Some_Category' mediawiki::http://wikiurl

git clone -c remote.origin.shallow='True' mediawiki::http://wikiurl

Thanks to notes metadata, it is possible to compare remote and local last
mediawiki revision to warn non-fast forward pushes and "everything
up-to-date" case.

When allowed, push looks for each commit between remotes/origin/master
and HEAD, catches every blob related to these commit and push them in
chronological order. To do so, it uses git rev-list --children HEAD and
travels the tree from remotes/origin/master to HEAD through children. In
other words :

	* Shortest path from remotes/origin/master to HEAD
	* For each commit encountered, push blobs related to this commit

Signed-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
Signed-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
Signed-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>
Signed-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>
Signed-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
Signed-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
So, after understanding better how import works, here's an updated
patch that gets rid of the hacky workaround to terminate and send the
"done" command at the right time.

Actually, push had the same problem but it just went unnoticed (the
remote has just one branch, so it's silly to try to push multiple
branches at the same time ...). This version handles push more
cleanly, giving accurate error message in cases like

  git push origin :master
  git push origin foo bar master

or perhaps more commonly

  git push --all

in a repository with branches other than master.

There are still a few TODO, but I think all of them can wait.

 contrib/mw-to-git/git-remote-mediawiki     |  758 ++++++++++++++++++++++++++++
 contrib/mw-to-git/git-remote-mediawiki.txt |    7 +
 2 files changed, 765 insertions(+), 0 deletions(-)
 create mode 100755 contrib/mw-to-git/git-remote-mediawiki
 create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
new file mode 100755
index 0000000..51f01cd
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -0,0 +1,758 @@
+#! /usr/bin/perl
+
+# Copyright (C) 2011
+#     Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
+#     Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
+#     Claire Fousse <claire.fousse@ensimag.imag.fr>
+#     David Amouyal <david.amouyal@ensimag.imag.fr>
+#     Matthieu Moy <matthieu.moy@grenoble-inp.fr>
+# License: GPL v2 or later
+
+# Gateway between Git and MediaWiki.
+#   https://github.com/Bibzball/Git-Mediawiki/wiki
+#
+# Known limitations:
+#
+# - Only wiki pages are managed, no support for [[File:...]]
+#   attachments.
+#
+# - Poor performance in the best case: it takes forever to check
+#   whether we're up-to-date (on fetch or push) or to fetch a few
+#   revisions from a large wiki, because we use exclusively a
+#   page-based synchronization. We could switch to a wiki-wide
+#   synchronization when the synchronization involves few revisions
+#   but the wiki is large.
+#
+# - Git renames could be turned into MediaWiki renames (see TODO
+#   below)
+#
+# - login/password support requires the user to write the password
+#   cleartext in a file (see TODO below).
+#
+# - No way to import "one page, and all pages included in it"
+#
+# - Multiple remote MediaWikis have not been very well tested.
+
+use strict;
+use MediaWiki::API;
+use DateTime::Format::ISO8601;
+use encoding 'utf8';
+
+# use encoding 'utf8' doesn't change STDERROR
+# but we're going to output UTF-8 filenames to STDERR
+binmode STDERR, ":utf8";
+
+use URI::Escape;
+use warnings;
+
+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced
+use constant SLASH_REPLACEMENT => "%2F";
+
+# It's not always possible to delete pages (may require some
+# priviledges). Deleted pages are replaced with this content.
+use constant DELETED_CONTENT => "[[Category:Deleted]]\n";
+
+# It's not possible to create empty pages. New empty files in Git are
+# sent with this content instead.
+use constant EMPTY_CONTENT => "<!-- empty page -->\n";
+
+# used to reflect file creation or deletion in diff.
+use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
+
+my $remotename = $ARGV[0];
+my $url = $ARGV[1];
+
+# Accept both space-separated and multiple keys in config file.
+# Spaces should be written as _ anyway because we'll use chomp.
+my @tracked_pages = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".pages"));
+chomp(@tracked_pages);
+
+# Just like @tracked_pages, but for MediaWiki categories.
+my @tracked_categories = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".categories"));
+chomp(@tracked_categories);
+
+my $wiki_login = run_git("config --get remote.". $remotename .".mwLogin");
+# TODO: ideally, this should be able to read from keyboard, but we're
+# inside a remote helper, so our stdin is connect to git, not to a
+# terminal.
+my $wiki_passwd = run_git("config --get remote.". $remotename .".mwPassword");
+chomp ($wiki_login);
+chomp ($wiki_passwd);
+
+# Import only last revisions (both for clone and fetch)
+my $shallow_import = run_git("config --get --bool remote.". $remotename .".shallow");
+chomp($shallow_import);
+$shallow_import = ($shallow_import eq "true");
+
+my $wiki_name = $url;
+$wiki_name =~ s/[^\/]*:\/\///;
+
+# Commands parser
+my $entry;
+my @cmd;
+while (<STDIN>) {
+	chomp;
+	@cmd = split(/ /);
+	if (defined($cmd[0])) {
+		# Line not blank
+		if ($cmd[0] eq "capabilities") {
+			die("Too many arguments for capabilities") unless (!defined($cmd[1]));
+			mw_capabilities();
+		} elsif ($cmd[0] eq "list") {
+			die("Too many arguments for list") unless (!defined($cmd[2]));
+			mw_list($cmd[1]);
+		} elsif ($cmd[0] eq "import") {
+			die("Invalid arguments for import") unless ($cmd[1] ne "" && !defined($cmd[2]));
+			mw_import($cmd[1]);
+		} elsif ($cmd[0] eq "option") {
+			die("Too many arguments for option") unless ($cmd[1] ne "" && $cmd[2] ne "" && !defined($cmd[3]));
+			mw_option($cmd[1],$cmd[2]);
+		} elsif ($cmd[0] eq "push") {
+			mw_push($cmd[1]);
+		} else {
+			print STDERR "Unknown command. Aborting...\n";
+			last;
+		}
+	} else {
+		# blank line: we should terminate
+		last;
+	}
+
+	BEGIN { $| = 1 } # flush STDOUT, to make sure the previous
+			 # command is fully processed.
+}
+
+########################## Functions ##############################
+
+# MediaWiki API instance, created lazily.
+my $mediawiki;
+
+sub mw_connect_maybe {
+	if ($mediawiki) {
+	    return;
+	}
+	$mediawiki = MediaWiki::API->new;
+	$mediawiki->{config}->{api_url} = "$url/api.php";
+	if ($wiki_login) {
+		if (!$mediawiki->login({
+			lgname => $wiki_login,
+			lgpassword => $wiki_passwd,
+		})) {
+			print STDERR "Failed to log in mediawiki user \"$wiki_login\" on $url\n";
+			print STDERR "(error " .
+			    $mediawiki->{error}->{code} . ': ' .
+			    $mediawiki->{error}->{details} . ")\n";
+			exit 1;
+		} else {
+			print STDERR "Logged in with user \"$wiki_login\".\n";
+		}
+	}
+}
+
+sub get_mw_first_pages {
+	my $some_pages = shift;
+	my @some_pages = @{$some_pages};
+
+	my $pages = shift;
+
+	# pattern 'page1|page2|...' required by the API
+	my $titles = join('|', @some_pages);
+
+	my $mw_pages = $mediawiki->api({
+		action => 'query',
+		titles => $titles,
+	});
+	if (!defined($mw_pages)) {
+		print STDERR "fatal: could not query the list of wiki pages.\n";
+		print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+		print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+		exit 1;
+	}
+	while (my ($id, $page) = each (%{$mw_pages->{query}->{pages}})) {
+		if ($id < 0) {
+			print STDERR "Warning: page $page->{title} not found on wiki\n";
+		} else {
+			$pages->{$page->{title}} = $page;
+		}
+	}
+}
+
+sub get_mw_pages {
+	mw_connect_maybe();
+
+	my %pages; # hash on page titles to avoid duplicates
+	my $user_defined;
+	if (@tracked_pages) {
+		$user_defined = 1;
+		# The user provided a list of pages titles, but we
+		# still need to query the API to get the page IDs.
+
+		my @some_pages = @tracked_pages;
+		while (@some_pages) {
+			my $last = 50;
+			if ($#some_pages < $last) {
+				$last = $#some_pages;
+			}
+			my @slice = @some_pages[0..$last];
+			get_mw_first_pages(\@slice, \%pages);
+			@some_pages = @some_pages[51..$#some_pages];
+		}
+	}
+	if (@tracked_categories) {
+		$user_defined = 1;
+		foreach my $category (@tracked_categories) {
+			if (index($category, ':') < 0) {
+				# Mediawiki requires the Category
+				# prefix, but let's not force the user
+				# to specify it.
+				$category = "Category:" . $category;
+			}
+			my $mw_pages = $mediawiki->list ( {
+				action => 'query',
+				list => 'categorymembers',
+				cmtitle => $category,
+				cmlimit => 'max' } )
+			    || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};
+			foreach my $page (@{$mw_pages}) {
+				$pages{$page->{title}} = $page;
+			}
+		}
+	}
+	if (!$user_defined) {
+		# No user-provided list, get the list of pages from
+		# the API.
+		my $mw_pages = $mediawiki->list({
+			action => 'query',
+			list => 'allpages',
+			aplimit => 500,
+		});
+		if (!defined($mw_pages)) {
+			print STDERR "fatal: could not get the list of wiki pages.\n";
+			print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+			print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+			exit 1;
+		}
+		foreach my $page (@{$mw_pages}) {
+			$pages{$page->{title}} = $page;
+		}
+	}
+	return values(%pages);
+}
+
+sub run_git {
+	open(my $git, "-|:encoding(UTF-8)", "git " . $_[0]);
+	my $res = do { local $/; <$git> };
+	close($git);
+
+	return $res;
+}
+
+
+sub get_last_local_revision {
+	# Get note regarding last mediawiki revision
+	my $note = run_git("notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null");
+	my @note_info = split(/ /, $note);
+
+	my $lastrevision_number;
+	if (!(defined($note_info[0]) && $note_info[0] eq "mediawiki_revision:")) {
+		print STDERR "No previous mediawiki revision found";
+		$lastrevision_number = 0;
+	} else {
+		# Notes are formatted : mediawiki_revision: #number
+		$lastrevision_number = $note_info[1];
+		chomp($lastrevision_number);
+		print STDERR "Last local mediawiki revision found is $lastrevision_number";
+	}
+	return $lastrevision_number;
+}
+
+sub get_last_remote_revision {
+	mw_connect_maybe();
+
+	my @pages = get_mw_pages();
+
+	my $max_rev_num = 0;
+
+	foreach my $page (@pages) {
+		my $id = $page->{pageid};
+
+		my $query = {
+			action => 'query',
+			prop => 'revisions',
+			rvprop => 'ids',
+			pageids => $id,
+		};
+
+		my $result = $mediawiki->api($query);
+
+		my $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});
+
+		$max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);
+	}
+
+	print STDERR "Last remote revision found is $max_rev_num.\n";
+	return $max_rev_num;
+}
+
+# Clean content before sending it to MediaWiki
+sub mediawiki_clean {
+	my $string = shift;
+	my $page_created = shift;
+	# Mediawiki does not allow blank space at the end of a page and ends with a single \n.
+	# This function right trims a string and adds a \n at the end to follow this rule
+	$string =~ s/\s+$//;
+	if ($string eq "" && $page_created) {
+		# Creating empty pages is forbidden.
+		$string = EMPTY_CONTENT;
+	}
+	return $string."\n";
+}
+
+# Filter applied on MediaWiki data before adding them to Git
+sub mediawiki_smudge {
+	my $string = shift;
+	if ($string eq EMPTY_CONTENT) {
+		$string = "";
+	}
+	# This \n is important. This is due to mediawiki's way to handle end of files.
+	return $string."\n";
+}
+
+sub mediawiki_clean_filename {
+	my $filename = shift;
+	$filename =~ s/@{[SLASH_REPLACEMENT]}/\//g;
+	# [, ], |, {, and } are forbidden by MediaWiki, even URL-encoded.
+	# Do a variant of URL-encoding, i.e. looks like URL-encoding,
+	# but with _ added to prevent MediaWiki from thinking this is
+	# an actual special character.
+	$filename =~ s/[\[\]\{\}\|]/sprintf("_%%_%x", ord($&))/ge;
+	# If we use the uri escape before
+	# we should unescape here, before anything
+
+	return $filename;
+}
+
+sub mediawiki_smudge_filename {
+	my $filename = shift;
+	$filename =~ s/\//@{[SLASH_REPLACEMENT]}/g;
+	$filename =~ s/ /_/g;
+	# Decode forbidden characters encoded in mediawiki_clean_filename
+	$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf("%c", hex($1))/ge;
+	return $filename;
+}
+
+sub literal_data {
+	my ($content) = @_;
+	print STDOUT "data ", bytes::length($content), "\n", $content;
+}
+
+sub mw_capabilities {
+	# Revisions are imported to the private namespace
+	# refs/mediawiki/$remotename/ by the helper and fetched into
+	# refs/remotes/$remotename later by fetch.
+	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
+	print STDOUT "import\n";
+	print STDOUT "list\n";
+	print STDOUT "push\n";
+	print STDOUT "\n";
+}
+
+sub mw_list {
+	# MediaWiki do not have branches, we consider one branch arbitrarily
+	# called master, and HEAD pointing to it.
+	print STDOUT "? refs/heads/master\n";
+	print STDOUT "\@refs/heads/master HEAD\n";
+	print STDOUT "\n";
+}
+
+sub mw_option {
+	print STDERR "remote-helper command 'option $_[0]' not yet implemented\n";
+	print STDOUT "unsupported\n";
+}
+
+sub fetch_mw_revisions_for_page {
+	my $page = shift;
+	my $id = shift;
+	my $fetch_from = shift;
+	my @page_revs = ();
+	my $query = {
+		action => 'query',
+		prop => 'revisions',
+		rvprop => 'ids',
+		rvdir => 'newer',
+		rvstartid => $fetch_from,
+		rvlimit => 500,
+		pageids => $id,
+	};
+
+	my $revnum = 0;
+	# Get 500 revisions at a time due to the mediawiki api limit
+	while (1) {
+		my $result = $mediawiki->api($query);
+
+		# Parse each of those 500 revisions
+		foreach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {
+			my $page_rev_ids;
+			$page_rev_ids->{pageid} = $page->{pageid};
+			$page_rev_ids->{revid} = $revision->{revid};
+			push (@page_revs, $page_rev_ids);
+			$revnum++;
+		}
+		last unless $result->{'query-continue'};
+		$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};
+	}
+	if ($shallow_import && @page_revs) {
+		print STDERR "  Found 1 revision (shallow import).\n";
+		@page_revs = sort {$b->{revid} <=> $a->{revid}} (@page_revs);
+		return $page_revs[0];
+	}
+	print STDERR "  Found ", $revnum, " revision(s).\n";
+	return @page_revs;
+}
+
+sub fetch_mw_revisions {
+	my $pages = shift; my @pages = @{$pages};
+	my $fetch_from = shift;
+
+	my @revisions = ();
+	my $n = 1;
+	foreach my $page (@pages) {
+		my $id = $page->{pageid};
+
+		print STDERR "page $n/", scalar(@pages), ": ". $page->{title} ."\n";
+		$n++;
+		my @page_revs = fetch_mw_revisions_for_page($page, $id, $fetch_from);
+		@revisions = (@page_revs, @revisions);
+	}
+
+	return ($n, @revisions);
+}
+
+sub import_file_revision {
+	my $commit = shift;
+	my %commit = %{$commit};
+	my $full_import = shift;
+	my $n = shift;
+
+	my $title = $commit{title};
+	my $comment = $commit{comment};
+	my $content = $commit{content};
+	my $author = $commit{author};
+	my $date = $commit{date};
+
+	print STDOUT "commit refs/mediawiki/$remotename/master\n";
+	print STDOUT "mark :$n\n";
+	print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+	literal_data($comment);
+
+	# If it's not a clone, we need to know where to start from
+	if (!$full_import && $n == 1) {
+		print STDOUT "from refs/mediawiki/$remotename/master^0\n";
+	}
+	if ($content ne DELETED_CONTENT) {
+		print STDOUT "M 644 inline $title.mw\n";
+		literal_data($content);
+		print STDOUT "\n\n";
+	} else {
+		print STDOUT "D $title.mw\n";
+	}
+
+	# mediawiki revision number in the git note
+	if ($full_import && $n == 1) {
+		print STDOUT "reset refs/notes/$remotename/mediawiki\n";
+	}
+	print STDOUT "commit refs/notes/$remotename/mediawiki\n";
+	print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+	literal_data("note added by git-mediawiki");
+	if (!$full_import && $n == 1) {
+		print STDOUT "from refs/notes/$remotename/mediawiki^0\n";
+	}
+	print STDOUT "N inline :$n\n";
+	literal_data("mediawiki_revision: " . $commit{mw_revision});
+	print STDOUT "\n\n";
+}
+
+# parse a sequence of
+# <cmd> <arg1>
+# <cmd> <arg2>
+# \n
+# (like batch sequence of import and sequence of push statements)
+sub get_more_refs {
+	my $cmd = shift;
+	my @refs;
+	while (1) {
+		my $line = <STDIN>;
+		if ($line =~ m/^$cmd (.*)$/) {
+			push(@refs, $1);
+		} elsif ($line eq "\n") {
+			return @refs;
+		} else {
+			die("Invalid command in a '$cmd' batch: ". $_);
+		}
+	}
+}
+
+sub mw_import {
+	# multiple import commands can follow each other.
+	my @refs = (shift, get_more_refs("import"));
+	foreach my $ref (@refs) {
+		mw_import_ref($ref);
+	}
+	print STDOUT "done\n";
+}
+
+sub mw_import_ref {
+	my $ref = shift;
+	# The remote helper will call "import HEAD" and
+	# "import refs/heads/master".
+	# Since HEAD is a symbolic ref to master (by convention,
+	# followed by the output of the command "list" that we gave),
+	# we don't need to do anything in this case.
+	if ($ref eq "HEAD") {
+		return;
+	}
+
+	mw_connect_maybe();
+
+	my @pages = get_mw_pages();
+
+	print STDERR "Searching revisions...\n";
+	my $last_local = get_last_local_revision();
+	my $fetch_from = $last_local + 1;
+	if ($fetch_from == 1) {
+		print STDERR ", fetching from beginning.\n";
+	} else {
+		print STDERR ", fetching from here.\n";
+	}
+	my ($n, @revisions) = fetch_mw_revisions(\@pages, $fetch_from);
+
+	# Creation of the fast-import stream
+	print STDERR "Fetching & writing export data...\n";
+
+	$n = 0;
+	my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined
+
+	foreach my $pagerevid (sort {$a->{revid} <=> $b->{revid}} @revisions) {
+		# fetch the content of the pages
+		my $query = {
+			action => 'query',
+			prop => 'revisions',
+			rvprop => 'content|timestamp|comment|user|ids',
+			revids => $pagerevid->{revid},
+		};
+
+		my $result = $mediawiki->api($query);
+
+		my $rev = pop(@{$result->{query}->{pages}->{$pagerevid->{pageid}}->{revisions}});
+
+		$n++;
+
+		my %commit;
+		$commit{author} = $rev->{user} || 'Anonymous';
+		$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
+		$commit{title} = mediawiki_smudge_filename(
+			$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}
+		    );
+		$commit{mw_revision} = $pagerevid->{revid};
+		$commit{content} = mediawiki_smudge($rev->{'*'});
+
+		if (!defined($rev->{timestamp})) {
+			$last_timestamp++;
+		} else {
+			$last_timestamp = $rev->{timestamp};
+		}
+		$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);
+
+		print STDERR "$n/", scalar(@revisions), ": Revision #$pagerevid->{revid} of $commit{title}\n";
+
+		import_file_revision(\%commit, ($fetch_from == 1), $n);
+	}
+
+	if ($fetch_from == 1 && $n == 0) {
+		print STDERR "You appear to have cloned an empty MediaWiki.\n";
+		# Something has to be done remote-helper side. If nothing is done, an error is
+		# thrown saying that HEAD is refering to unknown object 0000000000000000000
+		# and the clone fails.
+	}
+}
+
+sub error_non_fast_forward {
+	# Native git-push would show this after the summary.
+	# We can't ask it to display it cleanly, so print it
+	# ourselves before.
+	print STDERR "To prevent you from losing history, non-fast-forward updates were rejected\n";
+	print STDERR "Merge the remote changes (e.g. 'git pull') before pushing again. See the\n";
+	print STDERR "'Note about fast-forwards' section of 'git push --help' for details.\n";
+
+	print STDOUT "error $_[0] \"non-fast-forward\"\n";
+	return 0;
+}
+
+sub mw_push_file {
+	my $diff_info = shift;
+	# $diff_info contains a string in this format:
+	# 100644 100644 <sha1_of_blob_before_commit> <sha1_of_blob_now> <status>
+	my @diff_info_split = split(/[ \t]/, $diff_info);
+
+	# Filename, including .mw extension
+	my $complete_file_name = shift;
+	# Commit message
+	my $summary = shift;
+
+	my $new_sha1 = $diff_info_split[3];
+	my $old_sha1 = $diff_info_split[2];
+	my $page_created = ($old_sha1 eq NULL_SHA1);
+	my $page_deleted = ($new_sha1 eq NULL_SHA1);
+	$complete_file_name = mediawiki_clean_filename($complete_file_name);
+
+	if (substr($complete_file_name,-3) eq ".mw"){
+		my $title = substr($complete_file_name,0,-3);
+
+		my $file_content;
+		if ($page_deleted) {
+			# Deleting a page usually requires
+			# special priviledges. A common
+			# convention is to replace the page
+			# with this content instead:
+			$file_content = DELETED_CONTENT;
+		} else {
+			$file_content = run_git("cat-file blob $new_sha1");
+		}
+
+		mw_connect_maybe();
+
+		my $result = $mediawiki->edit( {
+			action => 'edit',
+			summary => $summary,
+			title => $title,
+			text => mediawiki_clean($file_content, $page_created),
+				  }, {
+					  skip_encoding => 1 # Helps with names with accentuated characters
+				  }) || die 'Fatal: Error ' .
+				  $mediawiki->{error}->{code} .
+				  ' from mediwiki: ' . $mediawiki->{error}->{details};
+		print STDERR "Pushed file : $new_sha1 - $title\n";
+	} else {
+		print STDERR "$complete_file_name not a mediawiki file (Not pushable on this version of git-remote-mediawiki).\n"
+	}
+}
+
+sub mw_push {
+	# multiple push statements can follow each other
+	my @refsspecs = (shift, get_more_refs("push"));
+	my %status;
+	my $pushed;
+	for my $refspec (@refsspecs) {
+		unless ($refspec =~ m/^(\+?)([^:]*):([^:]*)$/) {
+			die("Invalid refspec for push. Expected <src>:<dst> or +<src>:<dst>");
+		}
+		my ($force, $local, $remote) = ($1 eq "+", $2, $3);
+		if ($force) {
+			print STDERR "Warning: forced push not allowed on a MediaWiki.\n";
+		}
+		if ($local eq "") {
+			print STDERR "Cannot delete remote branch on a MediaWiki\n";
+			print STDOUT "error $remote cannot delete\n";
+			next;
+		}
+		if ($remote ne "refs/heads/master") {
+			print STDERR "Only push to the branch 'master' is supported on a MediaWiki\n";
+			print STDOUT "error $remote only master allowed\n";
+			next;
+		}
+		if (mw_push_revision($local, $remote)) {
+			$pushed = 1;
+		}
+	}
+
+	# Notify Git that the push is done
+	print STDOUT "\n";
+
+	if ($pushed) {
+		print STDERR "Just pushed some revisions to MediaWiki.\n";
+		print STDERR "The pushed revisions now have to be re-imported, and your current branch\n";
+		print STDERR "needs to be updated with these re-imported commits. You can do this with\n";
+		print STDERR "\n";
+		print STDERR "  git pull --rebase\n";
+		print STDERR "\n";
+	}
+}
+
+sub mw_push_revision {
+	my $local = shift;
+	my $remote = shift; # actually, this has to be "refs/heads/master" at this point.
+	my $last_local_revid = get_last_local_revision();
+	print STDERR ".\n"; # Finish sentence started by get_last_local_revision()
+	my $last_remote_revid = get_last_remote_revision();
+
+	# Get sha1 of commit pointed by local HEAD
+	my $HEAD_sha1 = run_git("rev-parse $local 2>/dev/null"); chomp($HEAD_sha1);
+	# Get sha1 of commit pointed by remotes/$remotename/master
+	my $remoteorigin_sha1 = run_git("rev-parse refs/remotes/$remotename/master 2>/dev/null");
+	chomp($remoteorigin_sha1);
+
+	if ($last_local_revid > 0 &&
+	    $last_local_revid < $last_remote_revid){
+		return error_non_fast_forward($remote);
+	}
+
+	if ($HEAD_sha1 eq $remoteorigin_sha1) {
+		# nothing to push
+		return 0;
+	}
+
+	# Get every commit in between HEAD and refs/remotes/origin/master,
+	# including HEAD and refs/remotes/origin/master
+	my @commit_pairs = ();
+	if ($last_local_revid > 0) {
+		my $parsed_sha1 = $remoteorigin_sha1;
+		# Find a path from last MediaWiki commit to pushed commit
+		while ($parsed_sha1 ne $HEAD_sha1) {
+			my @commit_info =  grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $local")));
+			if (!@commit_info) {
+				return error_non_fast_forward($remote);
+			}
+			my @commit_info_split = split(/ |\n/, $commit_info[0]);
+			# $commit_info_split[1] is the sha1 of the commit to export
+			# $commit_info_split[0] is the sha1 of its direct child
+			push (@commit_pairs, \@commit_info_split);
+			$parsed_sha1 = $commit_info_split[1];
+		}
+	} else {
+		# No remote mediawiki revision. Export the whole
+		# history (linearized with --first-parent)
+		print STDERR "Warning: no common ancestor, pushing complete history\n";
+		my $history = run_git("rev-list --first-parent --children $local");
+		my @history = split('\n', $history);
+		@history = @history[1..$#history];
+		foreach my $line (reverse @history) {
+			my @commit_info_split = split(/ |\n/, $line);
+			push (@commit_pairs, \@commit_info_split);
+		}
+	}
+
+	foreach my $commit_info_split (@commit_pairs) {
+		my $sha1_child = @{$commit_info_split}[0];
+		my $sha1_commit = @{$commit_info_split}[1];
+		my $diff_infos = run_git("diff-tree -r --raw -z $sha1_child $sha1_commit");
+		# TODO: we could detect rename, and encode them with a #redirect on the wiki.
+		# TODO: for now, it's just a delete+add
+		my @diff_info_list = split(/\0/, $diff_infos);
+		# Keep the first line of the commit message as mediawiki comment for the revision
+		my $commit_msg = (split(/\n/, run_git("show --pretty=format:\"%s\" $sha1_commit")))[0];
+		chomp($commit_msg);
+		# Push every blob
+		while (@diff_info_list) {
+			# git diff-tree -z gives an output like
+			# <metadata>\0<filename1>\0
+			# <metadata>\0<filename2>\0
+			# and we've split on \0.
+			my $info = shift(@diff_info_list);
+			my $file = shift(@diff_info_list);
+			mw_push_file($info, $file, $commit_msg);
+		}
+	}
+
+	print STDOUT "ok $remote\n";
+	return 1;
+}
diff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt
new file mode 100644
index 0000000..4d211f5
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki.txt
@@ -0,0 +1,7 @@
+Git-Mediawiki is a project which aims the creation of a gate
+between git and mediawiki, allowing git users to push and pull
+objects from mediawiki just as one would do with a classic git
+repository thanks to remote-helpers.
+
+For more information, visit the wiki at
+https://github.com/Bibzball/Git-Mediawiki/wiki
-- 
1.7.7.rc0.78.ge4b5a.dirty

Re: [PATCH v6] Add a remote helper to interact with mediawiki (fetch & push)

[Sverre Rabbelier]

Heya,

2011/8/31 Matthieu Moy <Matthieu.Moy@imag.fr>:
> So, after understanding better how import works, here's an updated
> patch that gets rid of the hacky workaround to terminate and send the
> "done" command at the right time.

So what do you think of the way the protocol works now? Do you agree
that (modulo lacking docs) it is better than previously?

> Actually, push had the same problem but it just went unnoticed (the
> remote has just one branch, so it's silly to try to push multiple
> branches at the same time ...). This version handles push more
> cleanly, giving accurate error message in cases like
>
>  git push origin :master
>  git push origin foo bar master
>
> or perhaps more commonly
>
>  git push --all
>
> in a repository with branches other than master.

My perl skills are minimal, but I'm curious how/where you implemented
this? Is this something that we can port to remote-testgit to document
the CPB on handling such things?




-- 
Cheers,

Sverre Rabbelier

Re: [PATCH v6] Add a remote helper to interact with mediawiki (fetch & push)

[Matthieu Moy]

Sverre Rabbelier <srabbelier@gmail.com> writes:

> Heya,
>
> 2011/8/31 Matthieu Moy <Matthieu.Moy@imag.fr>:
>> So, after understanding better how import works, here's an updated
>> patch that gets rid of the hacky workaround to terminate and send the
>> "done" command at the right time.
>
> So what do you think of the way the protocol works now? Do you agree
> that (modulo lacking docs) it is better than previously?

I'm not sure I understood exactly how it was before, but the current
protocol seems indeed at least reasonable:

* It's possible to specify a batch of imports, so the remote-helper has
  freedom to optimize the import of multiple refs.

* A batch of import is clearly delimited, both on stdin and stdout, so
  it is possible to alternate import batches and other commands.

I still have a few complaints, because even with a better doc, I still
found the debugging a bit hard. To make it easy for remote-helpers
authors, I think the transport-helper could have an explicit "done"
command, so that the command stream look like

import foo
import bar
\n
done

instead of

import foo
import bar
\n
\n

and to let the remote-helper's code be like

while($cmd = <read command>) {
   if ($cmd eq "command1") {
       do something;
   } elsif ($cmd eq "command2") {
       something else;
   } elsif ($cmd eq "done") {
       exit properly;
   }
}

I'm not sure whether changing this now is worth the trouble though.

I'd have appreciated if the transport-helper had given me an explicit
error message when writting to a broken pipe too. I finally got it with
gdb, but lost some time trying to understand (especially painfull since
there was a race condition between the remote-helper termination and git
writting to it, so the bug wasn't reproducible).

>> Actually, push had the same problem but it just went unnoticed (the
>> remote has just one branch, so it's silly to try to push multiple
>> branches at the same time ...). This version handles push more
>> cleanly, giving accurate error message in cases like
>>
>>  git push origin :master
>>  git push origin foo bar master
>>
>> or perhaps more commonly
>>
>>  git push --all
>>
>> in a repository with branches other than master.
>
> My perl skills are minimal, but I'm curious how/where you implemented
> this?

Here:

+	for my $refspec (@refsspecs) {
+		unless ($refspec =~ m/^(\+?)([^:]*):([^:]*)$/) {
+			die("Invalid refspec for push. Expected <src>:<dst> or +<src>:<dst>");
+		}
+		my ($force, $local, $remote) = ($1 eq "+", $2, $3);

At this point, $force is a boolean saying whether there were a +, and
$local and $remote are as you can guess.

+		if ($force) {
+			print STDERR "Warning: forced push not allowed on a MediaWiki.\n";
+		}
+		if ($local eq "") {
+			print STDERR "Cannot delete remote branch on a MediaWiki\n";
+			print STDOUT "error $remote cannot delete\n";

print STDERR goes to the console (i.e. to the user), and print STDOUT
goes to the Git's transport-helper.

+			next;
+		}
+		if ($remote ne "refs/heads/master") {
+			print STDERR "Only push to the branch 'master' is supported on a MediaWiki\n";
+			print STDOUT "error $remote only master allowed\n";
+			next;
+		}

> Is this something that we can port to remote-testgit to document the
> CPB on handling such things?

CPB = ?

Actually, my case is very particular, since the only thing to do with
branches is to make sure the user doesn't use them. In remote-testgit,
there are actually branches.

And testgit use the undocumented "export" feature, which does not seem
to support branch deletion:

git push origin :branch2
fatal: remote-helpers do not support ref deletion
moy@bauges:/tmp/clone$ Traceback (most recent call last):
  File "/home/moy/local/usr-squeeze/libexec/git-core/git-remote-testgit", line 252, in <module>
    sys.exit(main(sys.argv))
  File "/home/moy/local/usr-squeeze/libexec/git-core/git-remote-testgit", line 249, in main
    more = read_one_line(repo)
  File "/home/moy/local/usr-squeeze/libexec/git-core/git-remote-testgit", line 215, in read_one_line
    sys.stdout.flush()
IOError: [Errno 32] Broken pipe

(This comes from

transport-helper.c:750:                 die("remote-helpers do not support ref deletion");

called before starting the exporter)

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH v6] Add a remote helper to interact with mediawiki (fetch & push)

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:

> Here:
>
> +	for my $refspec (@refsspecs) {
> +		unless ($refspec =~ m/^(\+?)([^:]*):([^:]*)$/) {
> +			die("Invalid refspec for push. Expected <src>:<dst> or +<src>:<dst>");
> +		}
> +		my ($force, $local, $remote) = ($1 eq "+", $2, $3);
>
> At this point, $force is a boolean saying whether there were a +, and
> $local and $remote are as you can guess.

It may be slightly more Perl-ish to hoist the "0-or-1" outside the group
and rely on $1 becoming undef, like this:

        my ($force, $local, $remote) = $refspec =~ /^(\+)?([^:]*):([^:]*)$/
		or die(...);

Even though it largely is a matter of taste, I think.

Re: [PATCH v6] Add a remote helper to interact with mediawiki (fetch & push)

[Matthieu Moy]

Junio C Hamano <gitster@pobox.com> writes:

> Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:
>
>> Here:
>>
>> +	for my $refspec (@refsspecs) {
>> +		unless ($refspec =~ m/^(\+?)([^:]*):([^:]*)$/) {
>> +			die("Invalid refspec for push. Expected <src>:<dst> or +<src>:<dst>");
>> +		}
>> +		my ($force, $local, $remote) = ($1 eq "+", $2, $3);
>>
>> At this point, $force is a boolean saying whether there were a +, and
>> $local and $remote are as you can guess.
>
> It may be slightly more Perl-ish to hoist the "0-or-1" outside the group
> and rely on $1 becoming undef, like this:
>
>         my ($force, $local, $remote) = $refspec =~ /^(\+)?([^:]*):([^:]*)$/
> 		or die(...);

Thanks, I didn't know I could do this.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

[PATCH 0/2] Git-MediaWiki

[Matthieu Moy]

First patch is just a resend of v6 with style improvements (Junio's
remark, and whitespace fixes). The second patch is new, it can be
squashed into the first if needed, but I've left it appart since I
think "what to do with metadata on push?" is one of the crucial
questions to be answered when writting remote-helpers, so it may
(hopefully) lead to fruitfull discussions.

Jeremie Nikaes (1):
  Add a remote helper to interact with mediawiki (fetch & push)

Matthieu Moy (1):
  git-remote-mediawiki: allow push to set MediaWiki metadata

 contrib/mw-to-git/git-remote-mediawiki     |  787 ++++++++++++++++++++++++++++
 contrib/mw-to-git/git-remote-mediawiki.txt |    7 +
 2 files changed, 794 insertions(+), 0 deletions(-)
 create mode 100755 contrib/mw-to-git/git-remote-mediawiki
 create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt

-- 
1.7.7.rc0.75.g56f27

[PATCH 1/2 v7] Add a remote helper to interact with mediawiki (fetch & push)

[Matthieu Moy]

From: Jeremie Nikaes <jeremie.nikaes@ensimag.imag.fr>

Implement a gate between git and mediawiki, allowing git users to push
and pull objects from mediawiki just as one would do with a classic git
repository thanks to remote-helpers.

The following packages need to be installed (available on common
repositories):

  libmediawiki-api-perl
  libdatetime-format-iso8601-perl

Use remote helpers in order to be as transparent as possible to the git
user.

Download Mediawiki revisions through the Mediawiki API and then
fast-import into git.

Mediawiki revision number and git commits are linked thanks to notes
bound to commits.

The import part is done on a refs/mediawiki/<remote> branch before
coming to refs/remote/origin/master (Huge thanks to Jonathan Nieder
for his help)

We use UTF-8 everywhere: use encoding 'utf8'; does most of the job, but
we also read the output of Git commands in UTF-8 with the small helper
run_git, and write to the console (STDERR) in UTF-8. This allows a
seamless use of non-ascii characters in page titles, but hasn't been
tested on non-UTF-8 systems. In particular, UTF-8 encoding for filenames
could raise problems if different file systems handle UTF-8 filenames
differently. A uri_escape of mediawiki filenames could be imaginable, and
is still to be discussed further.

Partial cloning is supported using one of:

git clone -c remote.origin.pages='A_Page  Another_Page' mediawiki::http://wikiurl

git clone -c remote.origin.categories='Some_Category' mediawiki::http://wikiurl

git clone -c remote.origin.shallow='True' mediawiki::http://wikiurl

Thanks to notes metadata, it is possible to compare remote and local last
mediawiki revision to warn non-fast forward pushes and "everything
up-to-date" case.

When allowed, push looks for each commit between remotes/origin/master
and HEAD, catches every blob related to these commit and push them in
chronological order. To do so, it uses git rev-list --children HEAD and
travels the tree from remotes/origin/master to HEAD through children. In
other words:

* Shortest path from remotes/origin/master to HEAD
* For each commit encountered, push blobs related to this commit

Signed-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
Signed-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
Signed-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>
Signed-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>
Signed-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
Signed-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>

---
Changes since v6:

* Regexp match simplification, thanks to Junio

* Trivial whitespace fixes, for consistancy.

 contrib/mw-to-git/git-remote-mediawiki     |  756 ++++++++++++++++++++++++++++
 contrib/mw-to-git/git-remote-mediawiki.txt |    7 +
 2 files changed, 763 insertions(+), 0 deletions(-)
 create mode 100755 contrib/mw-to-git/git-remote-mediawiki
 create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
new file mode 100755
index 0000000..d4d1198
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -0,0 +1,756 @@
+#! /usr/bin/perl
+
+# Copyright (C) 2011
+#     Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
+#     Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
+#     Claire Fousse <claire.fousse@ensimag.imag.fr>
+#     David Amouyal <david.amouyal@ensimag.imag.fr>
+#     Matthieu Moy <matthieu.moy@grenoble-inp.fr>
+# License: GPL v2 or later
+
+# Gateway between Git and MediaWiki.
+#   https://github.com/Bibzball/Git-Mediawiki/wiki
+#
+# Known limitations:
+#
+# - Only wiki pages are managed, no support for [[File:...]]
+#   attachments.
+#
+# - Poor performance in the best case: it takes forever to check
+#   whether we're up-to-date (on fetch or push) or to fetch a few
+#   revisions from a large wiki, because we use exclusively a
+#   page-based synchronization. We could switch to a wiki-wide
+#   synchronization when the synchronization involves few revisions
+#   but the wiki is large.
+#
+# - Git renames could be turned into MediaWiki renames (see TODO
+#   below)
+#
+# - login/password support requires the user to write the password
+#   cleartext in a file (see TODO below).
+#
+# - No way to import "one page, and all pages included in it"
+#
+# - Multiple remote MediaWikis have not been very well tested.
+
+use strict;
+use MediaWiki::API;
+use DateTime::Format::ISO8601;
+use encoding 'utf8';
+
+# use encoding 'utf8' doesn't change STDERROR
+# but we're going to output UTF-8 filenames to STDERR
+binmode STDERR, ":utf8";
+
+use URI::Escape;
+use warnings;
+
+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced
+use constant SLASH_REPLACEMENT => "%2F";
+
+# It's not always possible to delete pages (may require some
+# priviledges). Deleted pages are replaced with this content.
+use constant DELETED_CONTENT => "[[Category:Deleted]]\n";
+
+# It's not possible to create empty pages. New empty files in Git are
+# sent with this content instead.
+use constant EMPTY_CONTENT => "<!-- empty page -->\n";
+
+# used to reflect file creation or deletion in diff.
+use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
+
+my $remotename = $ARGV[0];
+my $url = $ARGV[1];
+
+# Accept both space-separated and multiple keys in config file.
+# Spaces should be written as _ anyway because we'll use chomp.
+my @tracked_pages = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".pages"));
+chomp(@tracked_pages);
+
+# Just like @tracked_pages, but for MediaWiki categories.
+my @tracked_categories = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".categories"));
+chomp(@tracked_categories);
+
+my $wiki_login = run_git("config --get remote.". $remotename .".mwLogin");
+# TODO: ideally, this should be able to read from keyboard, but we're
+# inside a remote helper, so our stdin is connect to git, not to a
+# terminal.
+my $wiki_passwd = run_git("config --get remote.". $remotename .".mwPassword");
+chomp($wiki_login);
+chomp($wiki_passwd);
+
+# Import only last revisions (both for clone and fetch)
+my $shallow_import = run_git("config --get --bool remote.". $remotename .".shallow");
+chomp($shallow_import);
+$shallow_import = ($shallow_import eq "true");
+
+my $wiki_name = $url;
+$wiki_name =~ s/[^\/]*:\/\///;
+
+# Commands parser
+my $entry;
+my @cmd;
+while (<STDIN>) {
+	chomp;
+	@cmd = split(/ /);
+	if (defined($cmd[0])) {
+		# Line not blank
+		if ($cmd[0] eq "capabilities") {
+			die("Too many arguments for capabilities") unless (!defined($cmd[1]));
+			mw_capabilities();
+		} elsif ($cmd[0] eq "list") {
+			die("Too many arguments for list") unless (!defined($cmd[2]));
+			mw_list($cmd[1]);
+		} elsif ($cmd[0] eq "import") {
+			die("Invalid arguments for import") unless ($cmd[1] ne "" && !defined($cmd[2]));
+			mw_import($cmd[1]);
+		} elsif ($cmd[0] eq "option") {
+			die("Too many arguments for option") unless ($cmd[1] ne "" && $cmd[2] ne "" && !defined($cmd[3]));
+			mw_option($cmd[1],$cmd[2]);
+		} elsif ($cmd[0] eq "push") {
+			mw_push($cmd[1]);
+		} else {
+			print STDERR "Unknown command. Aborting...\n";
+			last;
+		}
+	} else {
+		# blank line: we should terminate
+		last;
+	}
+
+	BEGIN { $| = 1 } # flush STDOUT, to make sure the previous
+			 # command is fully processed.
+}
+
+########################## Functions ##############################
+
+# MediaWiki API instance, created lazily.
+my $mediawiki;
+
+sub mw_connect_maybe {
+	if ($mediawiki) {
+	    return;
+	}
+	$mediawiki = MediaWiki::API->new;
+	$mediawiki->{config}->{api_url} = "$url/api.php";
+	if ($wiki_login) {
+		if (!$mediawiki->login({
+			lgname => $wiki_login,
+			lgpassword => $wiki_passwd,
+		})) {
+			print STDERR "Failed to log in mediawiki user \"$wiki_login\" on $url\n";
+			print STDERR "(error " .
+			    $mediawiki->{error}->{code} . ': ' .
+			    $mediawiki->{error}->{details} . ")\n";
+			exit 1;
+		} else {
+			print STDERR "Logged in with user \"$wiki_login\".\n";
+		}
+	}
+}
+
+sub get_mw_first_pages {
+	my $some_pages = shift;
+	my @some_pages = @{$some_pages};
+
+	my $pages = shift;
+
+	# pattern 'page1|page2|...' required by the API
+	my $titles = join('|', @some_pages);
+
+	my $mw_pages = $mediawiki->api({
+		action => 'query',
+		titles => $titles,
+	});
+	if (!defined($mw_pages)) {
+		print STDERR "fatal: could not query the list of wiki pages.\n";
+		print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+		print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+		exit 1;
+	}
+	while (my ($id, $page) = each(%{$mw_pages->{query}->{pages}})) {
+		if ($id < 0) {
+			print STDERR "Warning: page $page->{title} not found on wiki\n";
+		} else {
+			$pages->{$page->{title}} = $page;
+		}
+	}
+}
+
+sub get_mw_pages {
+	mw_connect_maybe();
+
+	my %pages; # hash on page titles to avoid duplicates
+	my $user_defined;
+	if (@tracked_pages) {
+		$user_defined = 1;
+		# The user provided a list of pages titles, but we
+		# still need to query the API to get the page IDs.
+
+		my @some_pages = @tracked_pages;
+		while (@some_pages) {
+			my $last = 50;
+			if ($#some_pages < $last) {
+				$last = $#some_pages;
+			}
+			my @slice = @some_pages[0..$last];
+			get_mw_first_pages(\@slice, \%pages);
+			@some_pages = @some_pages[51..$#some_pages];
+		}
+	}
+	if (@tracked_categories) {
+		$user_defined = 1;
+		foreach my $category (@tracked_categories) {
+			if (index($category, ':') < 0) {
+				# Mediawiki requires the Category
+				# prefix, but let's not force the user
+				# to specify it.
+				$category = "Category:" . $category;
+			}
+			my $mw_pages = $mediawiki->list( {
+				action => 'query',
+				list => 'categorymembers',
+				cmtitle => $category,
+				cmlimit => 'max' } )
+			    || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};
+			foreach my $page (@{$mw_pages}) {
+				$pages{$page->{title}} = $page;
+			}
+		}
+	}
+	if (!$user_defined) {
+		# No user-provided list, get the list of pages from
+		# the API.
+		my $mw_pages = $mediawiki->list({
+			action => 'query',
+			list => 'allpages',
+			aplimit => 500,
+		});
+		if (!defined($mw_pages)) {
+			print STDERR "fatal: could not get the list of wiki pages.\n";
+			print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+			print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+			exit 1;
+		}
+		foreach my $page (@{$mw_pages}) {
+			$pages{$page->{title}} = $page;
+		}
+	}
+	return values(%pages);
+}
+
+sub run_git {
+	open(my $git, "-|:encoding(UTF-8)", "git " . $_[0]);
+	my $res = do { local $/; <$git> };
+	close($git);
+
+	return $res;
+}
+
+
+sub get_last_local_revision {
+	# Get note regarding last mediawiki revision
+	my $note = run_git("notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null");
+	my @note_info = split(/ /, $note);
+
+	my $lastrevision_number;
+	if (!(defined($note_info[0]) && $note_info[0] eq "mediawiki_revision:")) {
+		print STDERR "No previous mediawiki revision found";
+		$lastrevision_number = 0;
+	} else {
+		# Notes are formatted : mediawiki_revision: #number
+		$lastrevision_number = $note_info[1];
+		chomp($lastrevision_number);
+		print STDERR "Last local mediawiki revision found is $lastrevision_number";
+	}
+	return $lastrevision_number;
+}
+
+sub get_last_remote_revision {
+	mw_connect_maybe();
+
+	my @pages = get_mw_pages();
+
+	my $max_rev_num = 0;
+
+	foreach my $page (@pages) {
+		my $id = $page->{pageid};
+
+		my $query = {
+			action => 'query',
+			prop => 'revisions',
+			rvprop => 'ids',
+			pageids => $id,
+		};
+
+		my $result = $mediawiki->api($query);
+
+		my $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});
+
+		$max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);
+	}
+
+	print STDERR "Last remote revision found is $max_rev_num.\n";
+	return $max_rev_num;
+}
+
+# Clean content before sending it to MediaWiki
+sub mediawiki_clean {
+	my $string = shift;
+	my $page_created = shift;
+	# Mediawiki does not allow blank space at the end of a page and ends with a single \n.
+	# This function right trims a string and adds a \n at the end to follow this rule
+	$string =~ s/\s+$//;
+	if ($string eq "" && $page_created) {
+		# Creating empty pages is forbidden.
+		$string = EMPTY_CONTENT;
+	}
+	return $string."\n";
+}
+
+# Filter applied on MediaWiki data before adding them to Git
+sub mediawiki_smudge {
+	my $string = shift;
+	if ($string eq EMPTY_CONTENT) {
+		$string = "";
+	}
+	# This \n is important. This is due to mediawiki's way to handle end of files.
+	return $string."\n";
+}
+
+sub mediawiki_clean_filename {
+	my $filename = shift;
+	$filename =~ s/@{[SLASH_REPLACEMENT]}/\//g;
+	# [, ], |, {, and } are forbidden by MediaWiki, even URL-encoded.
+	# Do a variant of URL-encoding, i.e. looks like URL-encoding,
+	# but with _ added to prevent MediaWiki from thinking this is
+	# an actual special character.
+	$filename =~ s/[\[\]\{\}\|]/sprintf("_%%_%x", ord($&))/ge;
+	# If we use the uri escape before
+	# we should unescape here, before anything
+
+	return $filename;
+}
+
+sub mediawiki_smudge_filename {
+	my $filename = shift;
+	$filename =~ s/\//@{[SLASH_REPLACEMENT]}/g;
+	$filename =~ s/ /_/g;
+	# Decode forbidden characters encoded in mediawiki_clean_filename
+	$filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf("%c", hex($1))/ge;
+	return $filename;
+}
+
+sub literal_data {
+	my ($content) = @_;
+	print STDOUT "data ", bytes::length($content), "\n", $content;
+}
+
+sub mw_capabilities {
+	# Revisions are imported to the private namespace
+	# refs/mediawiki/$remotename/ by the helper and fetched into
+	# refs/remotes/$remotename later by fetch.
+	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
+	print STDOUT "import\n";
+	print STDOUT "list\n";
+	print STDOUT "push\n";
+	print STDOUT "\n";
+}
+
+sub mw_list {
+	# MediaWiki do not have branches, we consider one branch arbitrarily
+	# called master, and HEAD pointing to it.
+	print STDOUT "? refs/heads/master\n";
+	print STDOUT "\@refs/heads/master HEAD\n";
+	print STDOUT "\n";
+}
+
+sub mw_option {
+	print STDERR "remote-helper command 'option $_[0]' not yet implemented\n";
+	print STDOUT "unsupported\n";
+}
+
+sub fetch_mw_revisions_for_page {
+	my $page = shift;
+	my $id = shift;
+	my $fetch_from = shift;
+	my @page_revs = ();
+	my $query = {
+		action => 'query',
+		prop => 'revisions',
+		rvprop => 'ids',
+		rvdir => 'newer',
+		rvstartid => $fetch_from,
+		rvlimit => 500,
+		pageids => $id,
+	};
+
+	my $revnum = 0;
+	# Get 500 revisions at a time due to the mediawiki api limit
+	while (1) {
+		my $result = $mediawiki->api($query);
+
+		# Parse each of those 500 revisions
+		foreach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {
+			my $page_rev_ids;
+			$page_rev_ids->{pageid} = $page->{pageid};
+			$page_rev_ids->{revid} = $revision->{revid};
+			push(@page_revs, $page_rev_ids);
+			$revnum++;
+		}
+		last unless $result->{'query-continue'};
+		$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};
+	}
+	if ($shallow_import && @page_revs) {
+		print STDERR "  Found 1 revision (shallow import).\n";
+		@page_revs = sort {$b->{revid} <=> $a->{revid}} (@page_revs);
+		return $page_revs[0];
+	}
+	print STDERR "  Found ", $revnum, " revision(s).\n";
+	return @page_revs;
+}
+
+sub fetch_mw_revisions {
+	my $pages = shift; my @pages = @{$pages};
+	my $fetch_from = shift;
+
+	my @revisions = ();
+	my $n = 1;
+	foreach my $page (@pages) {
+		my $id = $page->{pageid};
+
+		print STDERR "page $n/", scalar(@pages), ": ". $page->{title} ."\n";
+		$n++;
+		my @page_revs = fetch_mw_revisions_for_page($page, $id, $fetch_from);
+		@revisions = (@page_revs, @revisions);
+	}
+
+	return ($n, @revisions);
+}
+
+sub import_file_revision {
+	my $commit = shift;
+	my %commit = %{$commit};
+	my $full_import = shift;
+	my $n = shift;
+
+	my $title = $commit{title};
+	my $comment = $commit{comment};
+	my $content = $commit{content};
+	my $author = $commit{author};
+	my $date = $commit{date};
+
+	print STDOUT "commit refs/mediawiki/$remotename/master\n";
+	print STDOUT "mark :$n\n";
+	print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+	literal_data($comment);
+
+	# If it's not a clone, we need to know where to start from
+	if (!$full_import && $n == 1) {
+		print STDOUT "from refs/mediawiki/$remotename/master^0\n";
+	}
+	if ($content ne DELETED_CONTENT) {
+		print STDOUT "M 644 inline $title.mw\n";
+		literal_data($content);
+		print STDOUT "\n\n";
+	} else {
+		print STDOUT "D $title.mw\n";
+	}
+
+	# mediawiki revision number in the git note
+	if ($full_import && $n == 1) {
+		print STDOUT "reset refs/notes/$remotename/mediawiki\n";
+	}
+	print STDOUT "commit refs/notes/$remotename/mediawiki\n";
+	print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+	literal_data("Note added by git-mediawiki during import");
+	if (!$full_import && $n == 1) {
+		print STDOUT "from refs/notes/$remotename/mediawiki^0\n";
+	}
+	print STDOUT "N inline :$n\n";
+	literal_data("mediawiki_revision: " . $commit{mw_revision});
+	print STDOUT "\n\n";
+}
+
+# parse a sequence of
+# <cmd> <arg1>
+# <cmd> <arg2>
+# \n
+# (like batch sequence of import and sequence of push statements)
+sub get_more_refs {
+	my $cmd = shift;
+	my @refs;
+	while (1) {
+		my $line = <STDIN>;
+		if ($line =~ m/^$cmd (.*)$/) {
+			push(@refs, $1);
+		} elsif ($line eq "\n") {
+			return @refs;
+		} else {
+			die("Invalid command in a '$cmd' batch: ". $_);
+		}
+	}
+}
+
+sub mw_import {
+	# multiple import commands can follow each other.
+	my @refs = (shift, get_more_refs("import"));
+	foreach my $ref (@refs) {
+		mw_import_ref($ref);
+	}
+	print STDOUT "done\n";
+}
+
+sub mw_import_ref {
+	my $ref = shift;
+	# The remote helper will call "import HEAD" and
+	# "import refs/heads/master".
+	# Since HEAD is a symbolic ref to master (by convention,
+	# followed by the output of the command "list" that we gave),
+	# we don't need to do anything in this case.
+	if ($ref eq "HEAD") {
+		return;
+	}
+
+	mw_connect_maybe();
+
+	my @pages = get_mw_pages();
+
+	print STDERR "Searching revisions...\n";
+	my $last_local = get_last_local_revision();
+	my $fetch_from = $last_local + 1;
+	if ($fetch_from == 1) {
+		print STDERR ", fetching from beginning.\n";
+	} else {
+		print STDERR ", fetching from here.\n";
+	}
+	my ($n, @revisions) = fetch_mw_revisions(\@pages, $fetch_from);
+
+	# Creation of the fast-import stream
+	print STDERR "Fetching & writing export data...\n";
+
+	$n = 0;
+	my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined
+
+	foreach my $pagerevid (sort {$a->{revid} <=> $b->{revid}} @revisions) {
+		# fetch the content of the pages
+		my $query = {
+			action => 'query',
+			prop => 'revisions',
+			rvprop => 'content|timestamp|comment|user|ids',
+			revids => $pagerevid->{revid},
+		};
+
+		my $result = $mediawiki->api($query);
+
+		my $rev = pop(@{$result->{query}->{pages}->{$pagerevid->{pageid}}->{revisions}});
+
+		$n++;
+
+		my %commit;
+		$commit{author} = $rev->{user} || 'Anonymous';
+		$commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
+		$commit{title} = mediawiki_smudge_filename(
+			$result->{query}->{pages}->{$pagerevid->{pageid}}->{title}
+		    );
+		$commit{mw_revision} = $pagerevid->{revid};
+		$commit{content} = mediawiki_smudge($rev->{'*'});
+
+		if (!defined($rev->{timestamp})) {
+			$last_timestamp++;
+		} else {
+			$last_timestamp = $rev->{timestamp};
+		}
+		$commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);
+
+		print STDERR "$n/", scalar(@revisions), ": Revision #$pagerevid->{revid} of $commit{title}\n";
+
+		import_file_revision(\%commit, ($fetch_from == 1), $n);
+	}
+
+	if ($fetch_from == 1 && $n == 0) {
+		print STDERR "You appear to have cloned an empty MediaWiki.\n";
+		# Something has to be done remote-helper side. If nothing is done, an error is
+		# thrown saying that HEAD is refering to unknown object 0000000000000000000
+		# and the clone fails.
+	}
+}
+
+sub error_non_fast_forward {
+	# Native git-push would show this after the summary.
+	# We can't ask it to display it cleanly, so print it
+	# ourselves before.
+	print STDERR "To prevent you from losing history, non-fast-forward updates were rejected\n";
+	print STDERR "Merge the remote changes (e.g. 'git pull') before pushing again. See the\n";
+	print STDERR "'Note about fast-forwards' section of 'git push --help' for details.\n";
+
+	print STDOUT "error $_[0] \"non-fast-forward\"\n";
+	return 0;
+}
+
+sub mw_push_file {
+	my $diff_info = shift;
+	# $diff_info contains a string in this format:
+	# 100644 100644 <sha1_of_blob_before_commit> <sha1_of_blob_now> <status>
+	my @diff_info_split = split(/[ \t]/, $diff_info);
+
+	# Filename, including .mw extension
+	my $complete_file_name = shift;
+	# Commit message
+	my $summary = shift;
+
+	my $new_sha1 = $diff_info_split[3];
+	my $old_sha1 = $diff_info_split[2];
+	my $page_created = ($old_sha1 eq NULL_SHA1);
+	my $page_deleted = ($new_sha1 eq NULL_SHA1);
+	$complete_file_name = mediawiki_clean_filename($complete_file_name);
+
+	if (substr($complete_file_name,-3) eq ".mw") {
+		my $title = substr($complete_file_name,0,-3);
+
+		my $file_content;
+		if ($page_deleted) {
+			# Deleting a page usually requires
+			# special priviledges. A common
+			# convention is to replace the page
+			# with this content instead:
+			$file_content = DELETED_CONTENT;
+		} else {
+			$file_content = run_git("cat-file blob $new_sha1");
+		}
+
+		mw_connect_maybe();
+
+		my $result = $mediawiki->edit( {
+			action => 'edit',
+			summary => $summary,
+			title => $title,
+			text => mediawiki_clean($file_content, $page_created),
+				  }, {
+					  skip_encoding => 1 # Helps with names with accentuated characters
+				  }) || die 'Fatal: Error ' .
+				  $mediawiki->{error}->{code} .
+				  ' from mediwiki: ' . $mediawiki->{error}->{details};
+		print STDERR "Pushed file : $new_sha1 - $title\n";
+	} else {
+		print STDERR "$complete_file_name not a mediawiki file (Not pushable on this version of git-remote-mediawiki).\n"
+	}
+}
+
+sub mw_push {
+	# multiple push statements can follow each other
+	my @refsspecs = (shift, get_more_refs("push"));
+	my %status;
+	my $pushed;
+	for my $refspec (@refsspecs) {
+		my ($force, $local, $remote) = $refspec =~ /^(\+)?([^:]*):([^:]*)$/
+		    or die("Invalid refspec for push. Expected <src>:<dst> or +<src>:<dst>");
+		if ($force) {
+			print STDERR "Warning: forced push not allowed on a MediaWiki.\n";
+		}
+		if ($local eq "") {
+			print STDERR "Cannot delete remote branch on a MediaWiki\n";
+			print STDOUT "error $remote cannot delete\n";
+			next;
+		}
+		if ($remote ne "refs/heads/master") {
+			print STDERR "Only push to the branch 'master' is supported on a MediaWiki\n";
+			print STDOUT "error $remote only master allowed\n";
+			next;
+		}
+		if (mw_push_revision($local, $remote)) {
+			$pushed = 1;
+		}
+	}
+
+	# Notify Git that the push is done
+	print STDOUT "\n";
+
+	if ($pushed) {
+		print STDERR "Just pushed some revisions to MediaWiki.\n";
+		print STDERR "The pushed revisions now have to be re-imported, and your current branch\n";
+		print STDERR "needs to be updated with these re-imported commits. You can do this with\n";
+		print STDERR "\n";
+		print STDERR "  git pull --rebase\n";
+		print STDERR "\n";
+	}
+}
+
+sub mw_push_revision {
+	my $local = shift;
+	my $remote = shift; # actually, this has to be "refs/heads/master" at this point.
+	my $last_local_revid = get_last_local_revision();
+	print STDERR ".\n"; # Finish sentence started by get_last_local_revision()
+	my $last_remote_revid = get_last_remote_revision();
+
+	# Get sha1 of commit pointed by local HEAD
+	my $HEAD_sha1 = run_git("rev-parse $local 2>/dev/null"); chomp($HEAD_sha1);
+	# Get sha1 of commit pointed by remotes/$remotename/master
+	my $remoteorigin_sha1 = run_git("rev-parse refs/remotes/$remotename/master 2>/dev/null");
+	chomp($remoteorigin_sha1);
+
+	if ($last_local_revid > 0 &&
+	    $last_local_revid < $last_remote_revid) {
+		return error_non_fast_forward($remote);
+	}
+
+	if ($HEAD_sha1 eq $remoteorigin_sha1) {
+		# nothing to push
+		return 0;
+	}
+
+	# Get every commit in between HEAD and refs/remotes/origin/master,
+	# including HEAD and refs/remotes/origin/master
+	my @commit_pairs = ();
+	if ($last_local_revid > 0) {
+		my $parsed_sha1 = $remoteorigin_sha1;
+		# Find a path from last MediaWiki commit to pushed commit
+		while ($parsed_sha1 ne $HEAD_sha1) {
+			my @commit_info =  grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $local")));
+			if (!@commit_info) {
+				return error_non_fast_forward($remote);
+			}
+			my @commit_info_split = split(/ |\n/, $commit_info[0]);
+			# $commit_info_split[1] is the sha1 of the commit to export
+			# $commit_info_split[0] is the sha1 of its direct child
+			push(@commit_pairs, \@commit_info_split);
+			$parsed_sha1 = $commit_info_split[1];
+		}
+	} else {
+		# No remote mediawiki revision. Export the whole
+		# history (linearized with --first-parent)
+		print STDERR "Warning: no common ancestor, pushing complete history\n";
+		my $history = run_git("rev-list --first-parent --children $local");
+		my @history = split('\n', $history);
+		@history = @history[1..$#history];
+		foreach my $line (reverse @history) {
+			my @commit_info_split = split(/ |\n/, $line);
+			push(@commit_pairs, \@commit_info_split);
+		}
+	}
+
+	foreach my $commit_info_split (@commit_pairs) {
+		my $sha1_child = @{$commit_info_split}[0];
+		my $sha1_commit = @{$commit_info_split}[1];
+		my $diff_infos = run_git("diff-tree -r --raw -z $sha1_child $sha1_commit");
+		# TODO: we could detect rename, and encode them with a #redirect on the wiki.
+		# TODO: for now, it's just a delete+add
+		my @diff_info_list = split(/\0/, $diff_infos);
+		# Keep the first line of the commit message as mediawiki comment for the revision
+		my $commit_msg = (split(/\n/, run_git("show --pretty=format:\"%s\" $sha1_commit")))[0];
+		chomp($commit_msg);
+		# Push every blob
+		while (@diff_info_list) {
+			# git diff-tree -z gives an output like
+			# <metadata>\0<filename1>\0
+			# <metadata>\0<filename2>\0
+			# and we've split on \0.
+			my $info = shift(@diff_info_list);
+			my $file = shift(@diff_info_list);
+			mw_push_file($info, $file, $commit_msg);
+		}
+	}
+
+	print STDOUT "ok $remote\n";
+	return 1;
+}
diff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt
new file mode 100644
index 0000000..4d211f5
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki.txt
@@ -0,0 +1,7 @@
+Git-Mediawiki is a project which aims the creation of a gate
+between git and mediawiki, allowing git users to push and pull
+objects from mediawiki just as one would do with a classic git
+repository thanks to remote-helpers.
+
+For more information, visit the wiki at
+https://github.com/Bibzball/Git-Mediawiki/wiki
-- 
1.7.7.rc0.75.g56f27

[PATCH 2/2] git-remote-mediawiki: allow push to set MediaWiki metadata

[Matthieu Moy]

Push can not set the commit note "mediawiki_revision:" and update the
remote reference. This avoids having to "git pull --rebase" after each
push, and is probably more natural. Make it the default, but let it be
configurable with mediawiki.dumbPush or remote.<remotename>.dumbPush.

Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki |   35 ++++++++++++++++++++++++++++++-
 1 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
index d4d1198..0ba88de 100755
--- a/contrib/mw-to-git/git-remote-mediawiki
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -84,6 +84,27 @@ my $shallow_import = run_git("config --get --bool remote.". $remotename .".shall
 chomp($shallow_import);
 $shallow_import = ($shallow_import eq "true");
 
+# Dumb push: don't update notes and mediawiki ref to reflect the last push.
+#
+# Configurable with mediawiki.dumbPush, or per-remote with
+# remote.<remotename>.dumbPush.
+#
+# This means the user will have to re-import the just-pushed
+# revisions. On the other hand, this means that the Git revisions
+# corresponding to MediaWiki revisions are all imported from the wiki,
+# regardless of whether they were initially created in Git or from the
+# web interface, hence all users will get the same history (i.e. if
+# the push from Git to MediaWiki loses some information, everybody
+# will get the history with information lost). If the import is
+# deterministic, this means everybody gets the same sha1 for each
+# MediaWiki revision.
+my $dumb_push = run_git("config --get --bool remote.$remotename.dumbPush");
+unless ($dumb_push) {
+	$dumb_push = run_git("config --get --bool mediawiki.dumbPush");
+}
+chomp($dumb_push);
+$dumb_push = ($dumb_push eq "true");
+
 my $wiki_name = $url;
 $wiki_name =~ s/[^\/]*:\/\///;
 
@@ -598,6 +619,9 @@ sub mw_push_file {
 	my $complete_file_name = shift;
 	# Commit message
 	my $summary = shift;
+	# MediaWiki revision number. Keep the previous one by default,
+	# in case there's no edit to perform.
+	my $newrevid = shift;
 
 	my $new_sha1 = $diff_info_split[3];
 	my $old_sha1 = $diff_info_split[2];
@@ -631,10 +655,12 @@ sub mw_push_file {
 				  }) || die 'Fatal: Error ' .
 				  $mediawiki->{error}->{code} .
 				  ' from mediwiki: ' . $mediawiki->{error}->{details};
+		$newrevid = $result->{edit}->{newrevid};
 		print STDERR "Pushed file : $new_sha1 - $title\n";
 	} else {
 		print STDERR "$complete_file_name not a mediawiki file (Not pushable on this version of git-remote-mediawiki).\n"
 	}
+	return $newrevid;
 }
 
 sub mw_push {
@@ -666,7 +692,7 @@ sub mw_push {
 	# Notify Git that the push is done
 	print STDOUT "\n";
 
-	if ($pushed) {
+	if ($pushed && $dumb_push) {
 		print STDERR "Just pushed some revisions to MediaWiki.\n";
 		print STDERR "The pushed revisions now have to be re-imported, and your current branch\n";
 		print STDERR "needs to be updated with these re-imported commits. You can do this with\n";
@@ -682,6 +708,7 @@ sub mw_push_revision {
 	my $last_local_revid = get_last_local_revision();
 	print STDERR ".\n"; # Finish sentence started by get_last_local_revision()
 	my $last_remote_revid = get_last_remote_revision();
+	my $mw_revision = $last_remote_revid;
 
 	# Get sha1 of commit pointed by local HEAD
 	my $HEAD_sha1 = run_git("rev-parse $local 2>/dev/null"); chomp($HEAD_sha1);
@@ -747,7 +774,11 @@ sub mw_push_revision {
 			# and we've split on \0.
 			my $info = shift(@diff_info_list);
 			my $file = shift(@diff_info_list);
-			mw_push_file($info, $file, $commit_msg);
+			$mw_revision = mw_push_file($info, $file, $commit_msg, $mw_revision);
+		}
+		unless ($dumb_push) {
+			run_git("notes --ref=$remotename/mediawiki add -m \"mediawiki_revision: $mw_revision\" $sha1_commit");
+			run_git("update-ref -m \"Git-MediaWiki push\" refs/mediawiki/$remotename/master $sha1_commit $sha1_child");
 		}
 	}
 
-- 
1.7.7.rc0.75.g56f27

Clean termination of remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))

[Matthieu Moy]

Matthieu Moy <Matthieu.Moy@imag.fr> writes:

> +BEGIN { $| = 1 };
> +if (!eof(STDIN)) {
> +	# Wait for Git to terminate. If we don't, git fetch
> +	# (transport-helper.c's sendline function) will try to write
> +	# to our stdin, which will be closed, and git fetch will be
> +	# killed. That's probably a bug in transport-helper.c, but in
> +	# the meantime ...
> +	sleep .1;
> +};

I was expecting this part to be more controversial, so I'm just
repeating it to draw more attention ;-).

I just found a "cleaner" way to terminate, but I still don't find it
really satisfactory:

# Inform Git that we're done, otherwise Git won't close it's stdin,
# and the next loop will be infinite.
close(STDOUT);
# Flush stdin before we terminate. If we don't, git fetch
# (transport-helper.c's sendline function) will try to write to our
# stdin, which may be closed, and git fetch will be killed. That's
# probably a bug in transport-helper.c, but in the meantime ...
while (<STDIN>) {};

This seems reliable (just did 100+ imports without crash).

If I comment-out the "close(STDOUT)", then the while loop is indeed
infinite. If I comment-out the "while (<STDIN>) {};", then the import
unreliably fails (for example, I just did 10 clones of a 1-page wiki,
and got one failure). No error message, just a non-zero exit status, and
in the case of "clone", the newly created repository is deleted before
the command terminates.

With debug activated in transport-helper.c, the last messages are just:

Debug: Disconnecting.
Debug: Remote helper: -> 

and gdb says:

Program received signal SIGPIPE, Broken pipe.
0xb7fe2424 in __kernel_vsyscall ()
(gdb) bt
#0  0xb7fe2424 in __kernel_vsyscall ()
#1  0xb7e66ff3 in __write_nocancel () at ../sysdeps/unix/syscall-template.S:82
#2  0x0811dc08 in xwrite (fd=8, buf=0x81bd840, len=1) at wrapper.c:137
#3  0x0811dc67 in write_in_full (fd=8, buf=0x81bd840, count=1) at wrapper.c:169
#4  0x08115035 in sendline (helper=0x81a2d00, buffer=0xbfffe674) at transport-helper.c:41
#5  0x081158f3 in disconnect_helper (transport=0x81a2cc0) at transport-helper.c:231
#6  release_helper (transport=0x81a2cc0) at transport-helper.c:324
#7  0x08111f1d in transport_disconnect (transport=0x81a2cc0) at transport.c:1144
#8  0x08061711 in cmd_clone (argc=3, argv=0xbfffeb58, prefix=0x0) at builtin/clone.c:739
#9  0x0804ba27 in run_builtin (argc=<value optimized out>, argv=<value optimized out>) at git.c:308
#10 handle_internal_command (argc=<value optimized out>, argv=<value optimized out>) at git.c:466
#11 0x0804bc33 in run_argv (argc=3, argv=0xbfffeb58) at git.c:512
#12 main (argc=3, argv=0xbfffeb58) at git.c:585

Any idea how to fix this?

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: Clean termination of remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))

[Sverre Rabbelier]

Heya,

On Wed, Aug 31, 2011 at 14:33, Matthieu Moy
<Matthieu.Moy@grenoble-inp.fr> wrote:
> I was expecting this part to be more controversial, so I'm just
> repeating it to draw more attention ;-).

Eek! :)

> # Inform Git that we're done, otherwise Git won't close it's stdin,
> # and the next loop will be infinite.
> close(STDOUT);
> # Flush stdin before we terminate. If we don't, git fetch
> # (transport-helper.c's sendline function) will try to write to our
> # stdin, which may be closed, and git fetch will be killed. That's
> # probably a bug in transport-helper.c, but in the meantime ...
> while (<STDIN>) {};

Is this caused by you not reading the terminating '\n' that git sends
when all commands are done? We previously didn't do this for imports
(we couldn't, hence the new import/done behavior), so perhaps you just
need to fix that? It's entirely possible you've found a bug though.

-- 
Cheers,

Sverre Rabbelier

Re: Clean termination of remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))

[Matthieu Moy]

Sverre Rabbelier <srabbelier@gmail.com> writes:

> Heya,
>
> On Wed, Aug 31, 2011 at 14:33, Matthieu Moy
> <Matthieu.Moy@grenoble-inp.fr> wrote:
>> I was expecting this part to be more controversial, so I'm just
>> repeating it to draw more attention ;-).
>
> Eek! :)
>
>> # Inform Git that we're done, otherwise Git won't close it's stdin,
>> # and the next loop will be infinite.
>> close(STDOUT);
>> # Flush stdin before we terminate. If we don't, git fetch
>> # (transport-helper.c's sendline function) will try to write to our
>> # stdin, which may be closed, and git fetch will be killed. That's
>> # probably a bug in transport-helper.c, but in the meantime ...
>> while (<STDIN>) {};
>
> Is this caused by you not reading the terminating '\n' that git sends
> when all commands are done?

Indeed. The stream sent by git looks like

import HEAD
import refs/heads/master
\n <-- this one closes the sequence of import
\n <-- this one closes the sequence of commands.

and I was interpreting it as

import HEAD
import refs/heads/master
\n <-- this one closes the sequence of commands
\n <-- what's this??

So it seems the only bug I've found is insufficient documentation. A
patch follows.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: Clean termination of remote-helpers (was Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push))

[Sverre Rabbelier]

Heya,

On Wed, Aug 31, 2011 at 16:53, Matthieu Moy
<Matthieu.Moy@grenoble-inp.fr> wrote:
> So it seems the only bug I've found is insufficient documentation. A
> patch follows.

Yay! Thanks :)

-- 
Cheers,

Sverre Rabbelier

Re: [PATCH 1/2] fast-import: initialize variable require_explicit_termination

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@imag.fr> writes:

> The uninitialized variable seems harmless in practice, but let's still be clean.

It is not "in practice", but by definition, file scope "static int"
variables are initialized to 0 by the C language (a typical implementation
achieves this by placing the variable in BSS section).

Please do not write unnecessary " = 0" there.

>
> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
> ---
> For some reason, remote helpers seem to be forced to use the "done"
> command now. Investing why, I found this, but that wasn't what I was
> looking for.
>
>  fast-import.c |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/fast-import.c b/fast-import.c
> index 7cc2262..ed8f3cd 100644
> --- a/fast-import.c
> +++ b/fast-import.c
> @@ -355,7 +355,7 @@ static unsigned int cmd_save = 100;
>  static uintmax_t next_mark;
>  static struct strbuf new_data = STRBUF_INIT;
>  static int seen_data_command;
> -static int require_explicit_termination;
> +static int require_explicit_termination = 0;
>  
>  /* Signal handling */
>  static volatile sig_atomic_t checkpoint_requested;

Re: [PATCH 1/2] fast-import: initialize variable require_explicit_termination

[Matthieu Moy]

Junio C Hamano <gitster@pobox.com> writes:

> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>
>> The uninitialized variable seems harmless in practice, but let's still be clean.
>
> It is not "in practice", but by definition, file scope "static int"
> variables are initialized to 0 by the C language

OK, I didn't know this was specified in C, but you're right.

> Please do not write unnecessary " = 0" there.

I prefer being explicit, but that's a matter of taste, and I don't
really care, so let's drop this patch.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH 1/2] fast-import: initialize variable require_explicit_termination

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:

>> Please do not write unnecessary " = 0" there.
>
> I prefer being explicit, but that's a matter of taste, and I don't
> really care, so let's drop this patch.

I prefer being explicit, too, but "static int foo;" is explicit enough if
you know C.