[PATCH 00/16] remote-hg: second round of improvements

[PATCH 00/16] remote-hg: second round of improvements

[Felipe Contreras]

Hi,

Now that the safter first round is merged to master, it's time for the second
round which is a little tricker. Most of the patches should be safe, but the
later ones might be not. All the tests pass, even gitifyhg ones, so everything
seems to be fine, but we shall see.

Among the important changes are:

 * Proper support for tags (they are global, and we can push them)
 * Improved e-mail sanitaztion (a bit similar to gitifyhg, but better)
 * Support for hg schemes (like bb://felipec/repo)
 * Support for tags, bookmarks and branches with spaces
 * Performance improvements

Good stuff we should probably merge to master if there are no issues.

Cheers.

Dusty Phillips (1):
  remote-helpers: avoid has_key

Felipe Contreras (15):
  remote-hg: safer bookmark pushing
  remote-hg: use python urlparse
  remote-hg: properly mark branches up-to-date
  remote-hg: add branch_tip() helper
  remote-hg: add support for tag objects
  remote-hg: custom method to write tags
  remote-hg: write tags in the appropriate branch
  remote-hg: add custom local tag write code
  remote-hg: improve email sanitation
  remote-hg: add support for schemes extension
  remote-hg: don't update bookmarks unnecessarily
  remote-hg: allow refs with spaces
  remote-hg: small performance improvement
  remote-hg: use marks instead of inlined files
  remote-hg: strip extra newline

 contrib/remote-helpers/git-remote-bzr |   2 +-
 contrib/remote-helpers/git-remote-hg  | 167 +++++++++++++++++++++++++++-------
 contrib/remote-helpers/test-hg.sh     |   8 +-
 3 files changed, 141 insertions(+), 36 deletions(-)

-- 
1.8.2.1.790.g4588561

[PATCH 01/16] remote-helpers: avoid has_key

[Felipe Contreras]

From: Dusty Phillips <dusty@linux.ca>

It is deprecated.

[fc: do the same in remote-bzr]

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-bzr | 2 +-
 contrib/remote-helpers/git-remote-hg  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
index aa7bc97..cc6609b 100755
--- a/contrib/remote-helpers/git-remote-bzr
+++ b/contrib/remote-helpers/git-remote-bzr
@@ -94,7 +94,7 @@ class Marks:
         return self.last_mark
 
     def is_marked(self, rev):
-        return self.marks.has_key(rev)
+        return str(rev) in self.marks
 
     def new_mark(self, rev, mark):
         self.marks[rev] = mark
diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 5481331..2cd1996 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -129,7 +129,7 @@ class Marks:
         self.last_mark = mark
 
     def is_marked(self, rev):
-        return self.marks.has_key(str(rev))
+        return str(rev) in self.marks
 
     def get_tip(self, branch):
         return self.tips.get(branch, 0)
-- 
1.8.2.1.790.g4588561

[PATCH 02/16] remote-hg: safer bookmark pushing

[Felipe Contreras]

It is possible that the remote has changed the bookmarks, so let's fetch
them before we make any assumptions, just the way mercurial does.

Probably doesn't make a difference, but better be safe than sorry.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 2cd1996..dcf6c98 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -782,6 +782,8 @@ def do_export(parser):
             continue
 
         if peer:
+            rb = peer.listkeys('bookmarks')
+            old = rb.get(bmark, '')
             if not peer.pushkey('bookmarks', bmark, old, new):
                 print "error %s" % ref
                 continue
-- 
1.8.2.1.790.g4588561

[PATCH 03/16] remote-hg: use python urlparse

[Felipe Contreras]

It's simpler, and we don't need to depend on certain Mercurial versions.

Also, now we don't update the URL if 'file://' is not present.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index dcf6c98..b6589a3 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -22,6 +22,7 @@ import shutil
 import subprocess
 import urllib
 import atexit
+import urlparse
 
 #
 # If you want to switch to hg-git compatibility mode:
@@ -793,11 +794,11 @@ def do_export(parser):
     print
 
 def fix_path(alias, repo, orig_url):
-    repo_url = util.url(repo.url())
-    url = util.url(orig_url)
-    if str(url) == str(repo_url):
+    url = urlparse.urlparse(orig_url, 'file')
+    if url.scheme != 'file' or os.path.isabs(url.path):
         return
-    cmd = ['git', 'config', 'remote.%s.url' % alias, "hg::%s" % repo_url]
+    abs_url = urlparse.urljoin("%s/" % os.getcwd(), orig_url)
+    cmd = ['git', 'config', 'remote.%s.url' % alias, "hg::%s" % abs_url]
     subprocess.call(cmd)
 
 def main(args):
-- 
1.8.2.1.790.g4588561

[PATCH 04/16] remote-hg: properly mark branches up-to-date

[Felipe Contreras]

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index b6589a3..a4db5b0 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -742,6 +742,10 @@ def do_export(parser):
 
     for ref, node in parsed_refs.iteritems():
         if ref.startswith('refs/heads/branches'):
+            branch = ref[len('refs/heads/branches/'):]
+            if branch in branches and node in branches[branch]:
+                # up to date
+                continue
             print "ok %s" % ref
         elif ref.startswith('refs/heads/'):
             bmark = ref[len('refs/heads/'):]
-- 
1.8.2.1.790.g4588561

[PATCH 05/16] remote-hg: add branch_tip() helper

[Felipe Contreras]

Idea from gitifyhg, the backwards compatibility is how Mercurial used to
do it.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index a4db5b0..bd93f82 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -457,6 +457,13 @@ def do_capabilities(parser):
 
     print
 
+def branch_tip(repo, branch):
+    # older versions of mercurial don't have this
+    if hasattr(repo, 'branchtip'):
+        return repo.branchtip(branch)
+    else:
+        return repo.branchtags()[branch]
+
 def get_branch_tip(repo, branch):
     global branches
 
@@ -467,9 +474,7 @@ def get_branch_tip(repo, branch):
     # verify there's only one head
     if (len(heads) > 1):
         warn("Branch '%s' has more than one head, consider merging" % branch)
-        # older versions of mercurial don't have this
-        if hasattr(repo, "branchtip"):
-            return repo.branchtip(branch)
+        return branch_tip(repo, branch)
 
     return heads[0]
 
-- 
1.8.2.1.790.g4588561

[PATCH 06/16] remote-hg: add support for tag objects

[Felipe Contreras]

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index bd93f82..4a1c637 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -722,7 +722,7 @@ def parse_tag(parser):
     data = parser.get_data()
     parser.next()
 
-    # nothing to do
+    parsed_tags[name] = (tagger, data)
 
 def do_export(parser):
     global parsed_refs, bmarks, peer
@@ -758,9 +758,11 @@ def do_export(parser):
             continue
         elif ref.startswith('refs/tags/'):
             tag = ref[len('refs/tags/'):]
+            author, msg = parsed_tags.get(tag, (None, None))
             if mode == 'git':
-                msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6]));
-                parser.repo.tag([tag], node, msg, False, None, {})
+                if not msg:
+                    msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6]));
+                parser.repo.tag([tag], node, msg, False, author[0], {})
             else:
                 parser.repo.tag([tag], node, None, True, None, {})
             print "ok %s" % ref
@@ -815,6 +817,7 @@ def main(args):
     global marks, blob_marks, parsed_refs
     global peer, mode, bad_mail, bad_name
     global track_branches, force_push, is_tmp
+    global parsed_tags
 
     alias = args[1]
     url = args[2]
@@ -857,6 +860,7 @@ def main(args):
     blob_marks = {}
     parsed_refs = {}
     marks = None
+    parsed_tags = {}
 
     repo = get_repo(url, alias)
     prefix = 'refs/hg/%s' % alias
-- 
1.8.2.1.790.g4588561

[PATCH 07/16] remote-hg: custom method to write tags

[Felipe Contreras]

The one from mercurial is meant for users, on top of the latest tip.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 4a1c637..f5e4ba7 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -724,6 +724,37 @@ def parse_tag(parser):
 
     parsed_tags[name] = (tagger, data)
 
+def write_tag(repo, tag, node, msg, author):
+    tip = repo['tip']
+
+    def getfilectx(repo, memctx, f):
+        try:
+            fctx = tip.filectx(f)
+            data = fctx.data()
+        except error.ManifestLookupError:
+            data = ""
+        content = data + "%s %s\n" % (hghex(node), tag)
+        return context.memfilectx(f, content, False, False, None)
+
+    p1 = tip.hex()
+    p2 = '\0' * 20
+    if not author:
+        author = (None, 0, 0)
+    user, date, tz = author
+
+    ctx = context.memctx(repo, (p1, p2), msg,
+            ['.hgtags'], getfilectx,
+            user, (date, tz), {})
+
+    tmp = encoding.encoding
+    encoding.encoding = 'utf-8'
+
+    tagnode = repo.commitctx(ctx)
+
+    encoding.encoding = tmp
+
+    return tagnode
+
 def do_export(parser):
     global parsed_refs, bmarks, peer
 
@@ -762,7 +793,7 @@ def do_export(parser):
             if mode == 'git':
                 if not msg:
                     msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6]));
-                parser.repo.tag([tag], node, msg, False, author[0], {})
+                write_tag(parser.repo, tag, node, msg, author)
             else:
                 parser.repo.tag([tag], node, None, True, None, {})
             print "ok %s" % ref
-- 
1.8.2.1.790.g4588561

[PATCH 08/16] remote-hg: write tags in the appropriate branch

[Felipe Contreras]

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index f5e4ba7..f685990 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -725,7 +725,9 @@ def parse_tag(parser):
     parsed_tags[name] = (tagger, data)
 
 def write_tag(repo, tag, node, msg, author):
-    tip = repo['tip']
+    branch = repo[node].branch()
+    tip = branch_tip(repo, branch)
+    tip = repo[tip]
 
     def getfilectx(repo, memctx, f):
         try:
@@ -744,7 +746,7 @@ def write_tag(repo, tag, node, msg, author):
 
     ctx = context.memctx(repo, (p1, p2), msg,
             ['.hgtags'], getfilectx,
-            user, (date, tz), {})
+            user, (date, tz), {'branch' : branch})
 
     tmp = encoding.encoding
     encoding.encoding = 'utf-8'
-- 
1.8.2.1.790.g4588561

[PATCH 09/16] remote-hg: add custom local tag write code

[Felipe Contreras]

There's no point in calling the tag method for such simple action. Not
that we care much about the hg-git compat mode, it's mostly just for
comparison testing purposes.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index f685990..3c6eeb7 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -797,7 +797,9 @@ def do_export(parser):
                     msg = 'Added tag %s for changeset %s' % (tag, hghex(node[:6]));
                 write_tag(parser.repo, tag, node, msg, author)
             else:
-                parser.repo.tag([tag], node, None, True, None, {})
+                fp = parser.repo.opener('localtags', 'a')
+                fp.write('%s %s\n' % (hghex(node), tag))
+                fp.close()
             print "ok %s" % ref
         else:
             # transport-helper/fast-export bugs
-- 
1.8.2.1.790.g4588561

[PATCH 10/16] remote-hg: improve email sanitation

[Felipe Contreras]

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 10 ++++++++--
 contrib/remote-helpers/test-hg.sh    |  8 ++++----
 2 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 3c6eeb7..0084709 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -51,6 +51,7 @@ import urlparse
 
 NAME_RE = re.compile('^([^<>]+)')
 AUTHOR_RE = re.compile('^([^<>]+?)? ?<([^<>]*)>$')
+EMAIL_RE = re.compile('^([^<>]+[^ \\\t<>])?\\b(?:[ \\t<>]*?)\\b([^ \\t<>]+@[^ \\t<>]+)')
 AUTHOR_HG_RE = re.compile('^(.*?) ?<(.*?)(?:>(.+)?)?$')
 RAW_AUTHOR_RE = re.compile('^(\w+) (?:(.+)? )?<(.*)> (\d+) ([+-]\d+)')
 
@@ -245,9 +246,14 @@ def fixup_user_git(user):
         name = m.group(1)
         mail = m.group(2).strip()
     else:
-        m = NAME_RE.match(user)
+        m = EMAIL_RE.match(user)
         if m:
-            name = m.group(1).strip()
+            name = m.group(1)
+            mail = m.group(2)
+        else:
+            m = NAME_RE.match(user)
+            if m:
+                name = m.group(1).strip()
     return (name, mail)
 
 def fixup_user_hg(user):
diff --git a/contrib/remote-helpers/test-hg.sh b/contrib/remote-helpers/test-hg.sh
index d5b8730..8de2aa7 100755
--- a/contrib/remote-helpers/test-hg.sh
+++ b/contrib/remote-helpers/test-hg.sh
@@ -137,15 +137,15 @@ test_expect_success 'authors' '
 
   author_test alpha "" "H G Wells <wells@example.com>" &&
   author_test beta "test" "test <unknown>" &&
-  author_test beta "test <test@example.com> (comment)" "test <unknown>" &&
+  author_test beta "test <test@example.com> (comment)" "test <test@example.com>" &&
   author_test gamma "<test@example.com>" "Unknown <test@example.com>" &&
   author_test delta "name<test@example.com>" "name <test@example.com>" &&
-  author_test epsilon "name <test@example.com" "name <unknown>" &&
+  author_test epsilon "name <test@example.com" "name <test@example.com>" &&
   author_test zeta " test " "test <unknown>" &&
   author_test eta "test < test@example.com >" "test <test@example.com>" &&
-  author_test theta "test >test@example.com>" "test <unknown>" &&
+  author_test theta "test >test@example.com>" "test <test@example.com>" &&
   author_test iota "test < test <at> example <dot> com>" "test <unknown>" &&
-  author_test kappa "test@example.com" "test@example.com <unknown>"
+  author_test kappa "test@example.com" "Unknown <test@example.com>"
   ) &&
 
   git clone "hg::$PWD/hgrepo" gitrepo &&
-- 
1.8.2.1.790.g4588561

[PATCH 11/16] remote-hg: add support for schemes extension

[Felipe Contreras]

So that we can use shortened URLs, for example 'bb:://felipec/repo'
(Bitbucket).

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 0084709..4f6c7b7 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -12,7 +12,7 @@
 # For remote repositories a local clone is stored in
 # "$GIT_DIR/hg/origin/clone/.hg/".
 
-from mercurial import hg, ui, bookmarks, context, util, encoding, node, error
+from mercurial import hg, ui, bookmarks, context, util, encoding, node, error, extensions
 
 import re
 import sys
@@ -305,6 +305,12 @@ def get_repo(url, alias):
     except subprocess.CalledProcessError:
         pass
 
+    try:
+        mod = extensions.load(myui, 'hgext.schemes', None)
+        mod.extsetup(myui)
+    except ImportError:
+        pass
+
     if hg.islocal(url):
         repo = hg.repository(myui, url)
     else:
-- 
1.8.2.1.790.g4588561

[PATCH 12/16] remote-hg: don't update bookmarks unnecessarily

[Felipe Contreras]

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 4f6c7b7..dbb4091 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -830,6 +830,9 @@ def do_export(parser):
         else:
             old = ''
 
+        if old == new:
+            continue
+
         if bmark == 'master' and 'master' not in parser.repo._bookmarks:
             # fake bookmark
             pass
-- 
1.8.2.1.790.g4588561

[PATCH 13/16] remote-hg: allow refs with spaces

[Felipe Contreras]

Mercurial supports them, Git doesn't.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index dbb4091..6f4afd7 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -75,6 +75,12 @@ def hgmode(mode):
 def hghex(node):
     return hg.node.hex(node)
 
+def hgref(ref):
+    return ref.replace('___', ' ')
+
+def gitref(ref):
+    return ref.replace(' ', '___')
+
 def get_config(config):
     cmd = ['git', 'config', '--get', config]
     process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
@@ -437,10 +443,10 @@ def export_ref(repo, name, kind, head):
     marks.set_tip(ename, rev)
 
 def export_tag(repo, tag):
-    export_ref(repo, tag, 'tags', repo[tag])
+    export_ref(repo, tag, 'tags', repo[hgref(tag)])
 
 def export_bookmark(repo, bmark):
-    head = bmarks[bmark]
+    head = bmarks[hgref(bmark)]
     export_ref(repo, bmark, 'bookmarks', head)
 
 def export_branch(repo, branch):
@@ -479,14 +485,14 @@ def branch_tip(repo, branch):
 def get_branch_tip(repo, branch):
     global branches
 
-    heads = branches.get(branch, None)
+    heads = branches.get(hgref(branch), None)
     if not heads:
         return None
 
     # verify there's only one head
     if (len(heads) > 1):
         warn("Branch '%s' has more than one head, consider merging" % branch)
-        return branch_tip(repo, branch)
+        return branch_tip(repo, hgref(branch))
 
     return heads[0]
 
@@ -508,6 +514,7 @@ def list_head(repo, cur):
             head = 'master'
         bmarks[head] = node
 
+    head = gitref(head)
     print "@refs/heads/%s HEAD" % head
     g_head = (head, node)
 
@@ -529,15 +536,15 @@ def do_list(parser):
                 branches[branch] = heads
 
         for branch in branches:
-            print "? refs/heads/branches/%s" % branch
+            print "? refs/heads/branches/%s" % gitref(branch)
 
     for bmark in bmarks:
-        print "? refs/heads/%s" % bmark
+        print "? refs/heads/%s" % gitref(bmark)
 
     for tag, node in repo.tagslist():
         if tag == 'tip':
             continue
-        print "? refs/tags/%s" % tag
+        print "? refs/tags/%s" % gitref(tag)
 
     print
 
@@ -674,7 +681,8 @@ def parse_commit(parser):
 
     # Check if the ref is supposed to be a named branch
     if ref.startswith('refs/heads/branches/'):
-        extra['branch'] = ref[len('refs/heads/branches/'):]
+        branch = ref[len('refs/heads/branches/'):]
+        extra['branch'] = hgref(branch)
 
     if mode == 'hg':
         i = data.find('\n--HG--\n')
@@ -803,6 +811,7 @@ def do_export(parser):
             continue
         elif ref.startswith('refs/tags/'):
             tag = ref[len('refs/tags/'):]
+            tag = hgref(tag)
             author, msg = parsed_tags.get(tag, (None, None))
             if mode == 'git':
                 if not msg:
-- 
1.8.2.1.790.g4588561

[PATCH 14/16] remote-hg: small performance improvement

[Felipe Contreras]

Load previous manifest first as Mercurial does; for caching reasons.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index 6f4afd7..f80236b 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -230,8 +230,9 @@ def get_filechanges(repo, ctx, parent):
     added = set()
     removed = set()
 
-    cur = ctx.manifest()
+    # load earliest manifest first for caching reasons
     prev = repo[parent].manifest().copy()
+    cur = ctx.manifest()
 
     for fn in cur:
         if fn in prev:
-- 
1.8.2.1.790.g4588561

[PATCH 15/16] remote-hg: use marks instead of inlined files

[Felipe Contreras]

So that we can find already exported ones. We can never be 100% sure
that we already exported such data, due to mercurial design, it at least
sometimes we should detect them, and so should give ups some performance
boost.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 41 +++++++++++++++++++++++++++++-------
 1 file changed, 33 insertions(+), 8 deletions(-)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index f80236b..d0e552c 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -126,6 +126,10 @@ class Marks:
     def to_rev(self, mark):
         return self.rev_marks[mark]
 
+    def next_mark(self):
+        self.last_mark += 1
+        return self.last_mark
+
     def get_mark(self, rev):
         self.last_mark += 1
         self.marks[str(rev)] = self.last_mark
@@ -218,12 +222,29 @@ def fix_file_path(path):
         return path
     return os.path.relpath(path, '/')
 
-def export_file(fc):
-    d = fc.data()
-    path = fix_file_path(fc.path())
-    print "M %s inline %s" % (gitmode(fc.flags()), path)
-    print "data %d" % len(d)
-    print d
+def export_files(files):
+    global marks, filenodes
+
+    final = []
+    for f in files:
+        fid = node.hex(f.filenode())
+
+        if fid in filenodes:
+            mark = filenodes[fid]
+        else:
+            mark = marks.next_mark()
+            filenodes[fid] = mark
+            d = f.data()
+
+            print "blob"
+            print "mark :%u" % mark
+            print "data %d" % len(d)
+            print d
+
+        path = fix_file_path(f.path())
+        final.append((gitmode(f.flags()), mark, path))
+
+    return final
 
 def get_filechanges(repo, ctx, parent):
     modified = set()
@@ -413,6 +434,8 @@ def export_ref(repo, name, kind, head):
         if len(parents) == 0 and rev:
             print 'reset %s/%s' % (prefix, ename)
 
+        modified_final = export_files(c.filectx(f) for f in modified)
+
         print "commit %s/%s" % (prefix, ename)
         print "mark :%d" % (marks.get_mark(rev))
         print "author %s" % (author)
@@ -425,8 +448,8 @@ def export_ref(repo, name, kind, head):
             if len(parents) > 1:
                 print "merge :%s" % (rev_to_mark(parents[1]))
 
-        for f in modified:
-            export_file(c.filectx(f))
+        for f in modified_final:
+            print "M %s :%u %s" % f
         for f in removed:
             print "D %s" % (fix_file_path(f))
         print
@@ -878,6 +901,7 @@ def main(args):
     global peer, mode, bad_mail, bad_name
     global track_branches, force_push, is_tmp
     global parsed_tags
+    global filenodes
 
     alias = args[1]
     url = args[2]
@@ -921,6 +945,7 @@ def main(args):
     parsed_refs = {}
     marks = None
     parsed_tags = {}
+    filenodes = {}
 
     repo = get_repo(url, alias)
     prefix = 'refs/hg/%s' % alias
-- 
1.8.2.1.790.g4588561

[PATCH 16/16] remote-hg: strip extra newline

[Felipe Contreras]

There's no functional change since mercurial commit operation strips
that anyway, but that's no excuse for us not to do the right thing. So
let's be explicit about it.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/remote-helpers/git-remote-hg | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
index d0e552c..fda4199 100755
--- a/contrib/remote-helpers/git-remote-hg
+++ b/contrib/remote-helpers/git-remote-hg
@@ -652,6 +652,10 @@ def parse_commit(parser):
         if parser.check('merge'):
             die('octopus merges are not supported yet')
 
+    # fast-export adds an extra newline
+    if data[-1] == '\n':
+        data = data[:-1]
+
     files = {}
 
     for line in parser:
-- 
1.8.2.1.790.g4588561

Re: [PATCH 01/16] remote-helpers: avoid has_key

[Junio C Hamano]

Felipe Contreras <felipe.contreras@gmail.com> writes:

> From: Dusty Phillips <dusty@linux.ca>
>
> It is deprecated.
>
> [fc: do the same in remote-bzr]
>
> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
> ---

No sign-off by the author?

It does not matter for something so trivial like this that there is
no other way to write, but it is a good habit you should encourage
your contributors to acquire, so that you do not have to waste time
with "please sign off" when their next contribution that is more
substantial comes.

>  contrib/remote-helpers/git-remote-bzr | 2 +-
>  contrib/remote-helpers/git-remote-hg  | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/contrib/remote-helpers/git-remote-bzr b/contrib/remote-helpers/git-remote-bzr
> index aa7bc97..cc6609b 100755
> --- a/contrib/remote-helpers/git-remote-bzr
> +++ b/contrib/remote-helpers/git-remote-bzr
> @@ -94,7 +94,7 @@ class Marks:
>          return self.last_mark
>  
>      def is_marked(self, rev):
> -        return self.marks.has_key(rev)
> +        return str(rev) in self.marks
>  
>      def new_mark(self, rev, mark):
>          self.marks[rev] = mark
> diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
> index 5481331..2cd1996 100755
> --- a/contrib/remote-helpers/git-remote-hg
> +++ b/contrib/remote-helpers/git-remote-hg
> @@ -129,7 +129,7 @@ class Marks:
>          self.last_mark = mark
>  
>      def is_marked(self, rev):
> -        return self.marks.has_key(str(rev))
> +        return str(rev) in self.marks
>  
>      def get_tip(self, branch):
>          return self.tips.get(branch, 0)

Re: [PATCH 13/16] remote-hg: allow refs with spaces

[Junio C Hamano]

Felipe Contreras <felipe.contreras@gmail.com> writes:

> Mercurial supports them, Git doesn't.

Another important thing to note is that you represent a SP with
three underscores on our side, no?

> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
> ---
>  contrib/remote-helpers/git-remote-hg | 25 +++++++++++++++++--------
>  1 file changed, 17 insertions(+), 8 deletions(-)
>
> diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
> index dbb4091..6f4afd7 100755
> --- a/contrib/remote-helpers/git-remote-hg
> +++ b/contrib/remote-helpers/git-remote-hg
> @@ -75,6 +75,12 @@ def hgmode(mode):
>  def hghex(node):
>      return hg.node.hex(node)
>  
> +def hgref(ref):
> +    return ref.replace('___', ' ')
> +
> +def gitref(ref):
> +    return ref.replace(' ', '___')
> +
>  def get_config(config):
>      cmd = ['git', 'config', '--get', config]
>      process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
> @@ -437,10 +443,10 @@ def export_ref(repo, name, kind, head):
>      marks.set_tip(ename, rev)
>  
>  def export_tag(repo, tag):
> -    export_ref(repo, tag, 'tags', repo[tag])
> +    export_ref(repo, tag, 'tags', repo[hgref(tag)])
>  
>  def export_bookmark(repo, bmark):
> -    head = bmarks[bmark]
> +    head = bmarks[hgref(bmark)]
>      export_ref(repo, bmark, 'bookmarks', head)
>  
>  def export_branch(repo, branch):
> @@ -479,14 +485,14 @@ def branch_tip(repo, branch):
>  def get_branch_tip(repo, branch):
>      global branches
>  
> -    heads = branches.get(branch, None)
> +    heads = branches.get(hgref(branch), None)
>      if not heads:
>          return None
>  
>      # verify there's only one head
>      if (len(heads) > 1):
>          warn("Branch '%s' has more than one head, consider merging" % branch)
> -        return branch_tip(repo, branch)
> +        return branch_tip(repo, hgref(branch))
>  
>      return heads[0]
>  
> @@ -508,6 +514,7 @@ def list_head(repo, cur):
>              head = 'master'
>          bmarks[head] = node
>  
> +    head = gitref(head)
>      print "@refs/heads/%s HEAD" % head
>      g_head = (head, node)
>  
> @@ -529,15 +536,15 @@ def do_list(parser):
>                  branches[branch] = heads
>  
>          for branch in branches:
> -            print "? refs/heads/branches/%s" % branch
> +            print "? refs/heads/branches/%s" % gitref(branch)
>  
>      for bmark in bmarks:
> -        print "? refs/heads/%s" % bmark
> +        print "? refs/heads/%s" % gitref(bmark)
>  
>      for tag, node in repo.tagslist():
>          if tag == 'tip':
>              continue
> -        print "? refs/tags/%s" % tag
> +        print "? refs/tags/%s" % gitref(tag)
>  
>      print
>  
> @@ -674,7 +681,8 @@ def parse_commit(parser):
>  
>      # Check if the ref is supposed to be a named branch
>      if ref.startswith('refs/heads/branches/'):
> -        extra['branch'] = ref[len('refs/heads/branches/'):]
> +        branch = ref[len('refs/heads/branches/'):]
> +        extra['branch'] = hgref(branch)
>  
>      if mode == 'hg':
>          i = data.find('\n--HG--\n')
> @@ -803,6 +811,7 @@ def do_export(parser):
>              continue
>          elif ref.startswith('refs/tags/'):
>              tag = ref[len('refs/tags/'):]
> +            tag = hgref(tag)
>              author, msg = parsed_tags.get(tag, (None, None))
>              if mode == 'git':
>                  if not msg:

Re: [PATCH 15/16] remote-hg: use marks instead of inlined files

[Junio C Hamano]

Felipe Contreras <felipe.contreras@gmail.com> writes:

> So that we can find already exported ones. We can never be 100% sure
> that we already exported such data, due to mercurial design, it at least
> sometimes we should detect them, and so should give ups some performance

s/ups/us/ (will locally tweak).

> boost.
>
> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
> ---
>  contrib/remote-helpers/git-remote-hg | 41 +++++++++++++++++++++++++++++-------
>  1 file changed, 33 insertions(+), 8 deletions(-)
>
> diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg
> index f80236b..d0e552c 100755
> --- a/contrib/remote-helpers/git-remote-hg
> +++ b/contrib/remote-helpers/git-remote-hg
> @@ -126,6 +126,10 @@ class Marks:
>      def to_rev(self, mark):
>          return self.rev_marks[mark]
>  
> +    def next_mark(self):
> +        self.last_mark += 1
> +        return self.last_mark
> +
>      def get_mark(self, rev):
>          self.last_mark += 1
>          self.marks[str(rev)] = self.last_mark
> @@ -218,12 +222,29 @@ def fix_file_path(path):
>          return path
>      return os.path.relpath(path, '/')
>  
> -def export_file(fc):
> -    d = fc.data()
> -    path = fix_file_path(fc.path())
> -    print "M %s inline %s" % (gitmode(fc.flags()), path)
> -    print "data %d" % len(d)
> -    print d
> +def export_files(files):
> +    global marks, filenodes
> +
> +    final = []
> +    for f in files:
> +        fid = node.hex(f.filenode())
> +
> +        if fid in filenodes:
> +            mark = filenodes[fid]
> +        else:
> +            mark = marks.next_mark()
> +            filenodes[fid] = mark
> +            d = f.data()
> +
> +            print "blob"
> +            print "mark :%u" % mark
> +            print "data %d" % len(d)
> +            print d
> +
> +        path = fix_file_path(f.path())
> +        final.append((gitmode(f.flags()), mark, path))
> +
> +    return final
>  
>  def get_filechanges(repo, ctx, parent):
>      modified = set()
> @@ -413,6 +434,8 @@ def export_ref(repo, name, kind, head):
>          if len(parents) == 0 and rev:
>              print 'reset %s/%s' % (prefix, ename)
>  
> +        modified_final = export_files(c.filectx(f) for f in modified)
> +
>          print "commit %s/%s" % (prefix, ename)
>          print "mark :%d" % (marks.get_mark(rev))
>          print "author %s" % (author)
> @@ -425,8 +448,8 @@ def export_ref(repo, name, kind, head):
>              if len(parents) > 1:
>                  print "merge :%s" % (rev_to_mark(parents[1]))
>  
> -        for f in modified:
> -            export_file(c.filectx(f))
> +        for f in modified_final:
> +            print "M %s :%u %s" % f
>          for f in removed:
>              print "D %s" % (fix_file_path(f))
>          print
> @@ -878,6 +901,7 @@ def main(args):
>      global peer, mode, bad_mail, bad_name
>      global track_branches, force_push, is_tmp
>      global parsed_tags
> +    global filenodes
>  
>      alias = args[1]
>      url = args[2]
> @@ -921,6 +945,7 @@ def main(args):
>      parsed_refs = {}
>      marks = None
>      parsed_tags = {}
> +    filenodes = {}
>  
>      repo = get_repo(url, alias)
>      prefix = 'refs/hg/%s' % alias

Re: [PATCH 01/16] remote-helpers: avoid has_key

[Felipe Contreras]

On Mon, Apr 22, 2013 at 5:28 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>> From: Dusty Phillips <dusty@linux.ca>
>>
>> It is deprecated.
>>
>> [fc: do the same in remote-bzr]
>>
>> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
>> ---
>
> No sign-off by the author?
>
> It does not matter for something so trivial like this that there is
> no other way to write, but it is a good habit you should encourage
> your contributors to acquire, so that you do not have to waste time
> with "please sign off" when their next contribution that is more
> substantial comes.

He is not a contributor, he sent a few patches, and then immediately
started gitifyhg, I haven't heard a word from him.

Still, he was the original author, so...

-- 
Felipe Contreras

Re: [PATCH 13/16] remote-hg: allow refs with spaces

[Felipe Contreras]

On Mon, Apr 22, 2013 at 5:32 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com> writes:
>
>> Mercurial supports them, Git doesn't.
>
> Another important thing to note is that you represent a SP with
> three underscores on our side, no?

Probably, I don't think it's really important. Perhaps some users
would benefit from this information, but they would be better served
by a README, or a wiki, the commit message doesn't help them.

-- 
Felipe Contreras

Re: [PATCH 01/16] remote-helpers: avoid has_key

[Dusty Phillips]

On 04/22/2013 04:35 PM, Felipe Contreras wrote:
> On Mon, Apr 22, 2013 at 5:28 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Felipe Contreras <felipe.contreras@gmail.com> writes:
>>
>>> From: Dusty Phillips <dusty@linux.ca>
>>>
>>> It is deprecated.
>>>
>>> [fc: do the same in remote-bzr]
>>>
>>> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
>>> ---
>> No sign-off by the author?
>>
>> It does not matter for something so trivial like this that there is
>> no other way to write, but it is a good habit you should encourage
>> your contributors to acquire, so that you do not have to waste time
>> with "please sign off" when their next contribution that is more
>> substantial comes.
> He is not a contributor, he sent a few patches, and then immediately
> started gitifyhg, I haven't heard a word from him.
>
> Still, he was the original author, so...

Acknowledged that I will not likely be contributing any more patches. I 
apologize for the lack of signoffs on the pull requests; I wasn't 
actually aware that they were going into git mainline already. I will 
state for the record that all patches I submitted to Felipe's github 
fork of git in November and December, 2012 were authored by me and only 
me and I submitted them with the understanding and expectation that they 
would be licensed under the GPL.

Dusty

Re: [PATCH 13/16] remote-hg: allow refs with spaces

[Junio C Hamano]

Felipe Contreras <felipe.contreras@gmail.com> writes:

> On Mon, Apr 22, 2013 at 5:32 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Felipe Contreras <felipe.contreras@gmail.com> writes:
>>
>>> Mercurial supports them, Git doesn't.
>>
>> Another important thing to note is that you represent a SP with
>> three underscores on our side, no?
>
> Probably, I don't think it's really important. Perhaps some users
> would benefit from this information, but they would be better served
> by a README, or a wiki, the commit message doesn't help them.

True, but it helps a reviewer to notice the missing documentation
;-)