Re: [PULL] svn pathnameencoding for git svn dcommit

[Kazutoshi Satoda <k_satoda@f2.dion.ne.jp>]

[-- Attachment #1: Type: text/plain, Size: 3300 bytes --]

On 2016/02/16 15:33 +0900, Eric Wong wrote:
> Kazutoshi Satoda <k_satoda@f2.dion.ne.jp> wrote:
>> Thank you for the tests. But, on my environment, both of them failed
>> unexpectedly. (Windows 7 SP1, x86_64 Cygwin, LANG=ja_JP.UTF-8)
...
>> > Untracked files:
>> >         svnrepo/
>> >         "\357\202\201\357\202\202"
>> >         "\357\202\201\357\202\207"
...
>> I can't see how "\357\202\201\357\202\202" came as output in the test.
> 
> I wonder if it's a shell or printf portability problem.  I've
> repushed the branch against master and implemented the weird
> character use inside Perl scripts.
> 
> Can you give it a try?

(Shouldn't the branch be based on maint, as these are bugfixes?)

Thank you. I tried it but got similar problem:
> $ ./t9115-git-svn-dcommit-funky-renames.sh -x --run='11-12'
(snip)
> expecting success:
>         git config svn.pathnameencoding cp932 &&
>         "$PERL_PATH" -w "$TEST_DIRECTORY"/t9115/neq.perl &&
>         git commit -m "neq" &&
>         git svn dcommit
>
> ++ git config svn.pathnameencoding cp932
> ++ /usr/bin/perl -w /home/k_satoda/project/git/t/t9115/neq.perl
> ++ git commit -m neq
> On branch master
>
> Initial commit
>
> Untracked files:
>         svnrepo/
>         "\357\202\201\357\202\202"
>
> nothing added to commit but untracked files present
> error: last command exited with $?=1
> not ok 11 - svn.pathnameencoding=cp932 new file on dcommit
> #
> #               git config svn.pathnameencoding cp932 &&
> #               "$PERL_PATH" -w "$TEST_DIRECTORY"/t9115/neq.perl &&
> #               git commit -m "neq" &&
> #               git svn dcommit
> #
>
> expecting success:
>         "$PERL_PATH" -w "$TEST_DIRECTORY"/t9115/inf.perl
>
> ++ /usr/bin/perl -w /home/k_satoda/project/git/t/t9115/inf.perl
> On branch master
>
> Initial commit
>
> Untracked files:
>         svnrepo/
>         "\357\202\201\357\202\202"
>         "\357\202\201\357\202\207"
>
> nothing added to commit but untracked files present
> git commit -m inf: 256 at /home/k_satoda/project/git/t/t9115/inf.perl line 5.
> error: last command exited with $?=1
> not ok 12 - svn.pathnameencoding=cp932 rename on dcommit
> #
> #               "$PERL_PATH" -w "$TEST_DIRECTORY"/t9115/inf.perl
> #


I found how "\357\202\201\357\202\202" (U+F081 U+F082 in UTF-8) could
come.
https://cygwin.com/cygwin-ug-net/using-specialnames.html#pathnames-specialchars
> Some characters are disallowed in filenames on Windows filesystems. ...
...
> ... All of the above characters, except for the backslash, are converted
> to special UNICODE characters in the range 0xf000 to 0xf0ff (the
> "Private use area") when creating or accessing files.
"U+F081 U+F082" seems the result of conversion from "0x8182" (neq in
cp932) as treating each of 2 bytes as disallowed characters.

And I also noticed that LANG and LC_ALL is set to "C" in test-lib.sh.

Setting LC_ALL=C.UTF-8 in the test 11-12 made them pass on Cygwin.
Same change made the previous version also pass. Please find the patch
in the attached output of git format-patch.

Could you please test with this on non-Cygwin environment?

If it made no harm, please tell me what should I do to proceed this patch.
Will you (Eric) please make further integration? Shall I make another
series (v2) of patches?

-- 
k_satoda

[-- Attachment #2: 0001-Add-LC_ALL-C.UTF-8-in-t9115-git-svn-dcommit-funky-re.patch --]
[-- Type: text/plain, Size: 1808 bytes --]

>From 7b827f2d65aaa859030ba5b08055020f2bda1f0f Mon Sep 17 00:00:00 2001
From: Kazutoshi SATODA <k_satoda@f2.dion.ne.jp>
Date: Wed, 17 Feb 2016 00:29:24 +0900
Subject: [PATCH] Add LC_ALL=C.UTF-8 in t9115-git-svn-dcommit-funky-renames.sh

This makes the test 11-12 pass on Cygwin.
---
 t/t9115-git-svn-dcommit-funky-renames.sh | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/t/t9115-git-svn-dcommit-funky-renames.sh b/t/t9115-git-svn-dcommit-funky-renames.sh
index 9828f05..59d086b 100755
--- a/t/t9115-git-svn-dcommit-funky-renames.sh
+++ b/t/t9115-git-svn-dcommit-funky-renames.sh
@@ -84,7 +84,16 @@ test_expect_success 'git svn rebase works inside a fresh-cloned repository' '
 		test -e test-rebase
 	)'
 
+# Without this, LC_ALL=C as set in test-lib.sh, and Cygwin converts
+# non-ASCII characters in filenames unexpectedly, and causes errors.
+# https://cygwin.com/cygwin-ug-net/using-specialnames.html#pathnames-specialchars
+# > Some characters are disallowed in filenames on Windows filesystems. ...
+# ...
+# > ... All of the above characters, except for the backslash, are converted
+# > to special UNICODE characters in the range 0xf000 to 0xf0ff (the
+# > "Private use area") when creating or accessing files.
 test_expect_success 'svn.pathnameencoding=cp932 new file on dcommit' '
+	export LC_ALL=C.UTF-8 &&
 	neq=$(printf "\201\202") &&
 	git config svn.pathnameencoding cp932 &&
 	echo neq >"$neq" &&
@@ -93,7 +102,9 @@ test_expect_success 'svn.pathnameencoding=cp932 new file on dcommit' '
 	git svn dcommit
 '
 
+# See the comment on the above test for setting of LC_ALL.
 test_expect_success 'svn.pathnameencoding=cp932 rename on dcommit' '
+	export LC_ALL=C.UTF-8 &&
 	inf=$(printf "\201\207") &&
 	git config svn.pathnameencoding cp932 &&
 	echo inf >"$inf" &&
-- 
2.7.0