Re: [PATCH/RFC] Gitweb: Convert UTF-8 encoded file names

["Jakub Narębski" <jnareb@gmail.com>]

W dniu 2014-05-16 19:05, Junio C Hamano pisze:
> Jakub Narębski <jnareb@gmail.com> writes:
> 
>>> Correct, but is "where does it appear" the question we are
>>> primarily interested in, wrt this breakage and its fix?
>>
>> That of course depends on how we want to test gitweb output.
>> The simplest solution, comparing with known output with perhaps
>> fragile / variable elements masked out could be done quickly...
>> but changes in output (even if they don't change functionality,
>> or don't change visible output) require regenerating test cases
>> (expected output) to test against - which might be source of
>> errors in test suite.
> 
> I agree with your "to test it fully, we need extra dependencies",
> but my point is that it does not have to be a full "HTML-validating,
> picking the expected attribute via XPATH matching" kind of test if
> what we want is only to add a new test to protect this particular
> fix from future breakages.
> 
> For example, I think it is sufficient to grep for 'href="...%xx%xx"'
> in the output after preparing a sample tree with one entry to show.
> The expected substring either exists (in which case we got it
> right), or it doesn't (in which case we are showing garbage).  Of
> course that depends on the assumption that its output is not too
> heavily contaminated with volatile parts outside our control, as I
> already mentioned in the message you are responding to.
> 
> But it all depends on "if" we wanted to add a new test ;-)

I tried to add such simple test to t9502, but instead of tests
failing with current version, the test setup fails but succeeds
(i.e. test library says that it failed, but manual examination
shows that everything is O.K.).

-- >8 --
From: Jakub Narebski <jnareb@gmail.com>
Subject: [PATCH/RFC] gitweb test: Test proper encoding of non US-ASCII filenames in output (WIP)

This t9502 test is intended to test for proper encoding of non
US-ASCII filenames (i.e. UTF-8 filenames) in generated links (which
need some form of URI encoding) and in generated HTML (which needs
HTML encoding / escaping).

For now it tests only 'tree' view (though incidentally it also tests
UTF-8 in commit subject), as this was the action where reportedly
there was bug in link encoding: $t{'name'} coming from the
"git ls-tree -z ..." command via @ntries array was not marked as
UTF-8, making Perl assume that it is in internal Perl format
i.e. iso-8859-1 encoding and URI-escaping it as if it was in
iso-8859-1 encoding (e.g. "Gütekriterien.txt" in UTF-8 is
"Gütekriterien.txt" if treated as iso-8859-1, and it then
encodes to "G%C3%83%C2%BCtekriterien.txt" instead of correct
"G%C3%BCtekriterien.txt").

UNFORTUNATELY test does not fail as it should, even though the issue
was not fixed... OTOH it fails in setup though it is successful.

Reported-by: Michael Wagner <accounts@mwagner.org>
Signed-off-by: Jakub Narębski <jnareb@gmail.com>
---
 t/t9502-gitweb-standalone-parse-output.sh |   34 +++++++++++++++++++++++++++++
 1 files changed, 34 insertions(+), 0 deletions(-)

diff --git a/t/t9502-gitweb-standalone-parse-output.sh b/t/t9502-gitweb-standalone-parse-output.sh
index 86dfee2..37246a3 100755
--- a/t/t9502-gitweb-standalone-parse-output.sh
+++ b/t/t9502-gitweb-standalone-parse-output.sh
@@ -201,4 +201,38 @@ test_expect_success 'xss checks' '
 	xss "a=rss&p=foo.git&f=$TAG"
 '
 
+link_check () {
+	grep -F   "%3C__%C2%A3%C3%A5%C3%AB%C3%AE%C3%B1%C3%B2%C3%BB%C3%BD%C2%B6" \
+		gitweb.body &&
+	! grep -F "%3C__%A3%E5%EB%EE%F1%F2%FB%FD%B6" \
+		gitweb.body
+}
+
+test_expect_success 'prepare UTF-8 output tests' '
+	FILENAME="<__£åëîñòûý¶  +;?&__>" &&
+	test_commit "Adding $FILENAME" "$FILENAME" "$FILENAME contents"
+'
+
+test_expect_success 'check URI-escaped UTF-8 filename in query-params link' '
+	cat >>gitweb_config.perl <<-\EOF &&
+	$feature{"pathinfo"}{"default"} = [0];
+	EOF
+	gitweb_run "p=.git;a=tree" &&
+	link_check
+'
+
+test_expect_success 'check URI-escaped UTF-8 filename in path_info link' '
+	cat >>gitweb_config.perl <<-\EOF &&
+	$feature{"pathinfo"}{"default"} = [1];
+	EOF
+	gitweb_run "" "/.git/tree" &&
+	link_check
+'
+
+test_expect_success 'check HTML-escaped UTF-8 filename in body' '
+	gitweb_run "p=.git;a=tree" &&
+	grep -F "&lt;__£åëîñòûý¶  +;?&amp;__&gt;" gitweb.body &&
+	! grep -F  "<__£åëîñòûý¶  +;?&__>" gitweb.body
+'
+
 test_done
-- 
1.7.1