Re: [PATCH v2] t: use user-specific utf-8 locale for testing

["Torsten Bögershausen" <tboegi@web.de>]

This all looks good.
Some suggestions about the commit message are inline.

On Sun, Jun 06, 2021 at 11:33:16PM +0700, Đoàn Trần Công Danh wrote:
> In some test-cases, utf-8 locale is required. To find such locale,
> we're using the first available UTF-8 locale that returned by
> "locale -a".

Good explanation.
I think that in generaral "utf-8" as a specification/specifier is better
written as "UTF-8", with uppercase.
"utf-8" or utf8 may be used inside the code, depending on the language.

>
> Despite being required by POSIX, locale(1) is unavailable in some
> systems, e.g. Linux with musl libc.  Some of those systems support
> utf-8 locale out of the box.
This reads a little bit harsh (the first sentence) and it is not
fully clear which systems do what (the second sentence).
Or are 2 things mentioned - the locale(1) utility and the support
of one UTF-8 locale "out of the box" ?
Does Linux with musl libs support an UTF-8 locale, but not
the locale(1) untility ?
Git itself supports many systems, that are not POSIX compliant,
strictly speaking. But if avaliable, the functions defined in POSIX
are used, whenever available.

Could we write:
However, the locale(1) utility is unavailable on some systems,
e.g. Linux with musl libc.

>
> However, without "locale -a", we can't guess provided UTF-8 locale.
>
> Let's give users of those systems an option to have better test
> coverage.

Add a Makefile knob GIT_TEST_UTF8_LOCALE and activate it for linux-musl

>
> This change also rename t/lib-git-svn.sh:prepare_a_utf8_locale to
> prepare_utf8_locale, since we no longer prepare the variable named
> "a_utf8_locale" but set up a fallback value for GIT_TEST_UTF8_LOCALE
> instead.  The fallback will be LC_ALL, LANG environment variable,
> or the first utf-8 locale from output of "locale -a", in that order.

rename -> renames, may be drop "This change", like this ?
Rename t/lib-git-svn.sh:prepare_a_utf8_locale into prepare_utf8_locale,
since we no longer prepare the variable named "a_utf8_locale",
but set up a fallback value for GIT_TEST_UTF8_LOCALE instead.
The fallback will be LC_ALL, LANG environment variable,
or the first UTF-8 locale from output of "locale -a", in that order.


>
> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> ---
> Range-diff against v1:
> 1:  d242ce64c4 ! 1:  f299ae2239 t: use user-specific utf-8 locale for testing
>     @@ Commit message
>          Let's give users of those systems an option to have better test
>          coverage.
>
>     +    This change also rename t/lib-git-svn.sh:prepare_a_utf8_locale to
>     +    prepare_utf8_locale, since we no longer prepare the variable named
>     +    "a_utf8_locale" but set up a fallback value for GIT_TEST_UTF8_LOCALE
>     +    instead.  The fallback will be LC_ALL, LANG environment variable,
>     +    or the first utf-8 locale from output of "locale -a", in that order.
>     +
>          Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
>
>       ## Makefile ##
>     @@ Makefile: all::
>       # with a different indexfile format version.  If it isn't set the index
>       # file format used is index-v[23].
>       #
>     -+# Define GIT_TEST_UTF8_LOCALE to prefered utf-8 locale for testing.
>     -+# If it isn't set, use the first utf-8 locale returned by "locale -a".
>     ++# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing.
>     ++# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8
>     ++# locale returned by "locale -a".
>      +#
>       # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
>       #
>     @@ Makefile: ifdef GIT_TEST_CMP
>       	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
>       ifdef GIT_PERF_REPEAT_COUNT
>
>     + ## ci/lib.sh ##
>     +@@ ci/lib.sh: linux-musl)
>     + 	CC=gcc
>     + 	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
>     + 	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
>     ++	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
>     + 	;;
>     + esac
>     +
>     +
>       ## t/lib-git-svn.sh ##
>      @@ t/lib-git-svn.sh: start_svnserve () {
>       		 --listen-host 127.0.0.1 &
>     @@ t/lib-git-svn.sh: start_svnserve () {
>      -}')
>      -	if test -n "$a_utf8_locale"
>      +prepare_utf8_locale () {
>     -+	if test -z "$GIT_TEST_UTF8_LOCALE"
>     ++	if test -n "$GIT_TEST_UTF8_LOCALE"
>     ++	then
>     ++		: test_set_prereq UTF8
>     ++	elif test -n "${LC_ALL:-$LANG}"
>      +	then
>     ++		case "${LC_ALL:-$LANG}" in
>     ++		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
>     ++			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
>     ++			;;
>     ++		esac
>     ++	else
>      +		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
>      +		p
>      +		q
>
>  Makefile                                 |  7 +++++++
>  ci/lib.sh                                |  1 +
>  t/lib-git-svn.sh                         | 24 ++++++++++++++++++------
>  t/t9100-git-svn-basic.sh                 | 14 +++-----------
>  t/t9115-git-svn-dcommit-funky-renames.sh |  6 +++---
>  t/t9129-git-svn-i18n-commitencoding.sh   |  4 ++--
>  6 files changed, 34 insertions(+), 22 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index c3565fc0f8..502e0c9a81 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -398,6 +398,10 @@ all::
>  # with a different indexfile format version.  If it isn't set the index
>  # file format used is index-v[23].
>  #
> +# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing.
> +# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8
> +# locale returned by "locale -a".
> +#
>  # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
>  #
>  # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
> @@ -2801,6 +2805,9 @@ ifdef GIT_TEST_CMP
>  endif
>  ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT
>  	@echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@+
> +endif
> +ifdef GIT_TEST_UTF8_LOCALE
> +	@echo GIT_TEST_UTF8_LOCALE=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_UTF8_LOCALE)))'\' >>$@+
>  endif
>  	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
>  ifdef GIT_PERF_REPEAT_COUNT
> diff --git a/ci/lib.sh b/ci/lib.sh
> index d848c036c5..476c3f369f 100755
> --- a/ci/lib.sh
> +++ b/ci/lib.sh
> @@ -229,6 +229,7 @@ linux-musl)
>  	CC=gcc
>  	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
>  	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
> +	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
>  	;;
>  esac
>
> diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh
> index 547eb3c31a..83efc17661 100644
> --- a/t/lib-git-svn.sh
> +++ b/t/lib-git-svn.sh
> @@ -121,12 +121,24 @@ start_svnserve () {
>  		 --listen-host 127.0.0.1 &
>  }
>
> -prepare_a_utf8_locale () {
> -	a_utf8_locale=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
> -	p
> -	q
> -}')
> -	if test -n "$a_utf8_locale"
> +prepare_utf8_locale () {
> +	if test -n "$GIT_TEST_UTF8_LOCALE"
> +	then
> +		: test_set_prereq UTF8
> +	elif test -n "${LC_ALL:-$LANG}"
> +	then
> +		case "${LC_ALL:-$LANG}" in
> +		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
> +			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
> +			;;
> +		esac
> +	else
> +		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
> +		p
> +		q
> +	}')
> +	fi
> +	if test -n "$GIT_TEST_UTF8_LOCALE"
>  	then
>  		test_set_prereq UTF8
>  	else
> diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
> index 1d3fdcc997..d5563ec35f 100755
> --- a/t/t9100-git-svn-basic.sh
> +++ b/t/t9100-git-svn-basic.sh
> @@ -4,21 +4,13 @@
>  #
>
>  test_description='git svn basic tests'
> -GIT_SVN_LC_ALL=${LC_ALL:-$LANG}
>
>  GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
>  export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
>
>  . ./lib-git-svn.sh
>
> -case "$GIT_SVN_LC_ALL" in
> -*.UTF-8)
> -	test_set_prereq UTF8
> -	;;
> -*)
> -	say "# UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)"
> -	;;
> -esac
> +prepare_utf8_locale
>
>  test_expect_success 'git svn --version works anywhere' '
>  	nongit git svn --version
> @@ -187,8 +179,8 @@ test_expect_success POSIXPERM,SYMLINKS "$name" '
>  	test ! -h "$SVN_TREE"/exec-2.sh &&
>  	test_cmp help "$SVN_TREE"/exec-2.sh'
>
> -name="commit with UTF-8 message: locale: $GIT_SVN_LC_ALL"
> -LC_ALL="$GIT_SVN_LC_ALL"
> +name="commit with UTF-8 message: locale: $GIT_TEST_UTF8_LOCALE"
> +LC_ALL="$GIT_TEST_UTF8_LOCALE"
>  export LC_ALL
>  # This test relies on the previous test, hence requires POSIXPERM,SYMLINKS
>  test_expect_success UTF8,POSIXPERM,SYMLINKS "$name" "
> diff --git a/t/t9115-git-svn-dcommit-funky-renames.sh b/t/t9115-git-svn-dcommit-funky-renames.sh
> index 9b44a44bc1..743fbe1fe4 100755
> --- a/t/t9115-git-svn-dcommit-funky-renames.sh
> +++ b/t/t9115-git-svn-dcommit-funky-renames.sh
> @@ -93,9 +93,9 @@ test_expect_success 'git svn rebase works inside a fresh-cloned repository' '
>  # > ... All of the above characters, except for the backslash, are converted
>  # > to special UNICODE characters in the range 0xf000 to 0xf0ff (the
>  # > "Private use area") when creating or accessing files.
> -prepare_a_utf8_locale
> +prepare_utf8_locale
>  test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new file on dcommit' '
> -	LC_ALL=$a_utf8_locale &&
> +	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
>  	export LC_ALL &&
>  	neq=$(printf "\201\202") &&
>  	git config svn.pathnameencoding cp932 &&
> @@ -107,7 +107,7 @@ test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new
>
>  # See the comment on the above test for setting of LC_ALL.
>  test_expect_success !MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 rename on dcommit' '
> -	LC_ALL=$a_utf8_locale &&
> +	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
>  	export LC_ALL &&
>  	inf=$(printf "\201\207") &&
>  	git config svn.pathnameencoding cp932 &&
> diff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh
> index 2c213ae654..01e1e8a8f7 100755
> --- a/t/t9129-git-svn-i18n-commitencoding.sh
> +++ b/t/t9129-git-svn-i18n-commitencoding.sh
> @@ -14,12 +14,12 @@ compare_git_head_with () {
>  	test_cmp current "$1"
>  }
>
> -prepare_a_utf8_locale
> +prepare_utf8_locale
>
>  compare_svn_head_with () {
>  	# extract just the log message and strip out committer info.
>  	# don't use --limit here since svn 1.1.x doesn't have it,
> -	LC_ALL="$a_utf8_locale" svn log $(git svn info --url) | perl -w -e '
> +	LC_ALL="$GIT_TEST_UTF8_LOCALE" svn log $(git svn info --url) | perl -w -e '
>  		use bytes;
>  		$/ = ("-"x72) . "\n";
>  		my @x = <STDIN>;
> --
> 2.32.0.rc3.5.gf3d78db977
>