[PATCH V4] git on Mac OS and precomposed unicode

[PATCH V4] git on Mac OS and precomposed unicode

[Torsten Bögershausen]

Allow git on Mac OS to store file names in the index in precomposed unicode,
while the file system uses decomposed unicode.

The problem:
When a file called "LATIN CAPITAL LETTER A WITH DIAERESIS"
(in utf-8 encoded as 0xc3 0x84) is created, the Mac OS filesystem
converts "precomposed unicode" into "decomposed unicode".
This means that readdir() will return 0x41 0xcc 0x88.

Git under Mac OS reverts the unicode decomposition of filenames.

This is useful when pulling/pushing from repositories containing utf-8
encoded filenames using precomposed utf-8 like Linux or Windows (*).

It allows sharing git repositories stored on a VFAT file system
(e.g. a USB stick), and mounted network share using samba.

* (Not all Windows versions support UTF-8 yet:
   Msysgit needs the unicode branch, cygwin supports UTF-8 since 1.7)

A new confguration variable is added: "core.precomposedunicode"

If set to false, git behaves exactly as older versions of git.
When a new git version is installed and there is a repository
where the configuration "core.precomposedunicode" is not present,
the new git is backward compatible.

The code in compat/precomposed_utf8.c implements basically 4 new functions:
precomposed_utf8_opendir(), precomposed_utf8_readdir(),
precomposed_utf8_closedir() precompose_argv()

In order to prevent that ever a file name in decomposed unicode is entering
the index, a "brute force" attempt is taken:
all arguments into git (argv[1]..argv[n]) are converted into
precomposed unicode.
This is done in git.c by calling precompose_argv().
This function is actually a #define, and it is only defined under Mac OS.
Nothing is converted on any other OS.

Auto sensing:
When creating a new git repository with "git init" or "git clone",
"core.precomposedunicode" will be set "false".

The user needs to activate this feature manually.
She typically sets core.precomposedunicode to "true" on HFS and VFAT,
or file systems mounted via SAMBA onto a Linux box.

Signed-off-by: Torsten Bögershausen <tboegi@web.de>
---
 Documentation/config.txt     |    9 ++
 Makefile                     |    3 +
 builtin/init-db.c            |    2 +
 compat/precomposed_utf8.c    |  208 ++++++++++++++++++++++++++++++++++++++++++
 compat/precomposed_utf8.h    |   30 ++++++
 git-compat-util.h            |    9 ++
 git.c                        |    1 +
 t/t3910-mac-os-precompose.sh |  117 +++++++++++++++++++++++
 8 files changed, 379 insertions(+), 0 deletions(-)
 create mode 100644 compat/precomposed_utf8.c
 create mode 100644 compat/precomposed_utf8.h
 create mode 100755 t/t3910-mac-os-precompose.sh

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 2959390..29ba4b0 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -175,6 +175,15 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]
 will probe and set core.ignorecase true if appropriate when the repository
 is created.
 
+core.precomposedunicode::
+	This option is only used by Mac OS implementation of git.
+	When core.precomposedunicode=true,
+	git reverts the unicode decomposition of filenames done by Mac OS.
+	This is useful when pulling/pushing from repositories containing utf-8
+	encoded filenames using precomposed unicode (like Linux).
+	When false, file names are handled fully transparent by git.
+	If in doubt, keep core.precomposedunicode=false.
+
 core.trustctime::
 	If false, the ctime differences between the index and the
 	working tree are ignored; useful when the inode change time
diff --git a/Makefile b/Makefile
index b21d2f1..a912b45 100644
--- a/Makefile
+++ b/Makefile
@@ -519,6 +519,7 @@ LIB_H += compat/bswap.h
 LIB_H += compat/cygwin.h
 LIB_H += compat/mingw.h
 LIB_H += compat/obstack.h
+LIB_H += compat/precomposed_utf8.h
 LIB_H += compat/win32/pthread.h
 LIB_H += compat/win32/syslog.h
 LIB_H += compat/win32/poll.h
@@ -884,6 +885,8 @@ ifeq ($(uname_S),Darwin)
 	endif
 	NO_MEMMEM = YesPlease
 	USE_ST_TIMESPEC = YesPlease
+	COMPAT_OBJS += compat/precomposed_utf8.o
+	BASIC_CFLAGS += -DPRECOMPOSED_UNICODE
 endif
 ifeq ($(uname_S),SunOS)
 	NEEDS_SOCKET = YesPlease
diff --git a/builtin/init-db.c b/builtin/init-db.c
index 0dacb8b..06953df 100644
--- a/builtin/init-db.c
+++ b/builtin/init-db.c
@@ -290,6 +290,8 @@ static int create_default_files(const char *template_path)
 		strcpy(path + len, "CoNfIg");
 		if (!access(path, F_OK))
 			git_config_set("core.ignorecase", "true");
+
+		probe_utf8_pathname_composition(path, len);
 	}
 
 	return reinit;
diff --git a/compat/precomposed_utf8.c b/compat/precomposed_utf8.c
new file mode 100644
index 0000000..285fb45
--- /dev/null
+++ b/compat/precomposed_utf8.c
@@ -0,0 +1,208 @@
+#define __PRECOMPOSED_UNICODE_C__
+
+#include "../cache.h"
+#include "../utf8.h"
+#include <stdlib.h>
+#include <string.h>
+#include <stdio.h>
+#include <stdint.h>
+#include "precomposed_utf8.h"
+
+static int mac_os_precomposed_unicode;
+const static char *repo_encoding = "UTF-8";
+const static char *path_encoding = "UTF-8-MAC";
+
+
+/* Code borrowed from utf8.c */
+#if defined(OLD_ICONV) || (defined(__sun__) && !defined(_XPG6))
+	typedef const char * iconv_ibp;
+#else
+	typedef char * iconv_ibp;
+#endif
+
+static char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv)
+{
+	size_t outsz, outalloc;
+	char *out, *outpos;
+	iconv_ibp cp;
+
+	outsz = insz;
+	outalloc = outsz + 1; /* for terminating NUL */
+	out = xmalloc(outalloc);
+	outpos = out;
+	cp = (iconv_ibp)in;
+
+	while (1) {
+		size_t cnt = iconv(conv, &cp, &insz, &outpos, &outsz);
+
+		if (cnt == -1) {
+			size_t sofar;
+			if (errno != E2BIG) {
+				free(out);
+				return NULL;
+			}
+			/* insz has remaining number of bytes.
+			 * since we started outsz the same as insz,
+			 * it is likely that insz is not enough for
+			 * converting the rest.
+			 */
+			sofar = outpos - out;
+			outalloc = sofar + insz * 2 + 32;
+			out = xrealloc(out, outalloc);
+			outpos = out + sofar;
+			outsz = outalloc - sofar - 1;
+		}
+		else {
+			*outpos = '\0';
+			break;
+		}
+	}
+	return out;
+}
+
+static size_t has_utf8(const char *s, size_t maxlen, size_t *strlen_c)
+{
+	const uint8_t *utf8p = (const uint8_t*) s;
+	size_t strlen_chars = 0;
+	size_t ret = 0;
+
+	if ((!utf8p) || (!*utf8p))
+		return 0;
+
+	while((*utf8p) && maxlen) {
+		if (*utf8p & 0x80)
+			ret++;
+		strlen_chars++;
+		utf8p++;
+		maxlen--;
+	}
+	if (strlen_c)
+		*strlen_c = strlen_chars;
+
+	return ret;
+}
+
+
+void probe_utf8_pathname_composition(char *path, int len)
+{
+	const static char *auml_nfc = "\xc3\xa4";
+	const static char *auml_nfd = "\x61\xcc\x88";
+	int output_fd;
+	path[len] = 0;
+	strcpy(path + len, auml_nfc);
+	output_fd = open(path, O_CREAT|O_EXCL|O_RDWR, 0600);
+	if (output_fd >=0) {
+		close(output_fd);
+		path[len] = 0;
+		strcpy(path + len, auml_nfd);
+		if (0 == access(path, R_OK))
+			git_config_set("core.precomposedunicode", "false");
+		path[len] = 0;
+		strcpy(path + len, auml_nfc);
+		unlink(path);
+	}
+}
+
+
+static int precomposed_unicode_config(const char *var, const char *value, void *cb)
+{
+	if (!strcasecmp(var, "core.precomposedunicode")) {
+		mac_os_precomposed_unicode = git_config_bool(var, value);
+		return 0;
+	}
+	return 1;
+}
+
+void precompose_argv(int argc, const char **argv)
+{
+	int i = 0;
+	const char *oldarg;
+	char *newarg;
+	iconv_t ic_precompose;
+
+	git_config(precomposed_unicode_config, NULL);
+	if (!mac_os_precomposed_unicode)
+		return;
+
+	ic_precompose = iconv_open(repo_encoding, path_encoding);
+	if (ic_precompose == (iconv_t) -1)
+		return;
+
+	while (i < argc) {
+		size_t namelen;
+		oldarg = argv[i];
+		if (has_utf8(oldarg, (size_t)-1, &namelen)) {
+			newarg = reencode_string_iconv(oldarg, namelen, ic_precompose);
+			if (newarg)
+				argv[i] = newarg;
+		}
+		i++;
+	}
+	iconv_close(ic_precompose);
+}
+
+
+PRECOMPOSED_UTF_DIR * precomposed_utf8_opendir(const char *dirname)
+{
+	PRECOMPOSED_UTF_DIR *precomposed_utf8_dir;
+	precomposed_utf8_dir = xmalloc(sizeof(PRECOMPOSED_UTF_DIR));
+
+	precomposed_utf8_dir->dirp = opendir(dirname);
+	if (!precomposed_utf8_dir->dirp) {
+		free(precomposed_utf8_dir);
+		return NULL;
+	}
+	precomposed_utf8_dir->ic_precompose = iconv_open(repo_encoding, path_encoding);
+	if (precomposed_utf8_dir->ic_precompose == (iconv_t) -1) {
+		closedir(precomposed_utf8_dir->dirp);
+		free(precomposed_utf8_dir);
+		return NULL;
+	}
+
+	return precomposed_utf8_dir;
+}
+
+struct dirent * precomposed_utf8_readdir(PRECOMPOSED_UTF_DIR *precomposed_utf8_dirp)
+{
+	struct dirent *res;
+	size_t namelen = 0;
+
+	res = readdir(precomposed_utf8_dirp->dirp);
+	if (res && mac_os_precomposed_unicode && has_utf8(res->d_name, (size_t)-1, &namelen)) {
+		int ret_errno = errno;
+		size_t outsz = sizeof(precomposed_utf8_dirp->dirent_nfc.d_name) - 1; /* one for \0 */
+		char *outpos = precomposed_utf8_dirp->dirent_nfc.d_name;
+		iconv_ibp cp;
+		size_t cnt;
+		size_t insz = namelen;
+		cp = (iconv_ibp)res->d_name;
+
+		/* Copy all data except the name */
+		memcpy(&precomposed_utf8_dirp->dirent_nfc, res,
+		       sizeof(precomposed_utf8_dirp->dirent_nfc)-sizeof(precomposed_utf8_dirp->dirent_nfc.d_name));
+		errno = 0;
+
+		cnt = iconv(precomposed_utf8_dirp->ic_precompose, &cp, &insz, &outpos, &outsz);
+		if (cnt < sizeof(precomposed_utf8_dirp->dirent_nfc.d_name) -1) {
+			*outpos = 0;
+			errno = ret_errno;
+			return &precomposed_utf8_dirp->dirent_nfc;
+		}
+		errno = ret_errno;
+	}
+	return res;
+}
+
+
+int precomposed_utf8_closedir(PRECOMPOSED_UTF_DIR *precomposed_utf8_dirp)
+{
+	int ret_value;
+	int ret_errno;
+	ret_value = closedir(precomposed_utf8_dirp->dirp);
+	ret_errno = errno;
+	if (precomposed_utf8_dirp->ic_precompose != (iconv_t)-1)
+		iconv_close(precomposed_utf8_dirp->ic_precompose);
+	free(precomposed_utf8_dirp);
+	errno = ret_errno;
+	return ret_value;
+}
diff --git a/compat/precomposed_utf8.h b/compat/precomposed_utf8.h
new file mode 100644
index 0000000..79e65e7
--- /dev/null
+++ b/compat/precomposed_utf8.h
@@ -0,0 +1,30 @@
+#ifndef __PRECOMPOSED_UNICODE_H__
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <dirent.h>
+#include <iconv.h>
+
+
+typedef struct {
+	iconv_t ic_precompose;
+	DIR *dirp;
+	struct dirent dirent_nfc;
+} PRECOMPOSED_UTF_DIR;
+
+char *precompose_str(const char *in, iconv_t ic_precompose);
+void precompose_argv(int argc, const char **argv);
+void probe_utf8_pathname_composition(char *, int);
+
+PRECOMPOSED_UTF_DIR *precomposed_utf8_opendir(const char *dirname);
+struct dirent *precomposed_utf8_readdir(PRECOMPOSED_UTF_DIR *dirp);
+int precomposed_utf8_closedir(PRECOMPOSED_UTF_DIR *dirp);
+
+#ifndef __PRECOMPOSED_UNICODE_C__
+#define opendir(n) precomposed_utf8_opendir(n)
+#define readdir(d) precomposed_utf8_readdir(d)
+#define closedir(d) precomposed_utf8_closedir(d)
+#define DIR PRECOMPOSED_UTF_DIR
+#endif /* __PRECOMPOSED_UNICODE_C__ */
+
+#define  __PRECOMPOSED_UNICODE_H__
+#endif /* __PRECOMPOSED_UNICODE_H__ */
diff --git a/git-compat-util.h b/git-compat-util.h
index 230e198..8911743 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -153,6 +153,15 @@
 #include "compat/msvc.h"
 #endif
 
+/* used on Mac OS X */
+#ifdef PRECOMPOSED_UNICODE
+#include "compat/precomposed_utf8.h"
+#else
+#define precompose_str(in,i_nfd2nfc)
+#define precompose_argv(c,v)
+#define probe_utf8_pathname_composition(a,b)
+#endif
+
 #ifndef NO_LIBGEN_H
 #include <libgen.h>
 #else
diff --git a/git.c b/git.c
index 8e34903..265db96 100644
--- a/git.c
+++ b/git.c
@@ -298,6 +298,7 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)
 		    startup_info->have_repository) /* get_git_dir() may set up repo, avoid that */
 			trace_repo_setup(prefix);
 	}
+	precompose_argv(argc, argv);
 	commit_pager_choice();
 
 	if (!help && p->option & NEED_WORK_TREE)
diff --git a/t/t3910-mac-os-precompose.sh b/t/t3910-mac-os-precompose.sh
new file mode 100755
index 0000000..ba3d83c
--- /dev/null
+++ b/t/t3910-mac-os-precompose.sh
@@ -0,0 +1,117 @@
+#!/bin/sh
+#
+# Copyright (c) 2012 Torsten Bögershausen
+#
+
+test_description='utf-8 decomposed (nfd) converted to precomposed (nfc)'
+
+. ./test-lib.sh
+
+Adiarnfc=`printf '\303\204'`
+Odiarnfc=`printf '\303\226'`
+Adiarnfd=`printf 'A\314\210'`
+Odiarnfd=`printf 'O\314\210'`
+
+mkdir junk &&
+>junk/"$Adiarnfc" &&
+case "$(cd junk && echo *)" in
+	"$Adiarnfd")
+	test_nfd=1
+	;;
+	*)	;;
+esac
+rm -rf junk
+
+if test "$test_nfd"
+then
+	test_expect_success "detect if nfd needed" '
+		precomposedunicode=`git config  core.precomposedunicode` &&
+		test "$precomposedunicode" = false &&
+		git config  core.precomposedunicode true
+	'
+	test_expect_success "setup" '
+		>x &&
+		git add x &&
+		git commit -m "1st commit" &&
+		git rm x &&
+		git commit -m "rm x"
+	'
+	test_expect_success "setup case mac" '
+		git checkout -b mac_os
+	'
+	# This will test nfd2nfc in readdir()
+	test_expect_success "add file Adiarnfc" '
+		echo f.Adiarnfc >f.$Adiarnfc &&
+		git add f.$Adiarnfc &&
+		git commit -m "add f.$Adiarnfc"
+	'
+	# This will test nfd2nfc in git stage()
+	test_expect_success "stage file d.Adiarnfd/f.Adiarnfd" '
+		mkdir d.$Adiarnfd &&
+		echo d.$Adiarnfd/f.$Adiarnfd >d.$Adiarnfd/f.$Adiarnfd &&
+		git stage d.$Adiarnfd/f.$Adiarnfd &&
+		git commit -m "add d.$Adiarnfd/f.$Adiarnfd"
+	'
+	test_expect_success "add link Adiarnfc" '
+		ln -s d.$Adiarnfd/f.$Adiarnfd l.$Adiarnfc &&
+		git add l.$Adiarnfc &&
+		git commit -m "add l.Adiarnfc"
+	'
+	# This will test git log
+	test_expect_success "git log f.Adiar" '
+		git log f.$Adiarnfc > f.Adiarnfc.log &&
+		git log f.$Adiarnfd > f.Adiarnfd.log &&
+		test -s f.Adiarnfc.log &&
+		test -s f.Adiarnfd.log &&
+		test_cmp f.Adiarnfc.log f.Adiarnfd.log &&
+		rm f.Adiarnfc.log f.Adiarnfd.log
+	'
+	# This will test git ls-files
+	test_expect_success "git lsfiles f.Adiar" '
+		git ls-files f.$Adiarnfc > f.Adiarnfc.log &&
+		git ls-files f.$Adiarnfd > f.Adiarnfd.log &&
+		test -s f.Adiarnfc.log &&
+		test -s f.Adiarnfd.log &&
+		test_cmp f.Adiarnfc.log f.Adiarnfd.log &&
+		rm f.Adiarnfc.log f.Adiarnfd.log
+	'
+	# This will test git mv
+	test_expect_success "git mv" '
+		git mv f.$Adiarnfd f.$Odiarnfc &&
+		git mv d.$Adiarnfd d.$Odiarnfc &&
+		git mv l.$Adiarnfd l.$Odiarnfc &&
+		git commit -m "mv Adiarnfd Odiarnfc"
+	'
+	# Files can be checked out as nfc
+	# And the link has been corrected from nfd to nfc
+	test_expect_success "git checkout nfc" '
+		rm f.$Odiarnfc &&
+		git checkout f.$Odiarnfc
+	'
+	# Make it possible to checkout files with their NFD names
+	test_expect_success "git checkout file nfd" '
+		rm -f f.* &&
+		git checkout f.$Odiarnfd
+	'
+	# Make it possible to checkout links with their NFD names
+	test_expect_success "git checkout link nfd" '
+		rm l.* &&
+		git checkout l.$Odiarnfd
+	'
+	test_expect_success "setup case mac2" '
+		git checkout master &&
+		git reset --hard &&
+		git checkout -b mac_os_2
+	'
+	# This will test nfd2nfc in git commit
+	test_expect_success "commit file d2.Adiarnfd/f.Adiarnfd" '
+		mkdir d2.$Adiarnfd &&
+		echo d2.$Adiarnfd/f.$Adiarnfd >d2.$Adiarnfd/f.$Adiarnfd &&
+		git add d2.$Adiarnfd/f.$Adiarnfd &&
+		git commit -m "add d2.$Adiarnfd/f.$Adiarnfd" -- d2.$Adiarnfd/f.$Adiarnfd
+	'
+else
+	 say "Skipping nfc/nfd tests"
+fi
+
+test_done
-- 
1.7.8.rc0.43.gb49a8

Re: [PATCH V4] git on Mac OS and precomposed unicode

[Carlos Martín Nieto]

[-- Attachment #1: Type: text/plain, Size: 1131 bytes --]

On Sat, 2012-01-21 at 20:36 +0100, Torsten Bögershausen wrote:
> Allow git on Mac OS to store file names in the index in precomposed unicode,
> while the file system uses decomposed unicode.
> 
> The problem:
> When a file called "LATIN CAPITAL LETTER A WITH DIAERESIS"
> (in utf-8 encoded as 0xc3 0x84) is created, the Mac OS filesystem
> converts "precomposed unicode" into "decomposed unicode".
> This means that readdir() will return 0x41 0xcc 0x88.
> 
> Git under Mac OS reverts the unicode decomposition of filenames.
> 
> This is useful when pulling/pushing from repositories containing utf-8
> encoded filenames using precomposed utf-8 like Linux or Windows (*).
> 
> It allows sharing git repositories stored on a VFAT file system
> (e.g. a USB stick), and mounted network share using samba.
> 
> * (Not all Windows versions support UTF-8 yet:
>    Msysgit needs the unicode branch, cygwin supports UTF-8 since 1.7)

This might be overly pedantic, but Windows doesn't really deal with
UTF-8. To use Unicode you need to use the "wide" variant of the
functions, and those take UTF-16.

   cmn



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [PATCH V4] git on Mac OS and precomposed unicode

[Junio C Hamano]

[Pinging Nguyen who has worked rather extensively on the start-up sequence
for ideas.]

Torsten Bögershausen <tboegi@web.de> writes:

I'll try to reword the log message a bit below.

> When a file called "LATIN CAPITAL LETTER A WITH DIAERESIS" (in utf-8
> encoded as 0xc3 0x84) is created, the Mac OS filesystem converts
> "precomposed unicode" into "decomposed unicode". readdir() will return
> 0x41 0xcc 0x88 for such a file, that does not match what the caller
> thought it created.
>
> To work around this braindamage, allow git on Mac OS to optionally use a
> wrapper for readdir() that converts decomposed unicode back into the
> precomposed form, which most other platforms use natively. This makes it
> easier for Mac OS users to work together on the same project with people
> on other platforms (Note that not all Windows versions support UTF-8
> yet. Msysgit needs the unicode branch, cygwin supports UTF-8 since
> 1.7). This allows sharing git repositories stored on a VFAT file system
> (e.g. a USB stick), and mounted network share using samba.
>
> This new feature is controlled by setting a new configuration variable
> "core.precomposedunicode" to "true". Unless the variable is set to "true",
> Git on Mac OS behaves exactly as before, for backward compatiblity.
>
> The code in compat/precomposed_utf8.c implements basically 4 new
> functions: precomposed_utf8_opendir(), precomposed_utf8_readdir(),
> precomposed_utf8_closedir() precompose_argv()
>
> In order to prevent that ever a file name in decomposed unicode is
> entering the index, a "brute force" attempt is taken: all arguments into
> git (argv[1]..argv[n]) are converted into precomposed unicode.  This is
> done in git.c by calling precompose_argv().  This function is actually a
> #define, and it is only defined under Mac OS.  Nothing is converted on
> any other platforms.

It may be just me, but the above looks more in line with the usual style
of writing in our existing log messages.

Is this UTF-8 decomposition only an issue on HFS+, or does it happen on
any filesystem mounted on a MacOS box? If the former, then the second line
of the first paragraph needs further rephrasing, e.g. "... is created,
HFS+, the primary filesystem on the Mac OS, converts ...".

> Auto sensing:
> When creating a new git repository with "git init" or "git clone",
> "core.precomposedunicode" will be set "false".
>
> The user needs to activate this feature manually.
> She typically sets core.precomposedunicode to "true" on HFS and VFAT,
> or file systems mounted via SAMBA onto a Linux box.

I am not sure about this design decision.

I agree that it is prudent to introduce a new feature disabled by default,
and I can understand that you tried to make the feature more discoverable
by setting it explicitly to "false".

But I do not think it is a good idea. If a user is on MacOS and has only
HFS+, then it would be more convenient to have the configuration set to
true in $HOME/.gitconfig once and for all, to affect all repositories on
the box. "git init" dropping the explicit "false" to any new repositories
defeats that.

Wouldn't it make more sense if your "git init" did it this way?

    * Do not do anything, if you know core.precomposedunicode is already
      set (in /etc/gitconfig or $HOME/.gitconfig);

    * Otherwise, if the "probe" says "yes, we are on HFS+", issue an
      advice message to suggest the user to set it either in the
      repository specific .git/config or in $HOME/.gitconfig file.

> +core.precomposedunicode::
> +	This option is only used by Mac OS implementation of git.
> +	When core.precomposedunicode=true,
> +	git reverts the unicode decomposition of filenames done by Mac OS.
> +	This is useful when pulling/pushing from repositories containing utf-8
> +	encoded filenames using precomposed unicode (like Linux).

I would imagine that if the caller of creat(2) named the path in the
decomposed form, Mac OS would store it unaltered; strictly speaking, we
shouldn't say "reverts". How about:

    When set to true, pathnames in decomposed UTF-8 read from the
    filesystem are converted to precomposed UTF-8 before they are used by
    Git, to improve interoperability with other platforms.

> +void precompose_argv(int argc, const char **argv)
> +{
> +	int i = 0;
> +	const char *oldarg;
> +	char *newarg;
> +	iconv_t ic_precompose;
> +
> +	git_config(precomposed_unicode_config, NULL);

As the first thing called after main(), I still doubt this is a safe thing
to do (Pinging Nguyen who has worked rather extensively on the start-up
sequence for ideas). This is ifdefed away and will not break things on
other platforms, which may make it even harder to diagnose breakages.

Re: [PATCH V4] git on Mac OS and precomposed unicode

[Nguyen Thai Ngoc Duy]

On Sun, Jan 22, 2012 at 5:56 AM, Junio C Hamano <gitster@pobox.com> wrote:
> [Pinging Nguyen who has worked rather extensively on the start-up sequence
> for ideas.]
>
> Torsten Bögershausen <tboegi@web.de> writes:
>
> I'll try to reword the log message a bit below.
>
>> When a file called "LATIN CAPITAL LETTER A WITH DIAERESIS" (in utf-8
>> encoded as 0xc3 0x84) is created, the Mac OS filesystem converts
>> "precomposed unicode" into "decomposed unicode". readdir() will return
>> 0x41 0xcc 0x88 for such a file, that does not match what the caller
>> thought it created.
>>
>> To work around this braindamage, allow git on Mac OS to optionally use a
>> wrapper for readdir() that converts decomposed unicode back into the
>> precomposed form, which most other platforms use natively. This makes it
>> easier for Mac OS users to work together on the same project with people
>> on other platforms (Note that not all Windows versions support UTF-8
>> yet. Msysgit needs the unicode branch, cygwin supports UTF-8 since
>> 1.7). This allows sharing git repositories stored on a VFAT file system
>> (e.g. a USB stick), and mounted network share using samba.

I just have a quick look, you reencode opendir, readdir, and
closedir() to precomposed form. But files are still in decomposed
form, does open(<precomposed file>) work when only <decomposed file>
exists?

>> In order to prevent that ever a file name in decomposed unicode is
>> entering the index, a "brute force" attempt is taken: all arguments into
>> git (argv[1]..argv[n]) are converted into precomposed unicode.  This is
>> done in git.c by calling precompose_argv().  This function is actually a
>> #define, and it is only defined under Mac OS.  Nothing is converted on
>> any other platforms.

This is not entirely safe. Filenames can be taken from a file for
example (--stdin option or similar). Unless I'm mistaken, all file
names must enter git through the index, the conversion at read-cache.c
may be a better option.

>> Auto sensing:
>> When creating a new git repository with "git init" or "git clone",
>> "core.precomposedunicode" will be set "false".

This is a general comment on init auto detection feature. Perhaps we
should allow detection to be done when reinitializing a repo. Or at
least make an option to auto detect, print out new config values and
user can decide if they want to change current values themselves.

>> +void precompose_argv(int argc, const char **argv)
>> +{
>> +     int i = 0;
>> +     const char *oldarg;
>> +     char *newarg;
>> +     iconv_t ic_precompose;
>> +
>> +     git_config(precomposed_unicode_config, NULL);
>
> As the first thing called after main(), I still doubt this is a safe thing
> to do (Pinging Nguyen who has worked rather extensively on the start-up
> sequence for ideas). This is ifdefed away and will not break things on
> other platforms, which may make it even harder to diagnose breakages.

This can't be worse than current state of pager and alias settings,
which need to be detected even before setup_git_directory* is run.

I'd rather encode at index level and read_directory() than at argv[].
But if reencoding argv is the only feasible way, perhaps put the
conversion in parse_options()? Config reading is usually done by then,
and we can move precomposed config reading to git_default_config. If
some commands do not use parse_options yet, this is a good opportunity
to do so (I'll send pack-objects's parse_options() patch soon).
-- 
Duy

Re: [PATCH V4] git on Mac OS and precomposed unicode

[Nguyen Thai Ngoc Duy]

2012/1/22 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
>>> In order to prevent that ever a file name in decomposed unicode is
>>> entering the index, a "brute force" attempt is taken: all arguments into
>>> git (argv[1]..argv[n]) are converted into precomposed unicode.

Forgot one more thing. We have case-insensitive support in place
already, we can hook precomposed form conversion there before
comparing. In other words we just need to support
{pre,de}composed-insensitive string compare.
-- 
Duy

Re: [PATCH V4] git on Mac OS and precomposed unicode

[Torsten Bögershausen]

On 22.01.12 10:58, Nguyen Thai Ngoc Duy wrote:
> On Sun, Jan 22, 2012 at 5:56 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> [Pinging Nguyen who has worked rather extensively on the start-up sequence
>> for ideas.]
>>
[snip]
> 
> I just have a quick look, you reencode opendir, readdir, and
> closedir() to precomposed form. But files are still in decomposed
> form, does open(<precomposed file>) work when only <decomposed file>
> exists?

Yes. All function like stat(), lstat(), open(), fopen(), unlink() behave the same
for precomped or decomposed. This is similar to the ignore case feature.
And because the default HFS+ is case preserving, case insenstive and unicode decomposing
all at the same time, a file name "Ä" could be reached under 4 different names.
Please see the output of the test script:
(which is at the end of this email)

tests/Darwin_i386/NFC file name created as nfc is readable as nfd
tests/Darwin_i386/NFC readdir returns nfd but expected is nfc
tests/Darwin_i386/NFD file name created as nfd is readable as nfc
tests/Darwin_i386/NFCNFD 1 file found in directory, but there should be 2
tests/Darwin_i386/NFCNFD nfc is missing, nfd is present
tests/Darwin_i386/NFCNFD nfc File content overwritten by nfd
tests/Darwin_i386/NFDNFC 1 file found in directory, but there should be 2
tests/Darwin_i386/NFDNFC nfc is missing, nfd is present
tests/Darwin_i386/NFDNFC nfd File content overwritten by nfc


> 
>>> In order to prevent that ever a file name in decomposed unicode is
>>> entering the index, a "brute force" attempt is taken: all arguments into
>>> git (argv[1]..argv[n]) are converted into precomposed unicode.  This is
>>> done in git.c by calling precompose_argv().  This function is actually a
>>> #define, and it is only defined under Mac OS.  Nothing is converted on
>>> any other platforms.
> 
> This is not entirely safe. Filenames can be taken from a file for
> example (--stdin option or similar). Unless I'm mistaken, all file
> names must enter git through the index, the conversion at read-cache.c
> may be a better option.
Good point, thanks. 
I added some code to read-cache.c, and it works for files, but not for directories.
I looked through the code for "case-ignoring" directory names, and couldn't
find something obvious. More work is to be done.
 

[snip]
> I'd rather encode at index level and read_directory() than at argv[].
>But if reencoding argv is the only feasible way, perhaps put the
>conversion in parse_options()?

I tried that, and found that git-lsfiles.c doesn't use parse_options.

[snip]

On the long run I want to get rid of the argv[] conversion completely,
but I'm not there yet.

Thanks for all comments and inspiration!

(and apologies for my long response times I use to have)
/Torsten



PS:
Here the script.
Mac OS writes decomposd unicode to HFS+, precomposed unicode to VFAT and SAMBA.
In any case readdir() returns decomposed.

=================
#!/bin/sh
errorandout() {
  echo Error: The shell can not handle nfd
  echo try to run /bin/bash $0
  rm -rf $DIR
  exit 1
}

checkDirNfcOrNfd() {
  DDNFCNFD=$1
  readdirexp=$2
  if test -r $DDNFCNFD/$aumlnfc; then
    x=`cat $DDNFCNFD/$aumlnfc`
    if test "$x" = nfd; then
      echo $DDNFCNFD file name created as nfd is readable as nfc
    fi
  fi
  if test -r $DDNFCNFD/$aumlnfd; then
    x=`cat $DDNFCNFD/$aumlnfd 2>/dev/null` || {
      echo $DDNFCNFD nfd is not readable, but readdir says that it exist
    }
    if test "$x" = nfc; then
      echo $DDNFCNFD file name created as nfc is readable as nfd
    fi
  fi
  readdirres=`echo $DDNFCNFD/*`
  if test "$readdirres" != "$DDNFCNFD/$readdirexp"; then
    if test "$readdirres" = $DDNFCNFD/$aumlnfd; then
      echo $DDNFCNFD readdir returns nfd but expected is nfc
    fi
    if test "$readdirres" = $DDNFCNFD/$aumlnfc; then
      echo $DDNFCNFD readdir returns nfc but expected is nfd
    fi
  fi
}

checkdirnfcnfd() {
  DDNFCNFD=$1
  if test `ls -1 $DDNFCNFD | wc -l` != 2; then
    if test `ls -1 $DDNFCNFD | wc -l` == 1; then
      echo $DDNFCNFD 1 file found in directory, but there should be 2
    else
      echo $DDNFCNFD 2 files should be in directory
    fi  
  fi

  x=`echo $DDNFCNFD/*`
  a=`echo $DDNFCNFD/$aumlnfd $DDNFCNFD/$aumlnfc`
  b=`echo $DDNFCNFD/$aumlnfc $DDNFCNFD/$aumlnfd`
  c=`echo $DDNFCNFD/$aumlnfc $DDNFCNFD/$aumlnfc`
  d=`echo $DDNFCNFD/$aumlnfd $DDNFCNFD/$aumlnfd`
  e=`echo $DDNFCNFD/$aumlnfc`
  f=`echo $DDNFCNFD/$aumlnfd`
  case "$x" in
    $a)
    ;;      
    $b)
    ;;
    $c)
    echo $DDNFCNFD nfd is hidden, nfc is listed twice
    ;;
    $d)
    echo $DDNFCNFD nfc is hidden, nfd is listed twice
    ;;
    $e)
    echo $DDNFCNFD nfd is missing, nfc is present
    ;;      
    $f)
    echo $DDNFCNFD nfc is missing, nfd is present
    ;;      
    *)
    echo $DDNFCNFD x`echo $x | xxd`
    ;;
  esac

  if ! test -r $DDNFCNFD/$aumlnfc; then
    echo $DDNFCNFD/nfc File does not exist
  else
    x=`cat $DDNFCNFD/$aumlnfc`
    if test "$x" != nfc; then
      echo $DDNFCNFD nfc File content overwritten by $x
    fi
  fi
  
  if ! test -r $DDNFCNFD/$aumlnfd; then
    echo $DDNFCNFD/nfd File does not exist
  else
    x=`cat $DDNFCNFD/$aumlnfd`
    if test "$x" != nfd; then
      echo $DDNFCNFD nfd File content overwritten by $x
    fi
  fi
}


aumlnfc=$(printf '\303\204')
aumlnfd=$(printf '\101\314\210')

DIR=tests/`uname -s`_`uname -m`
echo "DIR=$DIR"

rm -rf $DIR/NFC &&
rm -rf $DIR/NFD &&
rm -rf $DIR/NFCNFD &&
rm -rf $DIR/NFDNFC &&
mkdir -p $DIR/NFC &&
mkdir -p $DIR/NFD &&
mkdir -p $DIR/NFDNFC &&
mkdir -p $DIR/NFCNFD &&
echo nfc > $DIR/NFC/$aumlnfc &&
echo nfd > $DIR/NFD/$aumlnfd &&
echo nfd > $DIR/NFDNFC/$aumlnfd &&
echo nfc > $DIR/NFDNFC/$aumlnfc &&
echo nfc > $DIR/NFCNFD/$aumlnfc &&
echo nfd > $DIR/NFCNFD/$aumlnfd && {
    # test 1: basic if the shell handles nfd
    if ! test -r $DIR/NFD/$aumlnfd; then
      errorandout
    fi

  for DD in tests/*; do
    checkDirNfcOrNfd $DD/NFC  $aumlnfc
    checkDirNfcOrNfd $DD/NFD  $aumlnfd

    checkdirnfcnfd $DD/NFCNFD
    checkdirnfcnfd $DD/NFDNFC
  done
} || errorandout

Re: [PATCH V4] git on Mac OS and precomposed unicode

[Torsten Bögershausen]

> I tried that, and found that git-lsfiles.c doesn't use parse_options.
Oops, I shouldn't have written that: git-lsfiles uses parse_options.

Sorry for the noise.
/torsten

Re: [PATCH V4] git on Mac OS and precomposed unicode

[Erik Faye-Lund]

On Sat, Jan 21, 2012 at 11:28 PM, Carlos Martín Nieto <cmn@elego.de> wrote:
> On Sat, 2012-01-21 at 20:36 +0100, Torsten Bögershausen wrote:
>> * (Not all Windows versions support UTF-8 yet:
>>    Msysgit needs the unicode branch, cygwin supports UTF-8 since 1.7)
>
> This might be overly pedantic, but Windows doesn't really deal with
> UTF-8. To use Unicode you need to use the "wide" variant of the
> functions, and those take UTF-16.

This is exactly what the 'unicode'-branch in msysGit does, so the
comment is not incorrect at all.

Re: [PATCH V4] git on Mac OS and precomposed unicode

[Torsten Bögershausen]

On 22.01.12 11:03, Nguyen Thai Ngoc Duy wrote:
> 2012/1/22 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
>>>> In order to prevent that ever a file name in decomposed unicode is
>>>> entering the index, a "brute force" attempt is taken: all arguments into
>>>> git (argv[1]..argv[n]) are converted into precomposed unicode.
> 
> Forgot one more thing. We have case-insensitive support in place
> already, we can hook precomposed form conversion there before
> comparing. In other words we just need to support
> {pre,de}composed-insensitive string compare.
> 
>Forgot one more thing. We have case-insensitive support in place
>already, we can hook precomposed form conversion there before
>comparing. In other words we just need to support
>{pre,de}composed-insensitive string compare.
>-- Duy 

Yes, I like that idea.
After doing some experiments with precomposed and decomposed file names,
the motvation for the fix, or so to say the root cause, changed.

The Mac OS X file system on VFAT has a kind of schizophrenia:
- unicode file names are written as precomposed onto the disk
  and Linux/Windows see them as precomposed.
- readdir() returns always decomposed.
- open()/fopen() stat() lstat() works for both pre- and decomposed

Therefore a repository on VFAT under Mac OS X looks as follows:

- file names on disk are precomposed
- file names in the index are decomposed

As long as only Mac OS uses that device, there is no problem.

When we now move the e.g. USB stick using VFAT to Linux
the "decomposed" in the index seem to be deleted and the
precomposed on disk are untracked.
A complete desaster.

To keep the file names in the index and how they are stored
on disk the same, we can can set "core.precomposedunicode" = true.
(Side note: should we rename it to "i18n.precomposedunicode" ?)

Now we keep the index and file names on disk the same, and can move the
USB stick between Linux, Mac OS X, or Windows (since msysGit-1.7.10-
now called Git for Windows, or git under cygwin 1.7)

The same problem occurs when Mac OS mounts a network share from linux
using SAMBA:
readdir() returns decomposed, creat() stores file names in precomposed
on the remote linux machine.

If we put a file name with decomposed unicode on the linux machine,
it will be listed as decomposed by readdir() on the Mac OS side.
Trying to access this file failes, because Mac OS tries to open
the precomposed version, and that does not exist.


Another thing:
There are some reasons to avoid decomposed unicode in Linux and Windows:
Many user space programs don't handle decomposed unicode very well.
When e.g. an "û" should be displayed, the output looks like "u^" in many
programs.
And if we need more motivations: decomposed unicode is hard to enter on
the keyboard.



Then I went back to my original problem 
(versioning the "Documents" folder on my Linux $HOME under git
and access it from Windows and Mac OS using SAMBA, or cloning it
to a laptop...)

Knowing that
a) Mac OS X handles precomposed and decomposed the same in open()...
b) The user space program on Mac OS handle precomposed just fine
c) Many user space programs don't presentate decomposed as it should be
d) It is hard to enter decomposed unicode at the keyboard
e) and therefore decomposed unicode is seldom used on Linux
f) Mac OS using SAMBA puts file names in precomposed unicode on the 
   remote side

Do we have a motivation for pushing a solution that ignores 
the unicode composition ?

I'll send a V5 version with hopefully a better motivation

Re: [PATCH V4] git on Mac OS and precomposed unicode

[Robin Rosenberg]

Torsten Bögershausen skrev 2012-06-24 17.47:

> Do we have a motivation for pushing a solution that ignores
> the unicode composition ?

I say we do. I tried your patch and it worked fine.

> I'll send a V5 version with hopefully a better motivation

-- robin

Re: [PATCH V4] git on Mac OS and precomposed unicode

[Junio C Hamano]

Torsten Bögershausen <tboegi@web.de> writes:

> Changes since V3:
>
> - Be much more defensive:
>   Do not set core.precomposedunicode=true, even if the file system
>   probing indicates that it could be true.
>   (but we don't say that it should be true).
>   This is to keep better backward compatibility within git.
>   The user needs to manually enable the precomposition.
>   However, the .git/config indicates that we have a new
>   configuration variable, and will encourage people to use it.
>
> - compat/precomposed_utf8.c: 
>     - re-order #includes
>     - Added some empty lines to make code easier to read. 
>     - Small fixes (xmalloc, errno handling)
>
> - Make function names more consistent by renaming these functions:
>     argv_precompose() -> precompose_argv()
>     str_precompose() -> precompose_str()
> - Improved the commit message

Next time around, please place these after --- lines in the patch message
itself, without making a single-patch series with a separate cover letter.

Thanks.

[PATCH V4] git on Mac OS and precomposed unicode

[Torsten Bögershausen]

Changes since V3:

- Be much more defensive:
  Do not set core.precomposedunicode=true, even if the file system
  probing indicates that it could be true.
  (but we don't say that it should be true).
  This is to keep better backward compatibility within git.
  The user needs to manually enable the precomposition.
  However, the .git/config indicates that we have a new
  configuration variable, and will encourage people to use it.

- compat/precomposed_utf8.c: 
    - re-order #includes
    - Added some empty lines to make code easier to read. 
    - Small fixes (xmalloc, errno handling)

- Make function names more consistent by renaming these functions:
    argv_precompose() -> precompose_argv()
    str_precompose() -> precompose_str()
- Improved the commit message


Torsten Bögershausen (1):
  git on Mac OS and precomposed unicode

 Documentation/config.txt     |    9 ++
 Makefile                     |    3 +
 builtin/init-db.c            |    2 +
 compat/precomposed_utf8.c    |  208 ++++++++++++++++++++++++++++++++++++++++++
 compat/precomposed_utf8.h    |   30 ++++++
 git-compat-util.h            |    9 ++
 git.c                        |    1 +
 t/t3910-mac-os-precompose.sh |  117 +++++++++++++++++++++++
 8 files changed, 379 insertions(+), 0 deletions(-)
 create mode 100644 compat/precomposed_utf8.c
 create mode 100644 compat/precomposed_utf8.h
 create mode 100755 t/t3910-mac-os-precompose.sh

-- 
1.7.8.rc0.43.gb49a8