python3: use LC_CTYPE, not LC_ALL to set utf-8

python3: use LC_CTYPE, not LC_ALL to set utf-8

[Enrico Scholz]

Hello,

due to python3's misdesigned string handling, recent bitbake/OE enforces
'LC_ALL=en_US.utf-8'.  This causes a lot of problems:

* programs are giving out unicode quotes; e.g. look at the filenames in

  | env LC_ALL=en_US.utf-8 install foo bar

  (it is assmed that 'foo' does not exist).

* every non-C locale seems to have a strange LC_COLLATE behavior:

  - it mixes upper and lowercas ('aword' < 'Bword' < 'cword')

  - it mixes dotfiles and non dotfiles ('aword' < '.bword' < 'cword')

* almost everything in the en_US locale is different from the rest of
  the world (letter vs A4 paper, 12h vs 24h clock format, sunday instead
  of monday starts week, lb vs. kg, inch vs. cm)


I suggest to set only 'LC_CTYPE=en_US.utf-8' and request perhaaps
explicitly a language neutral behavior by 'LANG=C'.

I have

| export LC_CTYPE = "en_US.utf-8"
| LC_ALL[unexport] = "1"     # overrides bitbake.conf setting; might be removed

in my distribution.conf after removing some sanity checks in bitbake and
OE.



Enrico

Re: python3: use LC_CTYPE, not LC_ALL to set utf-8

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

On 8 August 2016 at 18:41, Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
wrote:

> due to python3's misdesigned string handling, recent bitbake/OE enforces
> 'LC_ALL=en_US.utf-8'.  This causes a lot of problems:
>

I can confirm that this combination solves the file encoding problem:

$ export LANG=C
$ python3 -c 'import sys; print(sys.getfilesystemencoding())'
ascii
$ export LC_CTYPE=en_US.UTF-8
$ python3 -c 'import sys; print(sys.getfilesystemencoding())'
utf-8

Ross

[-- Attachment #2: Type: text/html, Size: 1248 bytes --]