Re: Sphinx parallel build error: UnicodeEncodeError: 'latin-1' codec can't encode characters in position 18-20: ordinal not in range(256)

[Mauro Carvalho Chehab <mchehab@kernel.org>]

Em Wed, 12 May 2021 09:01:57 +0200
Michal Suchánek <msuchanek@suse.de> escreveu:

> On Wed, May 12, 2021 at 08:22:38AM +0200, Mauro Carvalho Chehab wrote:
> > Hi Michal,
> > 
> > Em Thu, 6 May 2021 19:48:49 +0200
> > Michal Suchánek <msuchanek@suse.de> escreveu:
> >   
> > > [  127s] + :
> > > [  127s] + locale
> > > [  128s] LANG=en_US
> > > [  128s] LC_CTYPE="en_US"
> > > [  128s] LC_NUMERIC="en_US"
> > > [  128s] LC_TIME="en_US"
> > > [  128s] LC_COLLATE="en_US"
> > > [  128s] LC_MONETARY="en_US"
> > > [  128s] LC_MESSAGES="en_US"
> > > [  128s] LC_PAPER="en_US"
> > > [  128s] LC_NAME="en_US"
> > > [  128s] LC_ADDRESS="en_US"
> > > [  128s] LC_TELEPHONE="en_US"
> > > [  128s] LC_MEASUREMENT="en_US"
> > > [  128s] LC_IDENTIFICATION="en_US"
> > > [  128s] LC_ALL=
> > > [  128s] + echo LC_ALL=
> > > [  128s] LC_ALL=
> > > [  128s] + echo LANG=en_US
> > > [  128s] LANG=en_US  
> > 
> > Where those the locale settings that you used when the build
> > failed?
> > 
> > I tried to reproduce the bug here with, disabling the parallel run (as
> > it masks the real error) with both:
> > 
> > 	$ for i in LANG LC_ALL LC_ADDRESS LC_IDENTIFICATION LC_MEASUREMENT LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_TIME; do echo $i=en_US; done
> > 	$ make cleandocs && make SPHINXOPTS=-j1 htmldocs
> > 
> > (this one caused lots of warnings on Debian, due to the
> >  settings at /etc/locale.gen)
> > 
> > and:
> > 
> > 	$ for i in LANG LC_ALL LC_ADDRESS LC_IDENTIFICATION LC_MEASUREMENT LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_TIME; do echo $i=en_US.ISO-8859-1; done
> > 	$ make cleandocs && make SPHINXOPTS=-j1 htmldocs
> > 
> > Without any success.
> > 
> > Could you please provide more details about the build VM and the git 
> > changeset that caused the issue?  
> 
> It depends on what character set your en_US locale implements.
> 
> ~> cat test.py   
> print("↑ᛏ个")
> ~> locale  
> LANG=en_US.utf8
> LC_CTYPE="en_US.utf8"
> LC_NUMERIC="en_US.utf8"
> LC_TIME="en_US.utf8"
> LC_COLLATE="en_US.utf8"
> LC_MONETARY="en_US.utf8"
> LC_MESSAGES="en_US.utf8"
> LC_PAPER="en_US.utf8"
> LC_NAME="en_US.utf8"
> LC_ADDRESS="en_US.utf8"
> LC_TELEPHONE="en_US.utf8"
> LC_MEASUREMENT="en_US.utf8"
> LC_IDENTIFICATION="en_US.utf8"
> LC_ALL=
> ~> python3 test.py   
> ↑ᛏ个
> ~> LANG=en_US python3 test.py   
> Traceback (most recent call last):
>   File "test.py", line 1, in <module>
>     print("\u2191\u16cf\u4e2a\uf8f9")
> UnicodeEncodeError: 'latin-1' codec can't encode characters in position 0-3: ordinal not in range(256)
> ~> LANG=C python3 test.py   
> ↑ᛏ个
> 

This is working as expected on my test machine:

	$ LANG=en_US.utf8 python3 test.py
	↑ᛏ个
	$ LANG=en_US python3 test.py
	Traceback (most recent call last):
	  File "test.py", line 1, in <module>
	    print("\u2191\u16cf\u4e2a\uf8f9")
	UnicodeEncodeError: 'latin-1' codec can't encode characters in position 0-3: ordinal not in range(256)

Yet, running:

	$ . /devel/v4l/docs/sphinx_3.3.1/bin/activate
	make cleandocs && LANG=en_US make SPHINXOPTS=-j1 htmldocs

Doesn't produce any UnicodeEncodeError errors.

See, here I'm testing it with Sphinx version 3.3.1, on Ubuntu 20.04,
using changeset 9f4ad9e425a1 Linux 5.12. Also, both UTF8 and iso8859-1
are on this machine's locale:

	$ more /etc/locale.gen |grep -v ^#
	de_DE.UTF-8 UTF-8
	en_US ISO-8859-1
	en_US.UTF-8 UTF-8

(On Debian/Ubuntu, python and other tools complain a lot if the used 
 locale is not at /etc/locale.gen)

Maybe you're using a different Sphinx version, or maybe the distro
on your VM is using has different locales installed on it or some
other different packages.

Thanks,
Mauro