Re: Sphinx parallel build error: UnicodeEncodeError: 'latin-1' codec can't encode characters in position 18-20: ordinal not in range(256)

From: Mauro Carvalho Chehab <mchehab@kernel.org>
To: Randy Dunlap <rdunlap@infradead.org>
Cc: "Matthew Wilcox" <willy@infradead.org>,
	"Markus Heiser" <markus.heiser@darmarit.de>,
	"Michal Suchánek" <msuchanek@suse.de>,
	linux-doc@vger.kernel.org, "Jonathan Corbet" <corbet@lwn.net>
Subject: Re: Sphinx parallel build error: UnicodeEncodeError: 'latin-1' codec can't encode characters in position 18-20: ordinal not in range(256)
Date: Fri, 7 May 2021 08:39:24 +0200	[thread overview]
Message-ID: <20210507083924.7b8ec1fe@coco.lan> (raw)
In-Reply-To: <be21de46-6655-152e-e431-144c2be6137c@infradead.org>

Em Thu, 6 May 2021 14:21:01 -0700
Randy Dunlap <rdunlap@infradead.org> escreveu:

> On 5/6/21 11:08 AM, Matthew Wilcox wrote:
> > On Thu, May 06, 2021 at 10:57:53AM -0700, Randy Dunlap wrote:  
> >> I have been going thru some of the Documentation/ files...
> >>
> >> Why do several of the files begin with
> >> (hex) ef bb bf    followed by "=================="
> >> for a heading, instead of just "===================".
> >> See e.g. Documentation/timers/no_hz.rst.  

No idea! It seems that the text editor I used on that time added
it for whatever reason.

> > 
> > 00000000  ef bb bf 3d 3d 3d 3d 3d  3d 3d 3d 3d 3d 3d 3d 3d  |...=============|
> > 
> > ef bb bf is utf8 for 0b1111'111011'111111 = 0xFEFF which is the
> > https://en.wikipedia.org/wiki/Byte_order_mark
> > 
> > We should delete it.
> >   
> 
> OK, thanks, I have started on that.
> 
> 
> Just another question: ("inquiring minds want to know")
> 
> Why is/are some docs using U+2217 '*' instead of ASCII '*'?
> E.g., Documentation/block/cdrom-standard.rst.

The cdrom doc is a very special case: it was originally written in LaTeX.
I don't remember any other document in LaTeX inside the Kernel docs during
the conversions I made. See:
	e327cfcb2542 ("docs: cdrom-standard.tex: convert from LaTeX to ReST")

In order to convert it to .rst, I used some tool to first turn it
into plain text (probably LaTeX, but I don't remember anymore), and then
I manually reviewed the entire file, adding ReST tags where needed.

I didn't realize that utf-8 chars were used instead of normal ASCII chars,
as both appear the same when editing it[1].

[1] I use Fedora here. Fedora changed the default charset to utf-8 a long
    time ago.

Anyway, we should be able of get rid of weird UTF-8 chars from it with:

	$ iconv -f utf-8 -t ascii//TRANSLIT Documentation/cdrom/cdrom-standard.rst

I'll prepare a patch fixing it. Some care should be taken, however, as
it has two places where UTF-8 chars should be used[2].

[2] There are two German person names that use UTF-8 chars:
    - 'o' + umlat;
    - a LATIN SMALL LETTER SHARP S (Eszett)

Thanks,
Mauro