From: Mauro Carvalho Chehab <mchehab@kernel.org>
To: "Michal Suchánek" <msuchanek@suse.de>
Cc: Randy Dunlap <rdunlap@infradead.org>,
Matthew Wilcox <willy@infradead.org>,
Markus Heiser <markus.heiser@darmarit.de>,
linux-doc@vger.kernel.org, Jonathan Corbet <corbet@lwn.net>
Subject: Re: Sphinx parallel build error: UnicodeEncodeError: 'latin-1' codec can't encode characters in position 18-20: ordinal not in range(256)
Date: Fri, 7 May 2021 11:02:37 +0200 [thread overview]
Message-ID: <20210507110237.5868bf4f@coco.lan> (raw)
In-Reply-To: <20210507083527.GL6564@kitsune.suse.cz>
Em Fri, 7 May 2021 10:35:27 +0200
Michal Suchánek <msuchanek@suse.de> escreveu:
> On Fri, May 07, 2021 at 10:04:35AM +0200, Mauro Carvalho Chehab wrote:
> > Em Fri, 7 May 2021 08:39:24 +0200
> > Mauro Carvalho Chehab <mchehab@kernel.org> escreveu:
> >
> > > Em Thu, 6 May 2021 14:21:01 -0700
> > > Randy Dunlap <rdunlap@infradead.org> escreveu:
> > >
> > > > On 5/6/21 11:08 AM, Matthew Wilcox wrote:
> > > > > On Thu, May 06, 2021 at 10:57:53AM -0700, Randy Dunlap wrote:
> > > > >> I have been going thru some of the Documentation/ files...
> > > > >>
> > > > >> Why do several of the files begin with
> > > > >> (hex) ef bb bf followed by "=================="
> > > > >> for a heading, instead of just "===================".
> > > > >> See e.g. Documentation/timers/no_hz.rst.
> > >
> > > No idea! It seems that the text editor I used on that time added
> > > it for whatever reason.
> > >
> > > > >
> > > > > 00000000 ef bb bf 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d |...=============|
> > > > >
> > > > > ef bb bf is utf8 for 0b1111'111011'111111 = 0xFEFF which is the
> > > > > https://en.wikipedia.org/wiki/Byte_order_mark
> > > > >
> > > > > We should delete it.
> > > > >
> > > >
> > > > OK, thanks, I have started on that.
> > > >
> > > >
> > > > Just another question: ("inquiring minds want to know")
> > > >
> > > > Why is/are some docs using U+2217 '*' instead of ASCII '*'?
> > > > E.g., Documentation/block/cdrom-standard.rst.
> > >
> > > The cdrom doc is a very special case: it was originally written in LaTeX.
> > > I don't remember any other document in LaTeX inside the Kernel docs during
> > > the conversions I made. See:
> > > e327cfcb2542 ("docs: cdrom-standard.tex: convert from LaTeX to ReST")
> > >
> > > In order to convert it to .rst, I used some tool to first turn it
> > > into plain text (probably LaTeX, but I don't remember anymore), and then
> > > I manually reviewed the entire file, adding ReST tags where needed.
> > >
> > > I didn't realize that utf-8 chars were used instead of normal ASCII chars,
> > > as both appear the same when editing it[1].
> > >
> > > [1] I use Fedora here. Fedora changed the default charset to utf-8 a long
> > > time ago.
> > >
> > > Anyway, we should be able of get rid of weird UTF-8 chars from it with:
> > >
> > > $ iconv -f utf-8 -t ascii//TRANSLIT Documentation/cdrom/cdrom-standard.rst
> > >
> > > I'll prepare a patch fixing it. Some care should be taken, however, as
> > > it has two places where UTF-8 chars should be used[2].
> > >
> > > [2] There are two German person names that use UTF-8 chars:
> > > - 'o' + umlat;
> > > - a LATIN SMALL LETTER SHARP S (Eszett)
> >
> > Btw, I did a quick check here: excluding translations, there are 182
> > files with UTF-8 chars at next-20210429. It seems that most of them
> > are on files that got converted from DocBook and html.
> >
> > Several of them are valid ones: the ones used on names
> > (like Günther, Alcôve, ...).
>
> > 2. Some UTF-8 symbols, like:
> >
> > - ®
> > - ™
> > - ² - used mainly for I²C
> > - …
> > - ⬍ ↑ ↓
> > - µs - used for microsseconds
>
> > 3. There are couple of places which uses UTF-8 graphic characters, like:
> >
> > /sys/devices/system/edac/
> > ├── mc
> > │ ├── mc0
> > │ │ ├── ce_count
> > │ │ ├── ce_noinfo_count
>
> > I'm preparing a patchset to address the UTF-8 issues on the top of
> > today's next, but before posting, it seems reasonable to discuss
> > what to do with the above cases. Comments?
>
> So the bottom line is that UTF-8 in the files will stay, and Sphinx
> cannot handle UTF-8 when the locale is not UTF-8.
Yes. We can reduce the number of UTF-8, but some documents need more
chars than ASCII can provide.
Btw, probably (almost?) all files under Documentation/translation use
UTF-8 charsets, due to obvious reasons.
> In the long run it might be nice to fix Sphinx to properly set the
> encoding of the files it reads and writes.
Agreed.
> Or maybe there is some parameter that specifies it?
>
> For the short term I think it is reasonable to run a python test script
> that prints fancy unicode characters before running Sphinx and bail if
> the test script fails.
>
> eg.
> echo 'print("↑ᛏ个")' > test.py
> python3 test.py
Actually, a better workaround could be introduced at conf.py. This
file is read/parsed by Sphinx on an early stage.
Something could be added there that would detect if the machine's
charset is not UTF-8 and either produce a warning before starts
building or would change the charset used by python to something
that won't crash with utf-8.
Thanks,
Mauro
next prev parent reply other threads:[~2021-05-07 9:02 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-06 10:39 Sphinx parallel build error: UnicodeEncodeError: 'latin-1' codec can't encode characters in position 18-20: ordinal not in range(256) Michal Suchánek
2021-05-06 11:20 ` Mauro Carvalho Chehab
2021-05-06 13:32 ` Michal Suchánek
2021-05-06 14:24 ` Mauro Carvalho Chehab
2021-05-06 14:35 ` Michal Suchánek
2021-05-06 15:57 ` Markus Heiser
2021-05-06 16:46 ` Mauro Carvalho Chehab
2021-05-06 17:04 ` Markus Heiser
2021-05-06 17:27 ` Mauro Carvalho Chehab
2021-05-06 17:53 ` Markus Heiser
2021-05-06 18:06 ` Michal Suchánek
2021-05-07 8:52 ` Mauro Carvalho Chehab
2021-05-06 17:57 ` Randy Dunlap
2021-05-06 18:08 ` Matthew Wilcox
2021-05-06 21:21 ` Randy Dunlap
2021-05-07 6:39 ` Mauro Carvalho Chehab
2021-05-07 6:49 ` Randy Dunlap
2021-05-07 8:04 ` Mauro Carvalho Chehab
2021-05-07 8:35 ` Michal Suchánek
2021-05-07 8:56 ` Markus Heiser
2021-05-07 9:14 ` Mauro Carvalho Chehab
2021-05-07 9:51 ` Markus Heiser
2021-05-07 10:29 ` Michal Suchánek
2021-05-07 9:02 ` Mauro Carvalho Chehab [this message]
2021-05-08 9:22 ` Mauro Carvalho Chehab
2021-05-08 10:41 ` Michal Suchánek
2021-05-08 14:41 ` Mauro Carvalho Chehab
2021-05-08 15:55 ` Randy Dunlap
2021-05-08 17:09 ` Michal Suchánek
2021-05-08 17:46 ` Randy Dunlap
2021-05-10 6:22 ` Mauro Carvalho Chehab
2021-05-10 8:17 ` Mauro Carvalho Chehab
2021-05-06 17:48 ` Michal Suchánek
2021-05-06 17:59 ` Markus Heiser
2021-05-06 18:16 ` Michal Suchánek
2021-05-12 6:22 ` Mauro Carvalho Chehab
2021-05-12 7:01 ` Michal Suchánek
2021-05-12 7:18 ` Markus Heiser
2021-05-12 7:37 ` Markus Heiser
2021-05-12 7:59 ` Mauro Carvalho Chehab
2021-05-17 13:10 ` Michal Suchánek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210507110237.5868bf4f@coco.lan \
--to=mchehab@kernel.org \
--cc=corbet@lwn.net \
--cc=linux-doc@vger.kernel.org \
--cc=markus.heiser@darmarit.de \
--cc=msuchanek@suse.de \
--cc=rdunlap@infradead.org \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).