The downside of math::

The downside of math::

[Jonathan Corbet]

Hey, Mauro,

So I was a little surprised to find that the htmldocs build now breaks on
one of my machines due to a lack of LaTeX.  A bit of digging turned up
the culprit: commit b7ff94df5628 (pixfmt-007.rst: use Sphinx math::
expressions).  The math:: directive uses LaTeX to process the math markup. 

What this means is that said commit has made LaTeX a dependency for the
htmldocs build.  I'm not convinced that this is a good idea; that's a
massive dependency to add for web builds that, ostensibly, should not
need it.  Certainly we shouldn't add it without discussion...:)

So I'll ask: how important to you is the math extension, and is there any
way we could get you your fancy math in a way that doesn't force
everybody to install the whole LaTeX package set?

Thanks,

jon

Re: The downside of math::

[Mauro Carvalho Chehab]

Hi Jon,

Em Wed, 19 Oct 2016 17:02:46 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Hey, Mauro,
> 
> So I was a little surprised to find that the htmldocs build now breaks on
> one of my machines due to a lack of LaTeX.  A bit of digging turned up
> the culprit: commit b7ff94df5628 (pixfmt-007.rst: use Sphinx math::
> expressions).  The math:: directive uses LaTeX to process the math markup. 
> 
> What this means is that said commit has made LaTeX a dependency for the
> htmldocs build.  I'm not convinced that this is a good idea; that's a
> massive dependency to add for web builds that, ostensibly, should not
> need it.  Certainly we shouldn't add it without discussion...:)
> 
> So I'll ask: how important to you is the math extension, and is there any
> way we could get you your fancy math in a way that doesn't force
> everybody to install the whole LaTeX package set?

As you know, image handling comes with complex expressions ;) We have lots
of expressions at the media documentation, but right now, the complex ones
that require ReST math:: directive are related to colorspace conversions:

	https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-007.html

On DocBook, we used to represent the same expressions using several
tricks, like superscript and used some UTF-8 math symbols. As
expected, they didn't look nice there:

	https://linuxtv.org/downloads/v4l-dvb-apis/ch02s06.html

The initial conversion of those math expressions from DocBook make
them look even worse. It also broke PDF generation, because LaTeX
fonts were not compatible with UTF-8 math symbols (such issue was
latter solved with xelatex - for some other remaining UTF-8 symbols).

That's said, we're currently discussing APIs for codecs on media.
Eventually, their documentation could come with a way more complex
expressions than what we have so far.

So, I really prefer not removing math support.

>From what's written at Sphinx documentation:
	http://www.sphinx-doc.org/en/stable/ext/math.html

It sounded to me that only sphinx.ext.imgmath extension is capable of
providing math on all outputs, by generating an image like this one,
using LaTeX:
	https://linuxtv.org/downloads/v4l-dvb-apis-new/_images/math/ae0a381368b3837b88b1308aaccedcd9a438757a.png

There are two alternative math extensions that handle math::, but both
seem to be specific for HTML, as they use JavaScript. Also, Sphinx
require using either one of the extension, so we cannot enable both,
one for PDF and the other one for HTML.

Perhaps there are some extensions that would allow using something else,
like this one:
	https://bitbucket.org/coh/sphinx-contrib-mathml/overview

But I haven't test. So, I'm not sure:
	- if it would work;
	- if it would be portable to both HTML and PDF outputs;
	- if it won't require an even weirder toolbox.

Thanks,
Mauro

Re: The downside of math::

[Mauro Carvalho Chehab]

Em Wed, 19 Oct 2016 22:26:18 -0200
Mauro Carvalho Chehab <mchehab@s-opensource.com> escreveu:

> Hi Jon,
> 
> Em Wed, 19 Oct 2016 17:02:46 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
> 
> > Hey, Mauro,
> > 
> > So I was a little surprised to find that the htmldocs build now breaks on
> > one of my machines due to a lack of LaTeX.

Btw, I was also surprised when I discovered such dependency myself... I
even commented about that at the media mailing list, as I didn't
understood why make htmldocs were not working on my server:

https://www.mail-archive.com/search?l=linux-media@vger.kernel.org&q=subject:%22Re%5C%3A+%5C%5BPATCH+0%5C%2F9%5C%5D+Prepare+Sphinx+to+build+media+PDF+books%22&o=newest&f=1

Regrds,
Mauro

Re: The downside of math::

[Jani Nikula]

On Thu, 20 Oct 2016, Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> Hi Jon,
>
> Em Wed, 19 Oct 2016 17:02:46 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
>
>> Hey, Mauro,
>> 
>> So I was a little surprised to find that the htmldocs build now breaks on
>> one of my machines due to a lack of LaTeX.  A bit of digging turned up
>> the culprit: commit b7ff94df5628 (pixfmt-007.rst: use Sphinx math::
>> expressions).  The math:: directive uses LaTeX to process the math markup. 
>> 
>> What this means is that said commit has made LaTeX a dependency for the
>> htmldocs build.  I'm not convinced that this is a good idea; that's a
>> massive dependency to add for web builds that, ostensibly, should not
>> need it.  Certainly we shouldn't add it without discussion...:)
>> 
>> So I'll ask: how important to you is the math extension, and is there any
>> way we could get you your fancy math in a way that doesn't force
>> everybody to install the whole LaTeX package set?
>
> As you know, image handling comes with complex expressions ;) We have lots
> of expressions at the media documentation, but right now, the complex ones
> that require ReST math:: directive are related to colorspace conversions:
>
> 	https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-007.html
>
> On DocBook, we used to represent the same expressions using several
> tricks, like superscript and used some UTF-8 math symbols. As
> expected, they didn't look nice there:
>
> 	https://linuxtv.org/downloads/v4l-dvb-apis/ch02s06.html
>
> The initial conversion of those math expressions from DocBook make
> them look even worse. It also broke PDF generation, because LaTeX
> fonts were not compatible with UTF-8 math symbols (such issue was
> latter solved with xelatex - for some other remaining UTF-8 symbols).
>
> That's said, we're currently discussing APIs for codecs on media.
> Eventually, their documentation could come with a way more complex
> expressions than what we have so far.
>
> So, I really prefer not removing math support.

I wonder if we could cook up a nice way to make the math:: usage
conditional on actually being able to render it.

I would rather have the math:: directive produce just the preformatted
indented block as-is in the html output, instead of failing, when latex
is not available.

Adapt to the environment, produce the prettiest thing when you can, but
fallback to something uglier if all the dependencies are not there. See
what I did with the sphinx_rtd_theme in conf.py - we use it if it's
there, but it's not a hard dependency.

BR,
Jani.


>
> From what's written at Sphinx documentation:
> 	http://www.sphinx-doc.org/en/stable/ext/math.html
>
> It sounded to me that only sphinx.ext.imgmath extension is capable of
> providing math on all outputs, by generating an image like this one,
> using LaTeX:
> 	https://linuxtv.org/downloads/v4l-dvb-apis-new/_images/math/ae0a381368b3837b88b1308aaccedcd9a438757a.png
>
> There are two alternative math extensions that handle math::, but both
> seem to be specific for HTML, as they use JavaScript. Also, Sphinx
> require using either one of the extension, so we cannot enable both,
> one for PDF and the other one for HTML.
>
> Perhaps there are some extensions that would allow using something else,
> like this one:
> 	https://bitbucket.org/coh/sphinx-contrib-mathml/overview
>
> But I haven't test. So, I'm not sure:
> 	- if it would work;
> 	- if it would be portable to both HTML and PDF outputs;
> 	- if it won't require an even weirder toolbox.
>
> Thanks,
> Mauro
> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Jani Nikula, Intel Open Source Technology Center

Re: The downside of math::

[Markus Heiser]

Am 20.10.2016 um 16:55 schrieb Jani Nikula <jani.nikula@linux.intel.com>:
>> So, I really prefer not removing math support.
> 
> I wonder if we could cook up a nice way to make the math:: usage
> conditional on actually being able to render it.
> 
> I would rather have the math:: directive produce just the preformatted
> indented block as-is in the html output, instead of failing, when latex
> is not available.
> 
> Adapt to the environment, produce the prettiest thing when you can, but
> fallback to something uglier if all the dependencies are not there. See
> what I did with the sphinx_rtd_theme in conf.py - we use it if it's
> there, but it's not a hard dependency.

Hi Jani, 

you are right. I will add this point to my TODO list. Give me just
a view days / ATM I'am working on a other project.

--Markus--

Re: The downside of math::

[Jonathan Corbet]

On Thu, 20 Oct 2016 17:55:21 +0300
Jani Nikula <jani.nikula@linux.intel.com> wrote:

> I wonder if we could cook up a nice way to make the math:: usage
> conditional on actually being able to render it.

I think that's the ideal solution.

I got the docs build working again on my Fedora machine, but it threw me
back into this loop for several iterations:

	while (htmldocs build fails)
		see which goddam LaTex file is missing now
		dig around to find which Fedora package provides it
		dnf install YA-texlive-package

This is just the kind of thing that I don't want to impose on anybody
wanting to build the docs; if we get back to a place where almost nobody
can do it again, we'll not have improved much.

LaTeX is a necessary evil for PDF output, it seems, but I really think
that, one way or another, we need to be able to build the other formats
without it.

Thanks,

jon

Re: The downside of math::

[Markus Heiser]

Am 21.10.2016 um 23:38 schrieb Jonathan Corbet <corbet@lwn.net>:

> On Thu, 20 Oct 2016 17:55:21 +0300
> Jani Nikula <jani.nikula@linux.intel.com> wrote:
> 
>> I wonder if we could cook up a nice way to make the math:: usage
>> conditional on actually being able to render it.
> 
> I think that's the ideal solution.
> 
> I got the docs build working again on my Fedora machine, but it threw me
> back into this loop for several iterations:
> 
> 	while (htmldocs build fails)
> 		see which goddam LaTex file is missing now
> 		dig around to find which Fedora package provides it
> 		dnf install YA-texlive-package

Hmm, when I deinstall latex on my Debian box, I only get a warning:

 WARNING: LaTeX command 'latex' cannot be run (needed for math display), check the pngmath_latex setting

IMO the warning is OK, if not read below.

> This is just the kind of thing that I don't want to impose on anybody
> wanting to build the docs; if we get back to a place where almost nobody
> can do it again, we'll not have improved much.

I guess, that Jon walks through the latex installation hell on fedora,
which mauro has already done in August. Please take a look at the
discussion we had about math and PDF generation:

 https://www.mail-archive.com/linux-doc@vger.kernel.org/msg05583.html

With this, I think we have a leak of knowledge and we should document
latex installation dependencies right now.

If you don't want to see a warning log, we have to consider 
a solution like Johannes Berg posted here:

 https://www.mail-archive.com/linux-doc@vger.kernel.org/msg07071.html

@johannes: since this thread and our "sequence diagrams" thread addressing
the same questions (how should we handle dependencies from extensions)
I propose to continue the discussion here in this thread with Jon.

> LaTeX is a necessary evil for PDF output, it seems, but I really think
> that, one way or another, we need to be able to build the other formats
> without it.

I wish, but I can't see any serious alternative. Thats why I think, we
should accept latex and have to document it well.

Further I think we should not generate more (and more) external requirements
like e.g. plantuml, Java or reportlab discussed here:

 https://www.mail-archive.com/linux-doc@vger.kernel.org/msg07035.html

I understand the need of sequences diagrams and I like to see such a feature
in the kernel documentation. On the other side, since we have to maintain all
dependencies well over a long time period on distributions with different 
package managers, I really hesitate to include dependencies which might flood
out the banks. Right here, in the discussion of the math:: extension (which
is part of the standard sphinx installation) we see, what problems can occur on
different systems installations.

-- Markus --

Re: The downside of math::

[Johannes Berg]

On Sun, 2016-10-23 at 12:58 +0200, Markus Heiser wrote:

> If you don't want to see a warning log, we have to consider 
> a solution like Johannes Berg posted here:
> 
>  https://www.mail-archive.com/linux-doc@vger.kernel.org/msg07071.html
> 
> @johannes: since this thread and our "sequence diagrams" thread
> addressing the same questions (how should we handle dependencies from
> extensions) I propose to continue the discussion here in this thread
> with Jon.

Sure.

As I said before, regarding the annoying "while (build fails)" loop
mentioned above - I think that's actually somewhat *better* than just
(silently?) failing the build because some dependency wasn't there.

Perhaps, with my code suitably adjusted (should be easy), we could
provide some kind of command line option like "DISABLE_PLUGINS=xyz" so
you could just build as though you had no plugins installed, to avoid
all the follow-up errors. (We'd implement this by changing my dummy
code to also pass the import statement, like this:

setup = create_setup("sphinxcontrib.plantuml", ["uml"])

and moving the try/except clause into the dummy, and then that can read
the environment variable easily and avoid even trying to import when
it's set - that way, no dependencies can be needed.

> Further I think we should not generate more (and more) external
> requirements like e.g. plantuml, Java or reportlab discussed here:

I still disagree, I think we should make it easy to "opt out" for the
build, but if we're really serious about writing good documentation we
shouldn't (artificially) limit the tools available.

Sure, we probably shouldn't add seqdiag, plantuml, and others that all
the do the same thing - if only for the sake of consistency in the
generated output! - but having the ability to have such things is very
nice and helpful for documentation.

johannes

Re: The downside of math::

[Jani Nikula]

On Mon, 24 Oct 2016, Johannes Berg <johannes@sipsolutions.net> wrote:
> On Sun, 2016-10-23 at 12:58 +0200, Markus Heiser wrote:
>> Further I think we should not generate more (and more) external
>> requirements like e.g. plantuml, Java or reportlab discussed here:
>
> I still disagree, I think we should make it easy to "opt out" for the
> build, but if we're really serious about writing good documentation we
> shouldn't (artificially) limit the tools available.

I'd like to refine: Do not add non-trivial hard dependencies. Do not add
dependencies the lack of which make large parts of generated
documentation useless.

Graceful degradation on unmet dependencies is the key here. Give a build
warn about missing dependencies. Try to do something sensible without
the dependency. For extension directives, all else failing, embed the
raw directive block contents using in the output, possibly accompanied
by a reStructuredText admonition block [1]. For the math and diagram
directives, I think this would work just fine.

BR,
Jani.


[1] http://docutils.sourceforge.net/docs/ref/rst/directives.html#specific-admonitions


-- 
Jani Nikula, Intel Open Source Technology Center

Re: The downside of math::

[Johannes Berg]

> I'd like to refine: Do not add non-trivial hard dependencies. Do not
> add dependencies the lack of which make large parts of generated
> documentation useless.
> 
> Graceful degradation on unmet dependencies is the key here. 

Agree.

> Give a build
> warn about missing dependencies. Try to do something sensible without
> the dependency. For extension directives, all else failing, embed the
> raw directive block contents using in the output, possibly
> accompanied
> by a reStructuredText admonition block [1]. For the math and diagram
> directives, I think this would work just fine.

All of that would be easy to achieve with something like the dummy
plugin I posted the other day :)


But yes, I also agree that we shouldn't add any dependencies where the
input isn't suitable as a fallback, since that also means that the
input is going to be difficult to read and write in the source, which
should remain a reasonable way to read things as well.

johannes

Re: The downside of math::

[Jani Nikula]

On Mon, 24 Oct 2016, Johannes Berg <johannes@sipsolutions.net> wrote:
> But yes, I also agree that we shouldn't add any dependencies where the
> input isn't suitable as a fallback, since that also means that the
> input is going to be difficult to read and write in the source, which
> should remain a reasonable way to read things as well.

Agreed, that's a very good point too.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center