Re: Sphinx version dependencies?

Re: Sphinx version dependencies?

[Markus Heiser]

Am Donnerstag, den 19.07.2018, 12:04 -0700 schrieb Darrick J. Wong:
> On Thu, Jul 19, 2018 at 02:15:56PM -0400, Theodore Y. Ts'o wrote:
> > Darrick has sent in patches to convert the ext4 documentation to use
> > rst and to be built as part of the full kernel documentation thanks.
> > In addition to that, he's imported the on-disk documentation from the
> > ext4 wiki into the kernel sources, so hopefully we can keep it more up
> > to date.
> > 
> > When I was experimenting with this, I had to actually build the kernel
> > docs using Sphinx for the first time.  I'm using Debian testing, so at
> > first I blindly followed the instructions by
> > ./scripts/sphinx-pre-install:
> > 
> > Detected OS: Debian GNU/Linux unstable (sid).
> > 	/usr/local/bin/virtualenv sphinx_1.4
> > 	. sphinx_1.4/bin/activate
> > 	pip install -r Documentation/sphinx/requirements.txt
> > 
> > But when I did that, Sphinx had heartburn over the ext4.rst file.
> > 
> >     ./include/linux/spi/spi.h:373: ERROR: Unexpected indentation.
> >     /usr/projects/linux/ext4/Documentation/filesystems/ext4/ext4.rst:139: ERROR: Malformed table.
> >     Column span alignment problem in table line 5.
> 
> Hmmm, apparently it's choking on the table heading borders not matching
> the text:
> 
> ====== ===========
> Foo    Bar
> ====== ===========        <-- need to extend to EOL
> Blah   Blah blah blah blah
> ====== ===========
> 
> Though weirdly while I /can/ get this error to reproduce with the
> virtualenv 1.4.9 release, I can't get it to reproduce with the 1.3.6 or
> 1.6.7 ubuntu packages.  Maybe it's a python3 thing, maybe not?  

Not a py2 or py3 thing. I *guess* it is caused by changes in the underlying
docutils.

> Seems
> pretty fragile to me.

Yes, this is a very confusing and we have to look at reST, docutils, Sphinx-doc,
virtualenv py2 & py3, all these distros and the philosophy of the Linux
community to understand.

Yes, this is a very confusing and we have to look at reST, docutils, Sphinx-doc,
virtualenv py2 & py3, all these distros and the philosophy of the Linux
community to understand.

reST: Is the markup.

docutils (python): Is the reST reference implementation with some changes in
  2017[1].

Sphinx-doc (python): reST application based on the docutils lib
  (docutils>=0.11)[2].  Take in mind, this dependency opens a new array of
  confusion, cause its fulfillment is based on what is installed on your system
  (or in your virtualenv, see below). Side node: the pdf-builder (LaTeX, to be
  correct) is in a continues redesign since the last two years and was horrible
  in older Sphinx-doc versions. E.g. take a look at the current 1.8.0
  development (section deprecated)[4].

python: py2 will retire in 2020 (py3 was released in 2008). All py applications
   we using are py2 and py3 compatible. If we have no bugs, py2 or py3 should
   never be the cause.

distros: Nearly all distros using using py2 as there default python interpreter
   (mostly since there package managers are written/tested with py2).

virtualenv: A virtualenv is build up in the $HOME folder where Sphinx-doc and
   docutils are installed from pypi, the python package manager[3]. Take in
   mind, using another package in parallel to the system package manager manager
   opens another array of confusion.

Normally I would say, lets use (and test against) state of the art. This is
easily done by installing py3 and using virtualenv. But this seems not match the
philosophy of the whole Linux community.

As one told me, the philosophy is to build the Kernel with less installations
from the web by using what the distro ships. IMO this might be right for the
Kernel but not for applications building viewing formats. When it comes to
build PDF you will realize how naive this approach is. IMO there is also no
need to build viewing formats as we can read the documentation in plain text
as we can do it since the beginning ... and this is one reason why we changed
from XML to a plain text markup.

The script named 'sphinx-pre-install' is the crutch that supports this
philosophy.

We have this discussion over and over again an I believe we will have it again
and again, as long as we do not change our POV .. building viewing formats is
an application not a part of the Kernel.
 
Anyway, I hope this helps to clarify, why we have those problems.

[1] http://docutils.sourceforge.net/HISTORY.html 
[2] https://github.com/sphinx-doc/sphinx/blob/master/setup.py#L22
[3] https://pypi.org/
[4] https://github.com/sphinx-doc/sphinx/blob/master/CHANGES

-- Markus --

> 
> Anyway, I'll fix ext4.rst and resubmit that part to get this moving.
> 
> --D
> 
> >     ...
> > 
> > After consulting with Darrick, it appears the problem is that Sphinx8
> > 1.4.9 was the problem.  This is the version that
> > Documentation/sphinx/requirements.txt calls for.  He did his rst
> > conversion work using Ubuntu 18.04's Sphinx 1.6.7.
> > 
> > As it turns out Debian testing/unstable already has Sphinx 1.7.5 in
> > its repository, so if I simply install Sphinx 1.7.5, it works fine.
> > That's what I've done for now.
> > 
> > So that leaves me with some questions:
> > 
> > * Is there a reason why scripts/sphinx-pre-install suggested using a
> >   Python virtual environment and installing Sphinx 1.4.9 instead of
> >   using the distro's pre-packaged Sphinx for Debian unstable/testing?
> > 
> > * Why does Documentation/sphinx/requirements.txt asking for such an
> >   old version of Sphinx?
> >  
> > * Is it a requirement that *.rst files that are checked into the
> >   kernel repo have to work with Sphinx 1.4.9?  Or is it sufficient
> >   that it works with Sphinx 1.6.7 and 1.7.5 (which are the prepackaged
> >   Debian and Ubuntu versions).  And it looks like Fedora 28 has Sphinx
> >   1.7.2 if I'm not mistaken.   How many versions of Sphinx are various
> >   automated build/test systems using?
> > 
> > Thanks,
> > 
> > 						- Ted
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Sphinx version dependencies?

[Theodore Y. Ts'o]

On Fri, Jul 20, 2018 at 09:30:33AM +0200, Markus Heiser wrote:
> Normally I would say, lets use (and test against) state of the art. This is
> easily done by installing py3 and using virtualenv. But this seems not match the
> philosophy of the whole Linux community.
> 
> As one told me, the philosophy is to build the Kernel with less installations
> from the web by using what the distro ships. IMO this might be right for the
> Kernel but not for applications building viewing formats. When it comes to
> build PDF you will realize how naive this approach is. IMO there is also no
> need to build viewing formats as we can read the documentation in plain text
> as we can do it since the beginning ... and this is one reason why we changed
> from XML to a plain text markup.u
> 
> The script named 'sphinx-pre-install' is the crutch that supports this
> philosophy.

It's clear we have an over-constrained problem.  On the one hand, we
want to make sure that the Linux kernel (but maybe not the docs?
unclear?) can build on a wide variety of Linux distros, from those
that are shipping (in some cases) decade-old, antiquated tools in
Enterprise Distrobutions, to bleeding edge Community Distro's.  On the
other, we want to be able to use the latest and greatest features.
And on yet a third hand, we need to deal with the fact that
historically, developers have hated it when they have to download
wierd tools, often ones that are not packaged by the distro, such as
cmake, imake, etc., etc.  (Fortunately with Python virtualenv makes
things relatively painless; unlike in the past.  But I think we
sometimes have biases on past experience.)

My observation though is that at the moment sphinx-pre-install doesn't
really seem to support any one philosophy consistently.  On the one
hand, if you are running Debian unstable, and you don't have the
distro-packaged Sphinx installed, it tells you to use virtualenv, and
force-installs Sphinx 1.4.9.  On the other hand, if you happen to have
the debian-packaged Sphinx installed already it says, "go forth and
build!"  This means that from the perspective of a Kernel developer,
we don't know whether someone will be running Sphinx 1.4.9, or some
other random Distro version.

One approach might be to build the virtualenv setup and download
directly into build sequence.  But the problem with that is it would
break hermetic build systems that some distros use, which (by design)
are not connected to the network for security reasons.

Another solution is to specify that the kernel docs *must* build on
1.4.9, and change sphinx-pre-install to always require this.  This
might require imposing a requirement on distro's to package more than
one version of Sphinx.  (There is precedence, at one point Debian
packaged something like four different versions of autoconf, because
it's needed because autoconf is not necessary backwards *or* forwards
compatible.)

Yet another solution is to impose the burden on kernel developers and
tell them that the kernel docs must be built on a variety of Sphinx
versions, and that over time, changes will break the docs even if
nothing has changed.  So in addition to testing on a large range of
Sphinx versions, over time we will have to continuously update the
kernel docs to deal with backwards incompatible changes in how Sphinx
processes the .rst format.

A forth solution is to force everyone to use something correleted to
what the community distro's are packaging, but roughly every 12-18
months, we leapfrog to a newer version of Sphinx.  This decreases the
continuous upgrade burden, but doesn't make it go away.

I'm not entirely sure what's the best approach.  Right now I just want
to understand --- do I have to make ext4.rst work against one, or many
versions of Sphinx?  And which version(s) of Sphinx do I need to
concern myself with?  If that turns out to be an onerous burden, I'm
sure I won't be the only person complaining.  :-)

							- Ted
							

Re: Sphinx version dependencies?

[Markus Heiser]

Am Freitag, den 20.07.2018, 09:12 -0400 schrieb Theodore Y. Ts'o:
> I'm not entirely sure what's the best approach.  Right now I just want
> to understand --- do I have to make ext4.rst work against one, or many
> versions of Sphinx?  And which version(s) of Sphinx do I need to
> concern myself with?  If that turns out to be an onerous burden, I'm
> sure I won't be the only person complaining.  :-)

In that case ...

> But when I did that, Sphinx had heartburn over the ext4.rst file.
> 
>     ./include/linux/spi/spi.h:373: ERROR: Unexpected indentation.
>     /usr/projects/linux/ext4/Documentation/filesystems/ext4/ext4.rst:139: ERROR: Malformed table.
>     Column span alignment problem in table line 5.

... its clear; the table was malformed. A markup error which is not detected
by older versions of docutils (very special case).

In general I recommend to install an up-to-date Sphinx-doc in a virtualenv .. this
is what I do in my projects ... and where I haven't seen those version problems ;)

If we have errors downward we have to handle them. The main thing is the markup
is correct.

  -- Markus --

Re: Sphinx version dependencies?

[Theodore Y. Ts'o]

On Fri, Jul 20, 2018 at 03:45:37PM +0200, Markus Heiser wrote:
> Am Freitag, den 20.07.2018, 09:12 -0400 schrieb Theodore Y. Ts'o:
> > I'm not entirely sure what's the best approach.  Right now I just want
> > to understand --- do I have to make ext4.rst work against one, or many
> > versions of Sphinx?  And which version(s) of Sphinx do I need to
> > concern myself with?  If that turns out to be an onerous burden, I'm
> > sure I won't be the only person complaining.  :-)
> 
> In that case ...
> 
> > But when I did that, Sphinx had heartburn over the ext4.rst file.
> > 
> >     ./include/linux/spi/spi.h:373: ERROR: Unexpected indentation.
> >     /usr/projects/linux/ext4/Documentation/filesystems/ext4/ext4.rst:139: ERROR: Malformed table.
> >     Column span alignment problem in table line 5.
> 
> ... its clear; the table was malformed. A markup error which is not detected
> by older versions of docutils (very special case).

... except that newer verions are A-OK with it.  Apparently 1.3.x was
OK with it, and 1.6.x and 1.7.x were ok with it.  ***ONLY*** Sphinx
1.4.9 blew up on the "malformed table".

So in this case, Darrick has come up with a patch that is makes it OK
with 1.4.9 without breaking on 1.7.5 --- and obviously, doing
something that makes it broadly portable is the right thing.

I'm asking a larger question, which is moving forward, which is more
important?  Make it work with Sphinx 1.4.9?  Or making it Sphinx work
with Sphinx 1.7.5?

And should we change Documentation/sphinx/requirements.txt to require
something newer, such as Sphinx 1.7.5?  And should we require that
Ubuntu 18.04 which is using Sphinx 1.6.8 use a virtualenv and use
download Sphinx 1.6.8?

My understanding that the Sphinx developers make no guarantees that if
we follow some external, version-indepedent spec, that it will work on
Sphinx version N, as well as Sphinx version N+1.  (In the ideal world,
if there was such an independent spec for .rst format files, and a
compliant .rst file doesn't work for Sphinx version N, it's a bug, and
we should expect somebody --- perhaps the Distro's --- to backport the
fix from Sphinx version N+1 to Sphinx version N.)  E.g., is there an
equivalent for ANSI C 1999 standard for .rst files?

		       	      	   	    - Ted

Re: Sphinx version dependencies?

[Markus Heiser]

Am Freitag, den 20.07.2018, 10:52 -0400 schrieb Theodore Y. Ts'o:
> On Fri, Jul 20, 2018 at 03:45:37PM +0200, Markus Heiser wrote:
> > Am Freitag, den 20.07.2018, 09:12 -0400 schrieb Theodore Y. Ts'o:
> > > I'm not entirely sure what's the best approach.  Right now I just want
> > > to understand --- do I have to make ext4.rst work against one, or many
> > > versions of Sphinx?  And which version(s) of Sphinx do I need to
> > > concern myself with?  If that turns out to be an onerous burden, I'm
> > > sure I won't be the only person complaining.  :-)
> > 
> > In that case ...
> > 
> > > But when I did that, Sphinx had heartburn over the ext4.rst file.
> > > 
> > >     ./include/linux/spi/spi.h:373: ERROR: Unexpected indentation.
> > >     /usr/projects/linux/ext4/Documentation/filesystems/ext4/ext4.rst:139: ERROR: Malformed table.
> > >     Column span alignment problem in table line 5.
> > 
> > ... its clear; the table was malformed. A markup error which is not detected
> > by older versions of docutils (very special case).
> 
> ... except that newer verions are A-OK with it.  Apparently 1.3.x was
> OK with it, and 1.6.x and 1.7.x were ok with it.  ***ONLY*** Sphinx
> 1.4.9 blew up on the "malformed table".

Are you sure that it was not due to the docutils version?
I can't reproduce it but the table parser is a part of docutils.

> 
> So in this case, Darrick has come up with a patch that is makes it OK
> with 1.4.9 without breaking on 1.7.5 --- and obviously, doing
> something that makes it broadly portable is the right thing.

Right, fix it by the markup .. is what I recommend.

> I'm asking a larger question, which is moving forward, which is more
> important?  Make it work with Sphinx 1.4.9?  Or making it Sphinx work
> with Sphinx 1.7.5?
> 
> And should we change Documentation/sphinx/requirements.txt to require
> something newer, such as Sphinx 1.7.5?  And should we require that
> Ubuntu 18.04 which is using Sphinx 1.6.8 use a virtualenv and use
> download Sphinx 1.6.8?

The requirements.txt came from commit fb947f3f47 [1] (inital 24071ac1a6).
Where Jon and Mauro decided to tag explicit versions ...

docutils==0.12
Sphinx==1.4.9
sphinx_rtd_theme

Maybe it is time to switch to something like .. ?

Sphinx>=1.4.9
sphinx_rtd_theme

I don't know. Mauro has tested on many distros, he has more experience with
the wide range of distros then I.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fb947f3f47

> 
> My understanding that the Sphinx developers make no guarantees that if
> we follow some external, version-indepedent spec, that it will work on
> Sphinx version N, as well as Sphinx version N+1.  (In the ideal world,
> if there was such an independent spec for .rst format files, and a
> compliant .rst file doesn't work for Sphinx version N, it's a bug, and
> we should expect somebody --- perhaps the Distro's --- to backport the
> fix from Sphinx version N+1 to Sphinx version N.)  E.g., is there an
> equivalent for ANSI C 1999 standard for .rst files?

The reST markup is specified here:

 http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html

but the (last) example of the simple table does not match your "1.4.9"
experience.

-- Markus --


> 
> 		       	      	   	    - Ted

Re: Sphinx version dependencies?

[Darrick J. Wong]

On Fri, Jul 20, 2018 at 06:00:28PM +0200, Markus Heiser wrote:
> Am Freitag, den 20.07.2018, 10:52 -0400 schrieb Theodore Y. Ts'o:
> > On Fri, Jul 20, 2018 at 03:45:37PM +0200, Markus Heiser wrote:
> > > Am Freitag, den 20.07.2018, 09:12 -0400 schrieb Theodore Y. Ts'o:
> > > > I'm not entirely sure what's the best approach.  Right now I just want
> > > > to understand --- do I have to make ext4.rst work against one, or many
> > > > versions of Sphinx?  And which version(s) of Sphinx do I need to
> > > > concern myself with?  If that turns out to be an onerous burden, I'm
> > > > sure I won't be the only person complaining.  :-)
> > > 
> > > In that case ...
> > > 
> > > > But when I did that, Sphinx had heartburn over the ext4.rst file.
> > > > 
> > > >     ./include/linux/spi/spi.h:373: ERROR: Unexpected indentation.
> > > >     /usr/projects/linux/ext4/Documentation/filesystems/ext4/ext4.rst:139: ERROR: Malformed table.
> > > >     Column span alignment problem in table line 5.
> > > 
> > > ... its clear; the table was malformed. A markup error which is not detected
> > > by older versions of docutils (very special case).
> > 
> > ... except that newer verions are A-OK with it.  Apparently 1.3.x was
> > OK with it, and 1.6.x and 1.7.x were ok with it.  ***ONLY*** Sphinx
> > 1.4.9 blew up on the "malformed table".
> 
> Are you sure that it was not due to the docutils version?
> I can't reproduce it but the table parser is a part of docutils.

No idea.  With the virtualenv instructions I get:

$ pip list | egrep -i '(sphinx|docutils)'
docutils         0.12   
Sphinx           1.4.9  
sphinx-rtd-theme 0.4.0  

With Ubuntu 18.04 I get:

$ dpkg -l | egrep -i '(sphinx|docutils)' | awk '{print $2, $3}' | sort -k 2
docutils-common 0.14+dfsg-3
python3-docutils 0.14+dfsg-3
python3-sphinx-rtd-theme 0.2.4-1
sphinx-rtd-theme-common 0.2.4-1
python3-alabaster 0.7.8-1
libjs-sphinxdoc 1.6.7-1ubuntu1
python3-sphinx 1.6.7-1ubuntu1
sphinx-common 1.6.7-1ubuntu1

Ok, newer docutils, maybe that's what it is?

With Ubuntu 16.04 I get:

$ dpkg -l | egrep -i '(sphinx|docutils)' | awk '{print $2, $3}' | sort -k 2
docutils-common 0.12+dfsg-1
python-docutils 0.12+dfsg-1
python-sphinx-rtd-theme 0.1.9-1
sphinx-rtd-theme-common 0.1.9-1
python-alabaster 0.7.7-1
libjs-sphinxdoc 1.3.6-2ubuntu1.1
python-sphinx 1.3.6-2ubuntu1.1
sphinx-common 1.3.6-2ubuntu1.1

and now I'm just confused since 16.04 has the same version of docutils
and an older sphinx and runs fine; but 18.04 has newer docutils and
newer sphinx and runs fine.

> > 
> > So in this case, Darrick has come up with a patch that is makes it OK
> > with 1.4.9 without breaking on 1.7.5 --- and obviously, doing
> > something that makes it broadly portable is the right thing.
> 
> Right, fix it by the markup .. is what I recommend.
> 
> > I'm asking a larger question, which is moving forward, which is more
> > important?  Make it work with Sphinx 1.4.9?  Or making it Sphinx work
> > with Sphinx 1.7.5?
> > 
> > And should we change Documentation/sphinx/requirements.txt to require
> > something newer, such as Sphinx 1.7.5?  And should we require that
> > Ubuntu 18.04 which is using Sphinx 1.6.8 use a virtualenv and use
> > download Sphinx 1.6.8?
> 
> The requirements.txt came from commit fb947f3f47 [1] (inital 24071ac1a6).
> Where Jon and Mauro decided to tag explicit versions ...
> 
> docutils==0.12
> Sphinx==1.4.9
> sphinx_rtd_theme
> 
> Maybe it is time to switch to something like .. ?
> 
> Sphinx>=1.4.9
> sphinx_rtd_theme
> 
> I don't know. Mauro has tested on many distros, he has more experience with
> the wide range of distros then I.
> 
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fb947f3f47
> 
> > 
> > My understanding that the Sphinx developers make no guarantees that if
> > we follow some external, version-indepedent spec, that it will work on
> > Sphinx version N, as well as Sphinx version N+1.  (In the ideal world,
> > if there was such an independent spec for .rst format files, and a
> > compliant .rst file doesn't work for Sphinx version N, it's a bug, and
> > we should expect somebody --- perhaps the Distro's --- to backport the
> > fix from Sphinx version N+1 to Sphinx version N.)  E.g., is there an
> > equivalent for ANSI C 1999 standard for .rst files?
> 
> The reST markup is specified here:
> 
>  http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html
> 
> but the (last) example of the simple table does not match your "1.4.9"
> experience.

Yes.  This makes writing broadly portable markup difficult -- originally
I did not take the '=' all the way to the right edge of the table
because I saw that last example in the above document and assumed that
it wasn't necessary to extend the '=' all the way to the right edge.
Neither Ubuntu system choked on it, so is this a bug in upstream?  Some
strange patch added by the distro?  Something that ended up in the
python wheel?  Or a bug in the spec?

--D

> -- Markus --
> 
> 
> > 
> > 		       	      	   	    - Ted

Re: Sphinx version dependencies?

[Markus Heiser]

Am Freitag, den 20.07.2018, 09:44 -0700 schrieb Darrick J. Wong:
> and now I'm just confused since 16.04 has the same version of docutils
> and an older sphinx and runs fine; but 18.04 has newer docutils and
> newer sphinx and runs fine.

Same here .. I'am completely confused by distros etc. .. I will never be able to
control that. Thats why I recommend the virtualenv workflow (as I wrote Christoph):

$ sudo apt install python3-virtualenv

To setup up a environment for building htmldocs:

$ python3 -m virtualenv py3env
$ ./py3env/bin/pip install -r ./Documentation/sphinx/requirements.txt

To build htmldocs with:

 $ SPHINXBUILD=./py3env/bin/sphinx-build make htmldocs

If the env is no longer needed:

 $ rm -r py3env

Am Freitag, den 20.07.2018, 09:44 -0700 schrieb Darrick J. Wong:
> Yes.  This makes writing broadly portable markup difficult -- originally
> I did not take the '=' all the way to the right edge of the table
> because I saw that last example in the above document and assumed that
> it wasn't necessary to extend the '=' all the way to the right edge.
> Neither Ubuntu system choked on it, so is this a bug in upstream?  Some
> strange patch added by the distro?  Something that ended up in the
> python wheel?  Or a bug in the spec?

I guess in the version history of docutils ;) .. use workflow above to escape
from distro chaos. Even missing '=' will be OK.

-- Markus --

Re: Sphinx version dependencies?

[Darrick J. Wong]

On Fri, Jul 20, 2018 at 06:58:57PM +0200, Markus Heiser wrote:
> Am Freitag, den 20.07.2018, 09:44 -0700 schrieb Darrick J. Wong:
> > and now I'm just confused since 16.04 has the same version of docutils
> > and an older sphinx and runs fine; but 18.04 has newer docutils and
> > newer sphinx and runs fine.
> 
> Same here .. I'am completely confused by distros etc. .. I will never be able to
> control that. Thats why I recommend the virtualenv workflow (as I wrote Christoph):

Well yes, but it's the virtualenv workflow that produced build errors
for Ted; that's what would seem to need fixing?

--D

> $ sudo apt install python3-virtualenv
> 
> To setup up a environment for building htmldocs:
> 
> $ python3 -m virtualenv py3env
> $ ./py3env/bin/pip install -r ./Documentation/sphinx/requirements.txt
> 
> To build htmldocs with:
> 
>  $ SPHINXBUILD=./py3env/bin/sphinx-build make htmldocs
> 
> If the env is no longer needed:
> 
>  $ rm -r py3env
> 
> Am Freitag, den 20.07.2018, 09:44 -0700 schrieb Darrick J. Wong:
> > Yes.  This makes writing broadly portable markup difficult -- originally
> > I did not take the '=' all the way to the right edge of the table
> > because I saw that last example in the above document and assumed that
> > it wasn't necessary to extend the '=' all the way to the right edge.
> > Neither Ubuntu system choked on it, so is this a bug in upstream?  Some
> > strange patch added by the distro?  Something that ended up in the
> > python wheel?  Or a bug in the spec?
> 
> I guess in the version history of docutils ;) .. use workflow above to escape
> from distro chaos. Even missing '=' will be OK.
> 
> -- Markus --

Re: Sphinx version dependencies?

[Theodore Y. Ts'o]

On Fri, Jul 20, 2018 at 10:10:20AM -0700, Darrick J. Wong wrote:
> 
> Well yes, but it's the virtualenv workflow that produced build errors
> for Ted; that's what would seem to need fixing?

What would delight me if there was a fixed docutils and Sphinx version
which is the **only** thing which subsystem maintainers need to test
against.  If it fails for that version, then I reject the patch; and
if it works on that version (say, 1.4.9), and it fails on some other
version that a Distro wants to use for its hermetic build environment
(say, 1.7.5), I can tell them, "not my problem, feel free to send me a
patch that makes things work for 1.7.5, and doesn't break on 1.4.9 ---
or package 1.4.9 for your distro build systems."

I don't really care what the mandated version is --- although given
that Fedora and Debian seem to be using 1.7.5, maybe that's the right
answer, and too bad for the enterprise distro build systems --- that's
why they get paid the big bucks.  I just want to know what I'm obliged
to test against.

So if Documentation/sphinx/requirements.txt is the only thing which is
guaranteed to work, that's fine.  But it might be good to document
that somewhere.

						- Ted

Re: Sphinx version dependencies?

[Markus Heiser]

On 20.07.2018 22:43, Theodore Y. Ts'o wrote:
> So if Documentation/sphinx/requirements.txt is the only thing which is
> guaranteed to work, that's fine.  But it might be good to document
> that somewhere.

Hm, I read it again and I guess this is more ore less what the documentation
will say, see 'Sphinx Install' section. [1]

- distros are fragile
- Documentation/sphinx/requirements.txt is the reference environment

.. even if this needs a virtualenv on specific distros. But what is with
the offline build systems Christoph mentioned? I can't see a solution on
the sometimes contradictory requirements on a *reference env* and the package
reposetorie of a distro.

[1] https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html?#sphinx-install

--Markus--

Re: Sphinx version dependencies?

[Jonathan Corbet]

On Fri, 20 Jul 2018 09:12:06 -0400
"Theodore Y. Ts'o" <tytso@mit.edu> wrote:

> A forth solution is to force everyone to use something correleted to
> what the community distro's are packaging, but roughly every 12-18
> months, we leapfrog to a newer version of Sphinx.  This decreases the
> continuous upgrade burden, but doesn't make it go away.

[I'm mostly AFK at the moment, so haven't been following this discussion
in anything approaching real time].

The above is roughly what I've been aiming for; I'd really like for
people to be able to just build the docs without having to go beyond
their distribution's repos.  It gets challenging at times because Kids
These Days seem to have a different view on issues like long-term
compatibility.  But we've mostly done it; it's pretty rare that we've had
docs build problems that have been tied to a specific version of Sphinx.

You all just got lucky :)

jon

Re: Sphinx version dependencies?

[Christoph Hellwig]

On Fri, Jul 20, 2018 at 09:30:33AM +0200, Markus Heiser wrote:
> We have this discussion over and over again an I believe we will have it again
> and again, as long as we do not change our POV .. building viewing formats is
> an application not a part of the Kernel.

Sorry, but this is complete and utter bullshit.

If that was the case we could skip all that RST stuff and just stick
to text files.  If we want fancy docs it better behave like the rest
of the kernel and just works.  python virtual environments are just
not going to cut it as they lead to a long-term maintainance nightmare.

Re: Sphinx version dependencies?

[Markus Heiser]

Am Freitag, den 20.07.2018, 08:04 -0700 schrieb Christoph Hellwig:
> On Fri, Jul 20, 2018 at 09:30:33AM +0200, Markus Heiser wrote:
> > We have this discussion over and over again an I believe we will have it again
> > and again, as long as we do not change our POV .. building viewing formats is
> > an application not a part of the Kernel.
> 
> Sorry, but this is complete and utter bullshit.

Maybe .. I'am not a Kernel developer .. I only try to help with the doc tool-
chain. But if I don't want to install anything from the web, the plain text
is not as bad.

> 
> If that was the case we could skip all that RST stuff and just stick
> to text files.  If we want fancy docs it better behave like the rest
> of the kernel and just works.  python virtual environments are just
> not going to cut it as they lead to a long-term maintainance nightmare.

Really a nightmare? This is what I do on my ubuntu/debian box ... I do
prefer py3 so I installed *virtualenv-py3* once from distro's package
manager:

$ sudo apt install python3-virtualenv

To setup up a environment for building htmldocs:

$ python3 -m virtualenv py3env
$ ./py3env/bin/pip install -r ./Documentation/sphinx/requirements.txt

now I have all installed in the in the py3env folder. To build htmldocs
with:

 $ SPHINXBUILD=./py3env/bin/sphinx-build make htmldocs

If the env is no longer needed:

 $ rm -r py3env

 -- Markus --


> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Sphinx version dependencies?

[Darrick J. Wong]

On Fri, Jul 20, 2018 at 06:28:09PM +0200, Markus Heiser wrote:
> Am Freitag, den 20.07.2018, 08:04 -0700 schrieb Christoph Hellwig:
> > On Fri, Jul 20, 2018 at 09:30:33AM +0200, Markus Heiser wrote:
> > > We have this discussion over and over again an I believe we will have it again
> > > and again, as long as we do not change our POV .. building viewing formats is
> > > an application not a part of the Kernel.
> > 
> > Sorry, but this is complete and utter bullshit.
> 
> Maybe .. I'am not a Kernel developer .. I only try to help with the doc tool-

It is (broken, that is, not bulls*).  Building the documentation as part
of the kernel build *is* an application.

The point of documentation is that we record knowledge and then anyone
ought to be able to read it, no matter what platform they choose, right?
So that means that for any given user, if their Linux distro packages
the doc build tools they ought to be able to build the fancy html stuff
and read it.  If they're more accustomed to virtualenv and pip, they
ought to be able to build the fancy html and read it too.

Anyway, this whole thread got started because Ted followed the
instructions, applied my Documentation/ patches, and the build broke.
From my POV this is just brokenness wrt to the rst spec in the docutils
egg/wheel/whatever python calls it now.

> chain. But if I don't want to install anything from the web, the plain text
> is not as bad.

It will be once we start uploading filesystem format documentation
that's full of juicy tables and diagrams...

> > 
> > If that was the case we could skip all that RST stuff and just stick
> > to text files.  If we want fancy docs it better behave like the rest
> > of the kernel and just works.  python virtual environments are just
> > not going to cut it as they lead to a long-term maintainance nightmare.
> 
> Really a nightmare? This is what I do on my ubuntu/debian box ... I do
> prefer py3 so I installed *virtualenv-py3* once from distro's package
> manager:
> 
> $ sudo apt install python3-virtualenv
> 
> To setup up a environment for building htmldocs:
> 
> $ python3 -m virtualenv py3env
> $ ./py3env/bin/pip install -r ./Documentation/sphinx/requirements.txt

This isn't going to fly in $corporate hermetically sealed kernel build
environment.  It's ok to add things to the list of packages pulled in by
the provisioning system prior to a kernel build, but we cannot support
downloading python wheels for kernel builds.  Therefore the distro
packages have to work.

Which is fine, because the problem here is that whatever pip installs is
less forgiving than the distro packaged versions. :)

--D

> 
> now I have all installed in the in the py3env folder. To build htmldocs
> with:
> 
>  $ SPHINXBUILD=./py3env/bin/sphinx-build make htmldocs
> 
> If the env is no longer needed:
> 
>  $ rm -r py3env
> 
>  -- Markus --
> 
> 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html