Re: parts of media docs sphinx re-building every time?

Re: parts of media docs sphinx re-building every time?

[Markus Heiser]

Hi Jani,

Am 08.08.2016 um 17:37 schrieb Jani Nikula <jani.nikula@intel.com>:

> 
> Hi Mauro & co -
> 
> I just noticed running 'make htmldocs' rebuilds parts of media docs
> every time on repeated runs. This shouldn't happen. Please investigate.
> 
> I wonder if it's related to Documentation/media/Makefile... which I have
> to say I am not impressed by. I was really hoping we could build all the
> documentation by standalone sphinx-build invocation too, relying only on
> the conf.py so that e.g. Read the Docs can build the docs. Part of that
> motivation was to keep the build clean in makefiles, and handing the
> dependency tracking completely to Sphinx.
> 
> I believe what's in Documentation/media/Makefile,
> Documentation/sphinx/parse-headers.pl, and
> Documentation/sphinx/kernel_include.py could be replaced by a Sphinx
> extension looking at the sources directly.

Yes, parse-headers.pl, kernel_include.py and media/Makefile are needed
for one feature ... not very straight forward.

If it makes sense to migrate the perl scripts functionality to a
Sphinx extension, may I can help ... depends on what Mauro thinks.

BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)

> (I presume kernel_include.py
> is mostly a workaround to keep out-of-tree builds working?)

Yes, e.g. with "make O=/tmp/kernel htmldocs" the parse-headers.pl output goes 
to /tmp/kernel and is included by ".. kernel-include: $BUILDDIR/xxx"

-- Markus --
 
> Anyway, the rebuild part is most important. This must be fixed.
> 
> 
> BR,
> Jani.
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center

Re: parts of media docs sphinx re-building every time?

[Mauro Carvalho Chehab]

Hi Jani,

Em Mon, 8 Aug 2016 18:07:10 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Hi Jani,
> 
> Am 08.08.2016 um 17:37 schrieb Jani Nikula <jani.nikula@intel.com>:
> 
> > 
> > Hi Mauro & co -
> > 
> > I just noticed running 'make htmldocs' rebuilds parts of media docs
> > every time on repeated runs. This shouldn't happen. Please investigate.

Perhaps there are some Makefile dependencies there that are not ok.
I'll look in to it.

> > 
> > I wonder if it's related to Documentation/media/Makefile... which I have
> > to say I am not impressed by. I was really hoping we could build all the
> > documentation by standalone sphinx-build invocation too, relying only on
> > the conf.py so that e.g. Read the Docs can build the docs. Part of that
> > motivation was to keep the build clean in makefiles, and handing the
> > dependency tracking completely to Sphinx.
> > 
> > I believe what's in Documentation/media/Makefile,
> > Documentation/sphinx/parse-headers.pl, and
> > Documentation/sphinx/kernel_include.py could be replaced by a Sphinx
> > extension looking at the sources directly.  
> 
> Yes, parse-headers.pl, kernel_include.py and media/Makefile are needed
> for one feature ... not very straight forward.
> 
> If it makes sense to migrate the perl scripts functionality to a
> Sphinx extension, may I can help ... depends on what Mauro thinks.
> 
> BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)

As discussed before, we need to be able to auto-generate cross references
from the headers. Unfortunately, Sphinx acts like a 5-years-old-boy by
painting source files some random colors, but not doing anything
useful like creating cross references with the documentation.

So, we need an extra script for the media build to convert the API headers
into rst files. This work is somewhat complex, as there are symbols that
we explicitly want to ignore, including ifdef symbols like:

	#define _UAPI__LINUX_VIDEODEV2_H
	#ifdef _UAPI__LINUX_VIDEODEV2_H
		...
	#endif

We also want to do things like:

	replace symbol V4L2_TUNER_ANALOG_TV v4l2-tuner-type
	replace symbol V4L2_TUNER_RADIO v4l2-tuner-type
	replace symbol V4L2_TUNER_RF v4l2-tuner-type
	replace symbol V4L2_TUNER_SDR v4l2-tuner-type

in order to make all symbols to point to the same element at the rst file,
that are usually inside a table.

(I actually want to change this to point to an specific row at the
table, but there are almost 400 symbols to be fixed, and changing it
will take some time, and will likely require manual work).

The goal of Documentation/sphinx/parse-headers.pl script is to generate
such parsed headers, with the cross-references modified by an exceptions
file at Documentation/media/*.h.rst.exceptions.

This returns back a feature that we used to have with DocBook.

The Documentation/media/Makefile rules what should be converted,
and what exception file will be used to generate the rst file:

	$(BUILDDIR)/audio.h.rst: ${UAPI}/dvb/audio.h ${PARSER} $(SRC_DIR)/audio.h.rst.exceptions
		@$($(quiet)gen_rst)

We might move that to Documentation/Makefile.sphinx, if you don't
like having another makefile, but IMHO, this will be messy and will
cause conflicts during the merge window.

With regards to use python, well... I don't program on python, 
nor I'm interested on doing it ATM... I actually wrote one python script
a long time ago - that I had to fix to work on a newer python 2.x version,
as the unicode API was changed - and very likely it won't work on python 3
anymore, as lots of API got changed.

The thing is: perl is reliable enough for not needing to rewrite the script
every time someone comes with some crazy idea that would break the language
API and force changes at the scripts. So, I prefer to keep that script in a 
language that doesn't bite me on upgrades. As a plus, it doesn't forces
me to adopt random alien code style of 4 space indentations, and not use
tabs. But that's me.

So, I'm not against porting it. Yet, what would be the advantage of porting
it to Python? If there's no clear advantage, let's keep it in perl, as it
is easier to maintain.

Thanks,
Mauro

Re: parts of media docs sphinx re-building every time?

[Jani Nikula]

On Mon, 08 Aug 2016, Jani Nikula <jani.nikula@intel.com> wrote:
> I wonder if it's related to Documentation/media/Makefile... which I have
> to say I am not impressed by. I was really hoping we could build all the
> documentation by standalone sphinx-build invocation too, relying only on
> the conf.py so that e.g. Read the Docs can build the docs. Part of that
> motivation was to keep the build clean in makefiles, and handing the
> dependency tracking completely to Sphinx.
>
> I believe what's in Documentation/media/Makefile,
> Documentation/sphinx/parse-headers.pl, and
> Documentation/sphinx/kernel_include.py could be replaced by a Sphinx
> extension looking at the sources directly. (I presume kernel_include.py
> is mostly a workaround to keep out-of-tree builds working?)

Additionally, 'make pdfdocs' fails with e.g. 

/path/to/linux/Documentation/media/uapi/cec/cec-header.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/cec.h.rst'.
/path/to/linux/Documentation/media/uapi/dvb/audio_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/audio.h.rst'.
/path/to/linux/Documentation/media/uapi/dvb/ca_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/ca.h.rst'.
/path/to/linux/Documentation/media/uapi/dvb/dmx_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/dmx.h.rst'.

because the makefile hack is only done on htmldocs target.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: parts of media docs sphinx re-building every time?

[Jani Nikula]

On Mon, 08 Aug 2016, Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> The goal of Documentation/sphinx/parse-headers.pl script is to generate
> such parsed headers, with the cross-references modified by an exceptions
> file at Documentation/media/*.h.rst.exceptions.

Would you be so kind as to state in a few lines what you want to
achieve? I can guess based on the current solution, but I'd like to hear
it from you. Please leave out rants about tools and languages etc. so we
can focus on the problem statement, and try to figure out the best
overall solution.

Thanks,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: parts of media docs sphinx re-building every time?

[Mauro Carvalho Chehab]

Em Wed, 10 Aug 2016 10:42:27 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Mon, 08 Aug 2016, Jani Nikula <jani.nikula@intel.com> wrote:
> > I wonder if it's related to Documentation/media/Makefile... which I have
> > to say I am not impressed by. I was really hoping we could build all the
> > documentation by standalone sphinx-build invocation too, relying only on
> > the conf.py so that e.g. Read the Docs can build the docs. Part of that
> > motivation was to keep the build clean in makefiles, and handing the
> > dependency tracking completely to Sphinx.
> >
> > I believe what's in Documentation/media/Makefile,
> > Documentation/sphinx/parse-headers.pl, and
> > Documentation/sphinx/kernel_include.py could be replaced by a Sphinx
> > extension looking at the sources directly. (I presume kernel_include.py
> > is mostly a workaround to keep out-of-tree builds working?)  
> 
> Additionally, 'make pdfdocs' fails with e.g. 
> 
> /path/to/linux/Documentation/media/uapi/cec/cec-header.rst:9: SEVERE: Problems with "kernel-include" directive path:
> InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/cec.h.rst'.
> /path/to/linux/Documentation/media/uapi/dvb/audio_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
> InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/audio.h.rst'.
> /path/to/linux/Documentation/media/uapi/dvb/ca_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
> InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/ca.h.rst'.
> /path/to/linux/Documentation/media/uapi/dvb/dmx_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
> InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/dmx.h.rst'.
> 
> because the makefile hack is only done on htmldocs target.

Yeah, we need to call the media Makefile on other targets. I'll see if I can
write a quick fix for that. Please notice, however, that I'm on a trip this
week, with not much time available for writing patches.

Yet, you probably missed some e-mails while you were in vacations.
rst2pdf is not capable of handling complex documents. It crashes
with the media docs. See this patch:
	https://patchwork.kernel.org/patch/9231687/

There is a thread about that too... but, after checking my backlogs,
I noticed that this was not c/c to the ML, with is a shame :(

I'm enclosing the part of the thread where we've discussed about
rst2pdf, that resulted on the above patch.

In summary:

- rst2pdf is completely broken with Sphinx 1.3.x;
- rst2pdf doesn't handle complex documents, even with Sphinx 1.4.5;
- rst2pdf always report a successful build, even when it fails;
- Markus recommended removing the media book from the build, as he doesn't
  know any fixup or replacement for the tool.

That's why I came up with that patch, and recommended to even disable
pdf production at the patch comments while we don't have a sane alternative.

Regards,
Mauro


Am 13.07.2016 um 17:41 schrieb Mauro Carvalho Chehab <mchehab@osg.samsung.com>:

> Em Wed, 13 Jul 2016 17:22:05 +0200
> Markus Heiser <markus.heiser@darmarit.de> escreveu:
>   
>> Am 13.07.2016 um 14:40 schrieb Mauro Carvalho Chehab <mchehab@osg.samsung.com>:
>>   
>>> Em Thu, 7 Jul 2016 10:12:36 +0200
>>> Markus Heiser <markus.heiser@darmarit.de> escreveu:

>>> With Sphinx version 1.4.4/1.4.5, no errors were produced from media
>>> DocBook, with is great.
>>> 
>>> Yet, trying to use pdfdocs produce this error:
>>> 
>>> ERROR] pdfbuilder.py:130 too many values to unpack
>>> Traceback (most recent call last):
>>> File "/home/mchehab/.local/lib/python2.7/site-packages/rst2pdf/pdfbuilder.py", line 122, in write
>>>  appendices=opts.get('pdf_appendices', self.config.pdf_appendices) or [])
>>> File "/home/mchehab/.local/lib/python2.7/site-packages/rst2pdf/pdfbuilder.py", line 209, in assemble_doctree
>>>  index_nodes=genindex_nodes(genindex)
>>> File "/home/mchehab/.local/lib/python2.7/site-packages/rst2pdf/pdfbuilder.py", line 385, in genindex_nodes
>>>  for entryname, (links, subitems) in entries:
>>> ValueError: too many values to unpack
>>> FAILED
>>> build succeeded.
>>> 
>>> (for rst2pdf to work, I had to use pip2, as the version it downloaded
>>> doesn't run with python3)    
>> 
>> There was a time, I thought this rst2pdf tool could be a solution
>> which only needs some bug fixes, but in the meantime I have to
>> say, that rst2pdf is "broken by design".
>> 
>> This is very sad, but after dig into rst2pdf to fix some bugs I had 
>> to realized that it has tons of access on non existing properties. 
>> 
>> https://github.com/rst2pdf/rst2pdf/issues/556#issuecomment-228779542
>> 
>> Getting a build without an exception is just a game of luck. :-(
>> 
>> Sorry for the bad news ...  
> 
> Gah, that sucks. It sucks even more that sphinx is uncapable of
> detecting that something got wrong, keep saying that:
> 	"build succedded"
> 
> and keeping the zero byte pdf stored there.  

It is not only sphinx, it is the rst2pdf extension which does not throw 
a SystemMessagePropagation exception on fatal errors.

> Are there any other alternative to produce pdfs?  

Yes, the sphinx-doc build-in LaTeX builder

* http://www.sphinx-doc.org/en/stable/config.html#options-for-latex-output

But it has some drawbacks, e.g. it produce LaTeX for the pdfTeX engine.
LaTeX is by default ASCII and it needs some "inputenc" to supportma wider
range of characters. This is not very helpful if you have a toolchain
in an international community.

The alternative to LaTeX is to use the XeTeX engine, which supports UTF-8
encoded input by default and supports TrueType/OpenType fonts directly.
Thats why I started to write a XeLaTeX builder ...

* https://github.com/return42/sphkerneldoc/blob/master/scripts/site-python/xelatex_ext/__init__.py#L15 

... but I can't predict when this will be finished ...

However which tool is used, my experience is, that building
PDF (books) with a minimum of quality is not simple.
Layout width tables, split table content over pages, switch
from landscape to portrait and versus, the flow of objects etc.
.. all this will need some manually interventions.  

> I suspect that Jani and Jon may not be happy by the fact that the media
> book causes pdf generation to break. So, maybe we'll need some sort of
> "magic" to make sphinx to exclude the build for the media docbook,
> if pdf was requested.  

No magic, only structure ;-)

in the conf.py you will find the settings (similar to man page settings)

pdf_documents = [
   ('index', u'Kernel', u'Kernel', u'J. Random Bozo'),
]

With "index", the PDF is build for the root node, means all
books and text files in one big PDF ... IMHO (see above)
this will never work.

Since we have this structure:


.. toctree::
  :maxdepth: 2

  kernel-documentation
  media/media_uapi

I recommend to edit the pdf_documents value to:

pdf_documents = [
   ('kernel-documentation', u'Kernel', u'Kernel', u'J. Random Bozo'),
]

this should prevent building PDF from media book

-- Markus --

-- 

Cheers,
Mauro

Re: parts of media docs sphinx re-building every time?

[Mauro Carvalho Chehab]

Em Mon, 08 Aug 2016 18:37:38 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> Hi Mauro & co -
> 
> I just noticed running 'make htmldocs' rebuilds parts of media docs
> every time on repeated runs. This shouldn't happen. Please investigate.

I was unable to reproduce it here. Are you passing any special options
to the building system?

According to media Makefile, it should run the script only on four
conditions:
	- if the dynamically-generated rst file is not found (e. g. after
	  make cleandocs);
	- if the header file is changed;
	- if the exceptions file changes;
	- if the perl parser is changed.

All rules are like:

$(BUILDDIR)/audio.h.rst: ${UAPI}/dvb/audio.h ${PARSER} $(SRC_DIR)/audio.h.rst.exceptions

Regards,
Mauro



Cheers,
Mauro

Re: parts of media docs sphinx re-building every time?

[Mauro Carvalho Chehab]

Em Wed, 10 Aug 2016 10:42:27 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Mon, 08 Aug 2016, Jani Nikula <jani.nikula@intel.com> wrote:
> > I wonder if it's related to Documentation/media/Makefile... which I have
> > to say I am not impressed by. I was really hoping we could build all the
> > documentation by standalone sphinx-build invocation too, relying only on
> > the conf.py so that e.g. Read the Docs can build the docs. Part of that
> > motivation was to keep the build clean in makefiles, and handing the
> > dependency tracking completely to Sphinx.
> >
> > I believe what's in Documentation/media/Makefile,
> > Documentation/sphinx/parse-headers.pl, and
> > Documentation/sphinx/kernel_include.py could be replaced by a Sphinx
> > extension looking at the sources directly. (I presume kernel_include.py
> > is mostly a workaround to keep out-of-tree builds working?)  
> 
> Additionally, 'make pdfdocs' fails with e.g. 
> 
> /path/to/linux/Documentation/media/uapi/cec/cec-header.rst:9: SEVERE: Problems with "kernel-include" directive path:
> InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/cec.h.rst'.
> /path/to/linux/Documentation/media/uapi/dvb/audio_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
> InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/audio.h.rst'.
> /path/to/linux/Documentation/media/uapi/dvb/ca_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
> InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/ca.h.rst'.
> /path/to/linux/Documentation/media/uapi/dvb/dmx_h.rst:9: SEVERE: Problems with "kernel-include" directive path:
> InputError: [Errno 2] No such file or directory: '/path/to/linux/Documentation/output/dmx.h.rst'.
> 
> because the makefile hack is only done on htmldocs target.

Hit send too early... this is what happens here if I run make with the
documents already built:


$ make  htmldocs
make BUILDDIR=Documentation/output -f ./Documentation/media/Makefile htmldocs
make[2]: Nothing to be done for 'htmldocs'.
  SPHINX  htmldocs
Running Sphinx v1.4.5
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
no targets are out of date.
build succeeded.

(docbook build messages suppressed


Cheers,
Mauro

Re: parts of media docs sphinx re-building every time?

[Mauro Carvalho Chehab]

Em Wed, 10 Aug 2016 10:46:44 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Mon, 08 Aug 2016, Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> > The goal of Documentation/sphinx/parse-headers.pl script is to generate
> > such parsed headers, with the cross-references modified by an exceptions
> > file at Documentation/media/*.h.rst.exceptions.  
> 
> Would you be so kind as to state in a few lines what you want to
> achieve? I can guess based on the current solution, but I'd like to hear
> it from you. Please leave out rants about tools and languages etc. so we
> can focus on the problem statement, and try to figure out the best
> overall solution.

It is basically what's written above: to produce a cross-referenced output
document from a source, were, clicking on the API symbols will navigate to
the place where the symbol was documented.

It should allow to add exceptions, as we don't want to add cross-references
for legacy symbols or helper macros, for example. 

This is a good example on what we want:
	https://linuxtv.org/downloads/v4l-dvb-apis-new/media/uapi/rc/lirc-header.html#lirc-h

The only parts of the header file that will have a different color will be
the symbols defined at the API.

This way, it is easy for us to visually discover what stuff is not yet
documented (like LIRC_MODE2_SPACE - that should be documented - and
_LINUX_LIRC_H macro - that should not be documented).
When something is not cross-referenced there, there's a single place to
look that will explain why this was not documented (at the exceptions file).

The main goal for documentation writers is to use the header
file to identify the documentation gaps. For code developers, the header
file works like an index to the document, where all the API can be seen
altogether, and more details can easily be obtained by clicking at the links.

Regard


Cheers,
Mauro

Re: parts of media docs sphinx re-building every time?

[Jani Nikula]

On Mon, 08 Aug 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Hi Jani,
>
> Am 08.08.2016 um 17:37 schrieb Jani Nikula <jani.nikula@intel.com>:
>
>> 
>> Hi Mauro & co -
>> 
>> I just noticed running 'make htmldocs' rebuilds parts of media docs
>> every time on repeated runs. This shouldn't happen. Please investigate.
>> 
>> I wonder if it's related to Documentation/media/Makefile... which I have
>> to say I am not impressed by. I was really hoping we could build all the
>> documentation by standalone sphinx-build invocation too, relying only on
>> the conf.py so that e.g. Read the Docs can build the docs. Part of that
>> motivation was to keep the build clean in makefiles, and handing the
>> dependency tracking completely to Sphinx.
>> 
>> I believe what's in Documentation/media/Makefile,
>> Documentation/sphinx/parse-headers.pl, and
>> Documentation/sphinx/kernel_include.py could be replaced by a Sphinx
>> extension looking at the sources directly.
>
> Yes, parse-headers.pl, kernel_include.py and media/Makefile are needed
> for one feature ... not very straight forward.
>
> If it makes sense to migrate the perl scripts functionality to a
> Sphinx extension, may I can help ... depends on what Mauro thinks.
>
> BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)

If I understand the need of all of this right, I think the cleanest and
fastest short term measure would be to make the kernel-include directive
extension do the same thing as the kernel-doc directive does: call the
perl script from the directive.

This lets you get rid of Documentation/media/Makefile and you don't have
to copy-paste all of Include.run method into kernel_include.py. You can
also get rid of specifying environment variables in rst files and
parsing them in the extension. We can get rid of the problematic
intermediate rst files. This design has been proven with the kernel-doc
extension and script already. It's much simpler.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: parts of media docs sphinx re-building every time?

[Mauro Carvalho Chehab]

Em Wed, 10 Aug 2016 12:15:34 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Mon, 08 Aug 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> > Hi Jani,
> >
> > Am 08.08.2016 um 17:37 schrieb Jani Nikula <jani.nikula@intel.com>:
> >  
> >> 
> >> Hi Mauro & co -
> >> 
> >> I just noticed running 'make htmldocs' rebuilds parts of media docs
> >> every time on repeated runs. This shouldn't happen. Please investigate.
> >> 
> >> I wonder if it's related to Documentation/media/Makefile... which I have
> >> to say I am not impressed by. I was really hoping we could build all the
> >> documentation by standalone sphinx-build invocation too, relying only on
> >> the conf.py so that e.g. Read the Docs can build the docs. Part of that
> >> motivation was to keep the build clean in makefiles, and handing the
> >> dependency tracking completely to Sphinx.
> >> 
> >> I believe what's in Documentation/media/Makefile,
> >> Documentation/sphinx/parse-headers.pl, and
> >> Documentation/sphinx/kernel_include.py could be replaced by a Sphinx
> >> extension looking at the sources directly.  
> >
> > Yes, parse-headers.pl, kernel_include.py and media/Makefile are needed
> > for one feature ... not very straight forward.
> >
> > If it makes sense to migrate the perl scripts functionality to a
> > Sphinx extension, may I can help ... depends on what Mauro thinks.
> >
> > BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)  
> 
> If I understand the need of all of this right, I think the cleanest and
> fastest short term measure would be to make the kernel-include directive
> extension do the same thing as the kernel-doc directive does: call the
> perl script from the directive.
> 
> This lets you get rid of Documentation/media/Makefile and you don't have
> to copy-paste all of Include.run method into kernel_include.py. You can
> also get rid of specifying environment variables in rst files and
> parsing them in the extension. We can get rid of the problematic
> intermediate rst files. This design has been proven with the kernel-doc
> extension and script already. It's much simpler.

Works for me. If someone comes with such patch, I'll happily ack it.

Cheers,
Mauro

Re: parts of media docs sphinx re-building every time?

[Jani Nikula]

On Wed, 10 Aug 2016, Mauro Carvalho Chehab <mchehab@infradead.org> wrote:
> Em Mon, 08 Aug 2016 18:37:38 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
>
>> Hi Mauro & co -
>> 
>> I just noticed running 'make htmldocs' rebuilds parts of media docs
>> every time on repeated runs. This shouldn't happen. Please investigate.
>
> I was unable to reproduce it here. Are you passing any special options
> to the building system?

Hmh, I can't reproduce this now either. I was able to hit this on
another machine consistently, even with 'make cleandocs' in
between. I'll check the environment on the other machine when I get my
hands on it.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: parts of media docs sphinx re-building every time?

[Markus Heiser]

Am 10.08.2016 um 11:22 schrieb Mauro Carvalho Chehab <mchehab@infradead.org>:

> Em Wed, 10 Aug 2016 12:15:34 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
> 
>> On Mon, 08 Aug 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>>> Hi Jani,
>>> 
>>> Am 08.08.2016 um 17:37 schrieb Jani Nikula <jani.nikula@intel.com>:
>>> 
>>>> 
>>>> Hi Mauro & co -
>>>> 
>>>> I just noticed running 'make htmldocs' rebuilds parts of media docs
>>>> every time on repeated runs. This shouldn't happen. Please investigate.
>>>> 
>>>> I wonder if it's related to Documentation/media/Makefile... which I have
>>>> to say I am not impressed by. I was really hoping we could build all the
>>>> documentation by standalone sphinx-build invocation too, relying only on
>>>> the conf.py so that e.g. Read the Docs can build the docs. Part of that
>>>> motivation was to keep the build clean in makefiles, and handing the
>>>> dependency tracking completely to Sphinx.
>>>> 
>>>> I believe what's in Documentation/media/Makefile,
>>>> Documentation/sphinx/parse-headers.pl, and
>>>> Documentation/sphinx/kernel_include.py could be replaced by a Sphinx
>>>> extension looking at the sources directly.  
>>> 
>>> Yes, parse-headers.pl, kernel_include.py and media/Makefile are needed
>>> for one feature ... not very straight forward.
>>> 
>>> If it makes sense to migrate the perl scripts functionality to a
>>> Sphinx extension, may I can help ... depends on what Mauro thinks.
>>> 
>>> BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)  
>> 
>> If I understand the need of all of this right, I think the cleanest and
>> fastest short term measure would be to make the kernel-include directive
>> extension do the same thing as the kernel-doc directive does: call the
>> perl script from the directive.
>> 
>> This lets you get rid of Documentation/media/Makefile and you don't have
>> to copy-paste all of Include.run method into kernel_include.py. You can
>> also get rid of specifying environment variables in rst files and
>> parsing them in the extension. We can get rid of the problematic
>> intermediate rst files. This design has been proven with the kernel-doc
>> extension and script already. It's much simpler.
> 
> Works for me. If someone comes with such patch, I'll happily ack it.
> 
> Cheers,
> Mauro

Hi Jani & Mauro,

I will give it a try ... but currently I'am working in some other tasks.
I think next week I will find some time to implement.

-- Markus --

Re: parts of media docs sphinx re-building every time?

[Mauro Carvalho Chehab]

Em Wed, 10 Aug 2016 11:58:48 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 10.08.2016 um 11:22 schrieb Mauro Carvalho Chehab <mchehab@infradead.org>:
> 
> > Em Wed, 10 Aug 2016 12:15:34 +0300
> > Jani Nikula <jani.nikula@intel.com> escreveu:
> >   
> >> On Mon, 08 Aug 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:  
> >>> Hi Jani,
> >>> 
> >>> Am 08.08.2016 um 17:37 schrieb Jani Nikula <jani.nikula@intel.com>:
> >>>   
> >>>> 
> >>>> Hi Mauro & co -
> >>>> 
> >>>> I just noticed running 'make htmldocs' rebuilds parts of media docs
> >>>> every time on repeated runs. This shouldn't happen. Please investigate.
> >>>> 
> >>>> I wonder if it's related to Documentation/media/Makefile... which I have
> >>>> to say I am not impressed by. I was really hoping we could build all the
> >>>> documentation by standalone sphinx-build invocation too, relying only on
> >>>> the conf.py so that e.g. Read the Docs can build the docs. Part of that
> >>>> motivation was to keep the build clean in makefiles, and handing the
> >>>> dependency tracking completely to Sphinx.
> >>>> 
> >>>> I believe what's in Documentation/media/Makefile,
> >>>> Documentation/sphinx/parse-headers.pl, and
> >>>> Documentation/sphinx/kernel_include.py could be replaced by a Sphinx
> >>>> extension looking at the sources directly.    
> >>> 
> >>> Yes, parse-headers.pl, kernel_include.py and media/Makefile are needed
> >>> for one feature ... not very straight forward.
> >>> 
> >>> If it makes sense to migrate the perl scripts functionality to a
> >>> Sphinx extension, may I can help ... depends on what Mauro thinks.
> >>> 
> >>> BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)    
> >> 
> >> If I understand the need of all of this right, I think the cleanest and
> >> fastest short term measure would be to make the kernel-include directive
> >> extension do the same thing as the kernel-doc directive does: call the
> >> perl script from the directive.
> >> 
> >> This lets you get rid of Documentation/media/Makefile and you don't have
> >> to copy-paste all of Include.run method into kernel_include.py. You can
> >> also get rid of specifying environment variables in rst files and
> >> parsing them in the extension. We can get rid of the problematic
> >> intermediate rst files. This design has been proven with the kernel-doc
> >> extension and script already. It's much simpler.  
> > 
> > Works for me. If someone comes with such patch, I'll happily ack it.
> > 
> > Cheers,
> > Mauro  
> 
> Hi Jani & Mauro,
> 
> I will give it a try ... but currently I'am working in some other tasks.
> I think next week I will find some time to implement.

Good enough to me, Thanks for looking into that!

Cheers,
Mauro

Re: parts of media docs sphinx re-building every time?

[Daniel Vetter]

On Wed, Aug 10, 2016 at 11:15 AM, Jani Nikula <jani.nikula@intel.com> wrote:
> On Mon, 08 Aug 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>> Hi Jani,
>>
>> Am 08.08.2016 um 17:37 schrieb Jani Nikula <jani.nikula@intel.com>:
>>
>>>
>>> Hi Mauro & co -
>>>
>>> I just noticed running 'make htmldocs' rebuilds parts of media docs
>>> every time on repeated runs. This shouldn't happen. Please investigate.
>>>
>>> I wonder if it's related to Documentation/media/Makefile... which I have
>>> to say I am not impressed by. I was really hoping we could build all the
>>> documentation by standalone sphinx-build invocation too, relying only on
>>> the conf.py so that e.g. Read the Docs can build the docs. Part of that
>>> motivation was to keep the build clean in makefiles, and handing the
>>> dependency tracking completely to Sphinx.
>>>
>>> I believe what's in Documentation/media/Makefile,
>>> Documentation/sphinx/parse-headers.pl, and
>>> Documentation/sphinx/kernel_include.py could be replaced by a Sphinx
>>> extension looking at the sources directly.
>>
>> Yes, parse-headers.pl, kernel_include.py and media/Makefile are needed
>> for one feature ... not very straight forward.
>>
>> If it makes sense to migrate the perl scripts functionality to a
>> Sphinx extension, may I can help ... depends on what Mauro thinks.
>>
>> BTW: parse-headers.pl is not the only perl script I like to migrate to py ;)
>
> If I understand the need of all of this right, I think the cleanest and
> fastest short term measure would be to make the kernel-include directive
> extension do the same thing as the kernel-doc directive does: call the
> perl script from the directive.
>
> This lets you get rid of Documentation/media/Makefile and you don't have
> to copy-paste all of Include.run method into kernel_include.py. You can
> also get rid of specifying environment variables in rst files and
> parsing them in the extension. We can get rid of the problematic
> intermediate rst files. This design has been proven with the kernel-doc
> extension and script already. It's much simpler.

I looked a bit at this and seems interesting ... a few questions:
- Are you using this just for uapi headers or also for other bits?
- My concern with out-of-line docs is always that people forget to
update them. How do you enforce that in the media subsystem?
- Atm we don't have any formal way to document drm ioctl, and this
could be a possible approach. Would it be possible to share this with
other subsystems, maybe extended/polished, perhaps even as the
official way to document uapi headers?

Just some thoughts, orthogonal to the discussion at hand here.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: parts of media docs sphinx re-building every time?

[Jonathan Corbet]

On Wed, 10 Aug 2016 12:23:16 +0300
Jani Nikula <jani.nikula@intel.com> wrote:

> >> I just noticed running 'make htmldocs' rebuilds parts of media docs
> >> every time on repeated runs. This shouldn't happen. Please investigate.  
> >
> > I was unable to reproduce it here. Are you passing any special options
> > to the building system?  
> 
> Hmh, I can't reproduce this now either. I was able to hit this on
> another machine consistently, even with 'make cleandocs' in
> between. I'll check the environment on the other machine when I get my
> hands on it.

Just FWIW, I've been trying to find a moment to come back to this because
I couldn't reproduce it either...

jon

Re: parts of media docs sphinx re-building every time?

[Markus Heiser]

Am 10.08.2016 um 15:46 schrieb Jonathan Corbet <corbet@lwn.net>:

> On Wed, 10 Aug 2016 12:23:16 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
> 
>>>> I just noticed running 'make htmldocs' rebuilds parts of media docs
>>>> every time on repeated runs. This shouldn't happen. Please investigate.  
>>> 
>>> I was unable to reproduce it here. Are you passing any special options
>>> to the building system?  
>> 
>> Hmh, I can't reproduce this now either. I was able to hit this on
>> another machine consistently, even with 'make cleandocs' in
>> between. I'll check the environment on the other machine when I get my
>> hands on it.
> 
> Just FWIW, I've been trying to find a moment to come back to this because
> I couldn't reproduce it either...
> 
> jon


Hmm, I have had problems with the relative BUILDDIR make environment, so I switched
to absolute pathname .. see my "more generic way" patch:

 htmldocs:
-	$(MAKE) BUILDDIR=$(BUILDDIR) -f $(srctree)/Documentation/media/Makefile $@
+	$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) -f $(srctree)/Documentation/media/Makefile $@

could this the reason why you can't reproduce it?

My problem was vice versa, if I called "make O=/tmp/kernel htmldocs" after
a make with normal output, the rst files has been found in Documents/output
and not regenerated in /tmp/kernel/Documents/output. 

And with "make O=/tmp/kernel clean", the rst files in Documents/output resists.

This was very confusing to me, so I changed it to absolute pathname.

--Markus--