Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be

[Jani Nikula <jani.nikula@intel.com>]

On Thu, 23 Jun 2022, Mauro Carvalho Chehab <mchehab@kernel.org> wrote:
> Em Thu, 23 Jun 2022 12:18:50 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
>
>> On Sat, 18 Jun 2022, Mauro Carvalho Chehab <mchehab@kernel.org> wrote:
>> > The big missing feature on Sphinx itself is with regards to per-C-type
>> > domain. So, if we have one struct and one function both called "foo",
>> > the cross-references will be broken. This issue is known since Sphinx
>> > 3.1, and there are already patches fixing it since then (I remember
>> > testing them) but, up to today, the Sphinx upstream patches meant
>> > to fix it weren't applied yet (as far as I can tell).  
>> 
>> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#namespacing
>> 
>> Integrating that needs to be done carefully, though, to not make a mess
>> of it.
>> 
>> > One thing that probably be solved on a different way is to have
>> > a better solution for things like:
>> >
>> > 	Functions for feature 1
>> > 	=======================
>> >
>> > 	.. kernel-doc:: include/some_header.h
>> > 	   :doc: Feature 1
>> >
>> > 	.. kernel-doc:: include/some_header.h
>> > 	   :functions:
>> > 		func1
>> > 		func2
>> >
>> > 	Functions for feature 2
>> > 	=======================
>> >
>> > 	.. kernel-doc:: include/some_header.h
>> > 	   :doc: Feature 2
>> >
>> > 	.. kernel-doc:: include/some_header.h
>> > 	   :functions:
>> > 		func3
>> > 		func4  
>> 
>> Yeah, currently that leads to parsing the header four times by
>> kernel-doc the script.
>
> Yes.
>
>> The solution would be to finally convert the
>> script to a proper python Sphinx extension that can do caching. (This is
>> how it works in Hawkmoth, FWIW.)
>
> That's one solution, but see: there is already a python extension
> that currently calls kernel-doc everytime. It could, instead,
> cache the rst returned by its first run (or a parsed version of it)
> and use the cached results the other 3 times.

This would mean having kernel-doc the perl script output meta
information about the constructs, complicating it even more.

> Porting kernel-doc to python could be doable, but not trivial, due to several
> reasons:
>
> - it should keep running standalone, as otherwise debugging parsing issues
>   on kernel-doc would be a lot harder. In particular, kernel-doc --none is
>   really helpful to report kernel-doc tag errors;

This is trivial and sensible to do in the python extension too.

> - regressions will likely be introduced on a change like that;

Any change can cause regressions... but previously a diff on the html
outputs generated by both versions has been an effective way of
verifying the changes.

> - regular contributors to kernel-doc will need to ramp up with the newer
>   version;

You say it like it's a bad thing. :p

Sometimes I regret not throwing kernel-doc the script in the curb while
doing the Sphinx conversion.

> - a port like that could increase the script run time, as the
>   optimizations and regular expressions there could behave different on
>   python.

This is certainly possible. It's somewhat countered by the python
extension being able to cache stuff.

>
>> > Perhaps we could change Kernel-doc in a way that doing just:
>> >
>> > 	.. kernel-doc:: include/some_header.h
>> >
>> > would be enough.  
>> 
>> The order in nicely flowing documentation is not necessarily the same as
>> the order in nicely flowing source code. I expect it to be much more
>> acceptable to tweak the rst to achieve this than to do source code
>> rearrangement to generate nice documentation.
>
> True, but independently if the script would be rewritten in python or not,
> one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention 
> there the symbols that belong to each part of the document, e. g. something
> like:
>
> 	/**
> 	 * DOC: foo
> 	 *
> 	 * Some comments...
> 	 *
> 	 * symbols:
> 	 *     foo
> 	 *     bar
> 	 */
>

I believe preprocessing of what's supposed to be rst leads to
problems. It's difficult to define proper escaping to ensure stuff gets
parsed at the correct stage in the correct way. It should just be rst.

Part of the reason the old docbook thing was such a big mess were the
"impedance mismatches" at each separate stage of processing.

In general, stick to the mechanisms provided by rst and Sphinx. Extend
on them using Sphinx extensions, not by manually parsing rst.

BR,
Jani.

> One advantage is that all documentation will be on a single place,
> so hopefully it would be easier to maintain.
>
> Also, on documents using `DOC:` with such new `symbols` tag, kernel-doc 
> could validate if all documented symbols are singly included at all `DOC:`
> sessions and if any symbols are missed/renamed/removed.
>
> Regards,
> Mauro

-- 
Jani Nikula, Intel Open Source Graphics Center