Functions and data structure cross references with Sphinx

Functions and data structure cross references with Sphinx

[Mauro Carvalho Chehab]

There's one remaining major issue I noticed after the conversion of the
media books to Sphinx:

While sphinx complains if a cross-reference (using :ref:) points to an
undefined reference, the same doesn't happen if the reference uses
:c:func: and :c:type:.

In practice, it means that, if we do some typo there, or if we forget to
add the function/struct prototype (or use the wrong domain, like :cpp:),
Sphinx won't generate the proper cross-reference, nor warning the user.

That's specially bad for media, as, while we're using the c domain for
the kAPI and driver-specific books, we need to use the cpp domain on the 
uAPI book - as the c domain doesn't allow multiple declarations for
syscalls, and we have multiple pages for read, write, open, close, 
poll and ioctl.

It would be good to have a way to run Sphinx on some "pedantic"
mode or have something similar to xmlint that would be complaining
about invalid c/cpp domain references.

Thanks,
Mauro

Re: Functions and data structure cross references with Sphinx

[Markus Heiser]

Am 01.08.2016 um 13:25 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:

> There's one remaining major issue I noticed after the conversion of the
> media books to Sphinx:
> 
> While sphinx complains if a cross-reference (using :ref:) points to an
> undefined reference, the same doesn't happen if the reference uses
> :c:func: and :c:type:.
> 
> In practice, it means that, if we do some typo there, or if we forget to
> add the function/struct prototype (or use the wrong domain, like :cpp:),
> Sphinx won't generate the proper cross-reference, nor warning the user.
> 
> That's specially bad for media, as, while we're using the c domain for
> the kAPI and driver-specific books, we need to use the cpp domain on the 
> uAPI book - as the c domain doesn't allow multiple declarations for
> syscalls, and we have multiple pages for read, write, open, close, 
> poll and ioctl.
> 
> It would be good to have a way to run Sphinx on some "pedantic"
> mode or have something similar to xmlint that would be complaining
> about invalid c/cpp domain references.
> 
> Thanks,
> Mauro

Hi Mauro,

there is a nit-picky mode [1], which could be activated by setting
"nitpicky=True" in the conf.py or alternative, set "-n" to the 
SPHINXOPTS:

  make SPHINXOPTS=-n htmldocs

Within nit-picky mode, Sphinx will warn about **all** references. This
might be more then you want. For this, in the conf.py you could
assemble a "nitpick_ignore" list [2]. But I think, assemble the
ignore list is quite a lot of work.

[1] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpicky
[2] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpick_ignore

-- Markus 

Re: Functions and data structure cross references with Sphinx

[Mauro Carvalho Chehab]

Em Fri, 5 Aug 2016 09:29:23 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 01.08.2016 um 13:25 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> 
> > There's one remaining major issue I noticed after the conversion of the
> > media books to Sphinx:
> > 
> > While sphinx complains if a cross-reference (using :ref:) points to an
> > undefined reference, the same doesn't happen if the reference uses
> > :c:func: and :c:type:.
> > 
> > In practice, it means that, if we do some typo there, or if we forget to
> > add the function/struct prototype (or use the wrong domain, like :cpp:),
> > Sphinx won't generate the proper cross-reference, nor warning the user.
> > 
> > That's specially bad for media, as, while we're using the c domain for
> > the kAPI and driver-specific books, we need to use the cpp domain on the 
> > uAPI book - as the c domain doesn't allow multiple declarations for
> > syscalls, and we have multiple pages for read, write, open, close, 
> > poll and ioctl.
> > 
> > It would be good to have a way to run Sphinx on some "pedantic"
> > mode or have something similar to xmlint that would be complaining
> > about invalid c/cpp domain references.
> > 
> > Thanks,
> > Mauro  
> 
> Hi Mauro,
> 
> there is a nit-picky mode [1], which could be activated by setting
> "nitpicky=True" in the conf.py or alternative, set "-n" to the 
> SPHINXOPTS:
> 
>   make SPHINXOPTS=-n htmldocs
> 
> Within nit-picky mode, Sphinx will warn about **all** references. This
> might be more then you want. For this, in the conf.py you could
> assemble a "nitpick_ignore" list [2]. But I think, assemble the
> ignore list is quite a lot of work.
> 
> [1] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpicky
> [2] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpick_ignore

Yeah, this is what I was looking for.

We indeed want to use this option on media, but there are some things
that need to be addressed. From some quick tests here, what I noticed
is:

1) Sphinx will generate several references that should be safely ignored
for everyone, like "enum", "u32", "int32_t", "bool", "NULL", etc;

2) the usage of cpp domain for system calls make several symbols to
not match, as the cpp function prototype will generate cross references
for the cpp domain, instead of using the c domain. So, we need a
better way to fix it using the c domain, or convert everything to the
cpp domain;

3) The references generated from the header files I'm parsing don't
use the c (or cpp) domain. They're declared at the media book as a
normal reference:
	Documentation/media/uapi/v4l/field-order.rst:.. _v4l2-field:

and cross-referenced with:
	ref:`v4l2_field <v4l2-field>`

Is there a way to change it to the c domain?


4) there are several references that, IMHO, should be nitpick-ignored
only when the book is generated stand alone. For example, at the
media docbooks, we have references for things like:

- pci_dev, mutex, off_t, container_of, etc - those are generic
  references for the symbols that every driver uses, but, as we
  don't have the books with those converted yet, nitpick complains.
  Once we have such references, they should be ignored *only* when
  the book is generated standalone. As those are "core" symbols,
  they should be already be documented, but the book was not
  ported from DocBook yet. Once we have everything ported to
  Sphinx, I would expect that they all will vanish (and, if not,
  IMHO, documenting them should be prioritized).

- References for subsystem-specific symbols like: spi_board_info,
  led_classdev_flash, i2c_adapter, etc. Those would require that
  the maintainers of the specific subsystems to add documentation
  to them, as I bet several such symbols won't be currently
  documented. So, even after the port, I afraid that we'll still
  have several such symbols missing.

To address (3), we need different sets of nitpick ignore lists.

At least in my case, I have two different procedures, depending
on the time at the Kernel release cycle:

a) daily patch merge workflow
   --------------------------

In any case, for (3), I don't want to see those warnings during
my daily patch handling process where I rebuild documentation for
every patch that touches a documented file. I want to see only
things like:
	Documentation/media/uapi/v4l/hist-v4l2.rst:1295: WARNING: c:type reference target not found: struct v4l2_buffer

With indicates that a new patch would be introducing documentation
gaps.

So, we need a way to have a per-subsystem nitpick_ignore list
(or a way to pass such list via command line), for us to be able to
ignore "alien" symbols that aren't part of the subsystem we're
maintaining.

b) preparation for the merge window
   --------------------------------

Late at the patch handling cycle, I run a very long task here that
builds the media subsystem for 50+ different archs. I also check and
fix smatch/sparse warnings. During such time, I would love to view
the full list of missing symbols, in order to be able to handle
eventual cross-subsystem wrong references (or eventually help documenting
something that we use at the media subsystem).

So, I would need to use a different nitpick_ignore list.

Is there a way for us to specify the nitpick_ignore list (or a different
conf.py) via command line?

Btw, I'm enclosing a patch adding several of those references that are
alien to the media subsystem and currently causes nitpick complains.


doc-rst: ignore several undefined symbols in nitpick mode
    
using Sphinx in nitpick mode is too verbose to make it useful.
So, we need to make it less verbose, in order to be able to
actually use it.
    
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>

diff --git a/Documentation/conf.py b/Documentation/conf.py
index 96b7aa66c89c..d1805b8710c3 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -419,3 +419,88 @@ pdf_documents = [
 # line arguments.
 kerneldoc_bin = '../scripts/kernel-doc'
 kerneldoc_srctree = '..'
+
+#
+# It is possible to run Sphinx in nickpick mode with:
+#	make SPHINXOPTS=-n htmldocs
+# In such case, it will complain about lots of missing references that
+#	1) are just typedefs like: bool, __u32, etc;
+#	2) It will complain for things like: enum, NULL;
+#	3) It will complain for symbols that should be on different
+#	   books (but currently aren't ported to ReST)
+# The list below has a list of such symbols to be ignored in nitpick mode
+#
+nitpick_ignore = [
+	("c:func", "clock_gettime"),
+	("c:func", "close"),
+	("c:func", "container_of"),
+	("c:func", "determine_valid_ioctls"),
+	("c:func", "ERR_PTR"),
+	("c:func", "ioctl"),
+	("c:func", "IS_ERR"),
+	("c:func", "mmap"),
+	("c:func", "open"),
+	("c:func", "pci_name"),
+	("c:func", "poll"),
+	("c:func", "PTR_ERR"),
+	("c:func", "read"),
+	("c:func", "release"),
+	("c:func", "set"),
+	("c:func", "struct fd_set"),
+	("c:func", "struct pollfd"),
+	("c:func", "usb_make_path"),
+	("c:func", "write"),
+	("c:type", "atomic_t"),
+	("c:type", "bool"),
+	("c:type", "buf_queue"),
+	("c:type", "device"),
+	("c:type", "device_driver"),
+	("c:type", "device_node"),
+	("c:type", "enum"),
+	("c:type", "file"),
+	("c:type", "i2c_adapter"),
+	("c:type", "i2c_board_info"),
+	("c:type", "i2c_client"),
+	("c:type", "ktime_t"),
+	("c:type", "led_classdev_flash"),
+	("c:type", "list_head"),
+	("c:type", "lock_class_key"),
+	("c:type", "module"),
+	("c:type", "mutex"),
+	("c:type", "pci_dev"),
+	("c:type", "pdvbdev"),
+	("c:type", "poll_table_struct"),
+	("c:type", "s32"),
+	("c:type", "s64"),
+	("c:type", "sd"),
+	("c:type", "spi_board_info"),
+	("c:type", "spi_device"),
+	("c:type", "spi_master"),
+	("c:type", "struct fb_fix_screeninfo"),
+	("c:type", "struct pollfd"),
+	("c:type", "struct timeval"),
+	("c:type", "struct video_capability"),
+	("c:type", "u16"),
+	("c:type", "u32"),
+	("c:type", "u64"),
+	("c:type", "u8"),
+	("c:type", "union"),
+	("c:type", "usb_device"),
+
+	("cpp:type", "boolean"),
+	("cpp:type", "fd"),
+	("cpp:type", "fd_set"),
+	("cpp:type", "int16_t"),
+	("cpp:type", "NULL"),
+	("cpp:type", "off_t"),
+	("cpp:type", "pollfd"),
+	("cpp:type", "size_t"),
+	("cpp:type", "ssize_t"),
+	("cpp:type", "timeval"),
+	("cpp:type", "__u16"),
+	("cpp:type", "__u32"),
+	("cpp:type", "__u64"),
+	("cpp:type", "uint16_t"),
+	("cpp:type", "uint32_t"),
+	("cpp:type", "video_system_t"),
+]

Re: Functions and data structure cross references with Sphinx

[Markus Heiser]

Am 05.08.2016 um 12:47 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:

> Em Fri, 5 Aug 2016 09:29:23 +0200
> Markus Heiser <markus.heiser@darmarit.de> escreveu:
> 
>> Am 01.08.2016 um 13:25 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
>> 
>>> There's one remaining major issue I noticed after the conversion of the
>>> media books to Sphinx:
>>> 
>>> While sphinx complains if a cross-reference (using :ref:) points to an
>>> undefined reference, the same doesn't happen if the reference uses
>>> :c:func: and :c:type:.
>>> 
>>> In practice, it means that, if we do some typo there, or if we forget to
>>> add the function/struct prototype (or use the wrong domain, like :cpp:),
>>> Sphinx won't generate the proper cross-reference, nor warning the user.
>>> 
>>> That's specially bad for media, as, while we're using the c domain for
>>> the kAPI and driver-specific books, we need to use the cpp domain on the 
>>> uAPI book - as the c domain doesn't allow multiple declarations for
>>> syscalls, and we have multiple pages for read, write, open, close, 
>>> poll and ioctl.
>>> 
>>> It would be good to have a way to run Sphinx on some "pedantic"
>>> mode or have something similar to xmlint that would be complaining
>>> about invalid c/cpp domain references.
>>> 
>>> Thanks,
>>> Mauro  
>> 
>> Hi Mauro,
>> 
>> there is a nit-picky mode [1], which could be activated by setting
>> "nitpicky=True" in the conf.py or alternative, set "-n" to the 
>> SPHINXOPTS:
>> 
>> make SPHINXOPTS=-n htmldocs
>> 
>> Within nit-picky mode, Sphinx will warn about **all** references. This
>> might be more then you want. For this, in the conf.py you could
>> assemble a "nitpick_ignore" list [2]. But I think, assemble the
>> ignore list is quite a lot of work.
>> 
>> [1] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpicky
>> [2] http://www.sphinx-doc.org/en/stable/config.html#confval-nitpick_ignore
> 
> Yeah, this is what I was looking for.
> 
> We indeed want to use this option on media, but there are some things
> that need to be addressed. From some quick tests here, what I noticed
> is:
> 
> 1) Sphinx will generate several references that should be safely ignored
> for everyone, like "enum", "u32", "int32_t", "bool", "NULL", etc;
> 
> 2) the usage of cpp domain for system calls make several symbols to
> not match, as the cpp function prototype will generate cross references
> for the cpp domain, instead of using the c domain. So, we need a
> better way to fix it using the c domain, or convert everything to the
> cpp domain;
> 
> 3) The references generated from the header files I'm parsing don't
> use the c (or cpp) domain. They're declared at the media book as a
> normal reference:
> 	Documentation/media/uapi/v4l/field-order.rst:.. _v4l2-field:
> 
> and cross-referenced with:
> 	ref:`v4l2_field <v4l2-field>`
> 
> Is there a way to change it to the c domain?
> 
> 
> 4) there are several references that, IMHO, should be nitpick-ignored
> only when the book is generated stand alone. For example, at the
> media docbooks, we have references for things like:
> 
> - pci_dev, mutex, off_t, container_of, etc - those are generic
> references for the symbols that every driver uses, but, as we
> don't have the books with those converted yet, nitpick complains.
> Once we have such references, they should be ignored *only* when
> the book is generated standalone. As those are "core" symbols,
> they should be already be documented, but the book was not
> ported from DocBook yet. Once we have everything ported to
> Sphinx, I would expect that they all will vanish (and, if not,
> IMHO, documenting them should be prioritized).
> 
> - References for subsystem-specific symbols like: spi_board_info,
> led_classdev_flash, i2c_adapter, etc. Those would require that
> the maintainers of the specific subsystems to add documentation
> to them, as I bet several such symbols won't be currently
> documented. So, even after the port, I afraid that we'll still
> have several such symbols missing.
> 
> To address (3), we need different sets of nitpick ignore lists.
> 
> At least in my case, I have two different procedures, depending
> on the time at the Kernel release cycle:
> 
> a) daily patch merge workflow
> --------------------------
> 
> In any case, for (3), I don't want to see those warnings during
> my daily patch handling process where I rebuild documentation for
> every patch that touches a documented file. I want to see only
> things like:
> 	Documentation/media/uapi/v4l/hist-v4l2.rst:1295: WARNING: c:type reference target not found: struct v4l2_buffer
> 
> With indicates that a new patch would be introducing documentation
> gaps.
> 
> So, we need a way to have a per-subsystem nitpick_ignore list
> (or a way to pass such list via command line), for us to be able to
> ignore "alien" symbols that aren't part of the subsystem we're
> maintaining.
> 
> b) preparation for the merge window
> --------------------------------
> 
> Late at the patch handling cycle, I run a very long task here that
> builds the media subsystem for 50+ different archs. I also check and
> fix smatch/sparse warnings. During such time, I would love to view
> the full list of missing symbols, in order to be able to handle
> eventual cross-subsystem wrong references (or eventually help documenting
> something that we use at the media subsystem).
> 
> So, I would need to use a different nitpick_ignore list.
> 
> Is there a way for us to specify the nitpick_ignore list (or a different
> conf.py) via command line?

Since conf.py is python, we could implement something, which loads a
"conditional configuration", e.g. loading a config with a nitpick_ignore
list in. Depending on an environment 

 "SPHINX_BUILD_CONF=[strong|lazy]-nit-picking.py"

we could load individual build-configs overwriting the default settings
from the origin conf.py.

On which repo/branch you are working? .. I send you a RFC patch 
for "conditional configurations".

Back to (3) ... as far as I know, there is no way to add a 
*Internal Hyperlink Target* (e.g. "_v4l2-field:") to the C or 
CPP domain.

There is a ":any:" directive does something vice versa.

http://www.sphinx-doc.org/en/stable/markup/inline.html#role-any

But I think, this will not help referencing a type from a function
prototype to a *Internal Hyperlink Target*, like the struct described
under the "_v4l2-field:" target.

If ":any:" does not help, we might find a solution with an additional
crossref-type or something similar:

http://www.sphinx-doc.org/en/stable/extdev/appapi.html#sphinx.application.Sphinx.add_crossref_type

But this needs some more thoughts.

-- Markus --


> Btw, I'm enclosing a patch adding several of those references that are
> alien to the media subsystem and currently causes nitpick complains.
> 
> 
> doc-rst: ignore several undefined symbols in nitpick mode
> 
> using Sphinx in nitpick mode is too verbose to make it useful.
> So, we need to make it less verbose, in order to be able to
> actually use it.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> 
> diff --git a/Documentation/conf.py b/Documentation/conf.py
> index 96b7aa66c89c..d1805b8710c3 100644
> --- a/Documentation/conf.py
> +++ b/Documentation/conf.py
> @@ -419,3 +419,88 @@ pdf_documents = [
> # line arguments.
> kerneldoc_bin = '../scripts/kernel-doc'
> kerneldoc_srctree = '..'
> +
> +#
> +# It is possible to run Sphinx in nickpick mode with:
> +#	make SPHINXOPTS=-n htmldocs
> +# In such case, it will complain about lots of missing references that
> +#	1) are just typedefs like: bool, __u32, etc;
> +#	2) It will complain for things like: enum, NULL;
> +#	3) It will complain for symbols that should be on different
> +#	   books (but currently aren't ported to ReST)
> +# The list below has a list of such symbols to be ignored in nitpick mode
> +#
> +nitpick_ignore = [
> +	("c:func", "clock_gettime"),
> +	("c:func", "close"),
> +	("c:func", "container_of"),
> +	("c:func", "determine_valid_ioctls"),
> +	("c:func", "ERR_PTR"),
> +	("c:func", "ioctl"),
> +	("c:func", "IS_ERR"),
> +	("c:func", "mmap"),
> +	("c:func", "open"),
> +	("c:func", "pci_name"),
> +	("c:func", "poll"),
> +	("c:func", "PTR_ERR"),
> +	("c:func", "read"),
> +	("c:func", "release"),
> +	("c:func", "set"),
> +	("c:func", "struct fd_set"),
> +	("c:func", "struct pollfd"),
> +	("c:func", "usb_make_path"),
> +	("c:func", "write"),
> +	("c:type", "atomic_t"),
> +	("c:type", "bool"),
> +	("c:type", "buf_queue"),
> +	("c:type", "device"),
> +	("c:type", "device_driver"),
> +	("c:type", "device_node"),
> +	("c:type", "enum"),
> +	("c:type", "file"),
> +	("c:type", "i2c_adapter"),
> +	("c:type", "i2c_board_info"),
> +	("c:type", "i2c_client"),
> +	("c:type", "ktime_t"),
> +	("c:type", "led_classdev_flash"),
> +	("c:type", "list_head"),
> +	("c:type", "lock_class_key"),
> +	("c:type", "module"),
> +	("c:type", "mutex"),
> +	("c:type", "pci_dev"),
> +	("c:type", "pdvbdev"),
> +	("c:type", "poll_table_struct"),
> +	("c:type", "s32"),
> +	("c:type", "s64"),
> +	("c:type", "sd"),
> +	("c:type", "spi_board_info"),
> +	("c:type", "spi_device"),
> +	("c:type", "spi_master"),
> +	("c:type", "struct fb_fix_screeninfo"),
> +	("c:type", "struct pollfd"),
> +	("c:type", "struct timeval"),
> +	("c:type", "struct video_capability"),
> +	("c:type", "u16"),
> +	("c:type", "u32"),
> +	("c:type", "u64"),
> +	("c:type", "u8"),
> +	("c:type", "union"),
> +	("c:type", "usb_device"),
> +
> +	("cpp:type", "boolean"),
> +	("cpp:type", "fd"),
> +	("cpp:type", "fd_set"),
> +	("cpp:type", "int16_t"),
> +	("cpp:type", "NULL"),
> +	("cpp:type", "off_t"),
> +	("cpp:type", "pollfd"),
> +	("cpp:type", "size_t"),
> +	("cpp:type", "ssize_t"),
> +	("cpp:type", "timeval"),
> +	("cpp:type", "__u16"),
> +	("cpp:type", "__u32"),
> +	("cpp:type", "__u64"),
> +	("cpp:type", "uint16_t"),
> +	("cpp:type", "uint32_t"),
> +	("cpp:type", "video_system_t"),
> +]
> 
> 
> 

Re: Functions and data structure cross references with Sphinx

[Mauro Carvalho Chehab]

Em Fri, 5 Aug 2016 14:22:19 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 05.08.2016 um 12:47 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> 
> > Em Fri, 5 Aug 2016 09:29:23 +0200
> > Markus Heiser <markus.heiser@darmarit.de> escreveu:
> >   

> > Is there a way for us to specify the nitpick_ignore list (or a different
> > conf.py) via command line?  
> 
> Since conf.py is python, we could implement something, which loads a
> "conditional configuration", e.g. loading a config with a nitpick_ignore
> list in. Depending on an environment 
> 
>  "SPHINX_BUILD_CONF=[strong|lazy]-nit-picking.py"
> 
> we could load individual build-configs overwriting the default settings
> from the origin conf.py.

That would be interesting!

> On which repo/branch you are working? .. I send you a RFC patch 
> for "conditional configurations".

You can send it against either the media_tree.git[1] or the upstream 
one[2]. All Sphinx patches I have were merged there already (except for 
the experimental nickpick patch I attached on the previous e-mail).

[1] https://git.linuxtv.org/media_tree.git/log/
[2] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/

> > 3) The references generated from the header files I'm parsing don't
> > use the c (or cpp) domain. They're declared at the media book as a
> > normal reference:
> > 	Documentation/media/uapi/v4l/field-order.rst:.. _v4l2-field:
> > 

> Back to (3) ... as far as I know, there is no way to add a 
> *Internal Hyperlink Target* (e.g. "_v4l2-field:") to the C or 
> CPP domain.

Argh!

> 
> There is a ":any:" directive does something vice versa.
> 
> http://www.sphinx-doc.org/en/stable/markup/inline.html#role-any
> 
> But I think, this will not help referencing a type from a function
> prototype to a *Internal Hyperlink Target*, like the struct described
> under the "_v4l2-field:" target.

Hmm... it might help, but this is Sphinx 1.3 or upper only.
As we decide to set the minimal version bar to 1.2, we should
avoid using it.

> 
> If ":any:" does not help, we might find a solution with an additional
> crossref-type or something similar:
> 
> http://www.sphinx-doc.org/en/stable/extdev/appapi.html#sphinx.application.Sphinx.add_crossref_type
> 
> But this needs some more thoughts.

That sounds more promising, but we'll need a replacement for the
:c:func: tag to use it.


Thanks,
Mauro