[Qemu-devel] Exploring Sphinx, autodoc, apidoc, and coverage tools for python/qemu

[Qemu-devel] Exploring Sphinx, autodoc, apidoc, and coverage tools for python/qemu

[John Snow]

Has anyone on this list experimented with these tools?

I was hoping to use them to document things like the python/machine.py
and python/qmp.py modules to help demonstrate some of our internal
tooling API (for test writers, GSoC/Outreachy interns, folks who want to
script QEMU at a level between writing a CLI driver and using libvirt.)

What follows below is my process trying to enable this and some of the
problems I'm still stuck with, summarized below at the end of this more
exploratory text.


Enabling autodoc:

First, it appears as if enabling the "sphinx-autodoc" tool is not
sufficient for actually generating anything at all when you invoke the
sphinx-generated "make html" target. It just enables understanding
certain directives.

So apparently you need to generate module "stubs" using sphinx-autodoc.
Sphinx uses the sphinx-autodoc extension to understand how to consume
the directives in these stubs.

That strikes me as odd, because these stubs might need to be changed
frequently as code comes and goes; it seems strange that it isn't
integrated at the top level. (Do I have the wrong idea on how these
tools should be used?)

So you need to run:
> sphinx-apidoc --separate --module-first -o docs/ python/qemu/

which generates stubs to docs:

Creating file docs/qemu.machine.rst.
Creating file docs/qemu.qmp.rst.
Creating file docs/qemu.qtest.rst.
Creating file docs/qemu.rst.
Creating file docs/modules.rst.

And then you can edit e.g. the top-level index.rst TOC in docs/index.rst
to look like this:

```
.. toctree::
   :maxdepth: 2
   :caption: Contents:

   interop/index
   devel/index
   specs/index
   modules
```

And then finally generating the build; manually removing the -W option
from the Makefile: there are a lot of warnings in here.

> sphinx-build -n -b html -D version=4.0.92 -D release="4.0.92
(v4.1.0-rc2-34-g160802eb07-dirty)" -d .doctrees/
/home/bos/jhuston/src/qemu/docs/ docs/

Great! that will generate output to docs/index.html which indeed shows
APIdoc comments generated from our Python files. Good.

However, where this gets a little strange is if you look at the
generated stubs. For instance, qemu.machine.rst looks like this:

```
.. automodule:: qemu.machine
    :members:
    :undoc-members:
    :show-inheritance:
```

:undoc-members: says that we want to "document" any members that don't
have a matching apidoc comment by generating a stub.

Oops, but the presence of that stub will cause the sphinx coverage tool
to happily report 100% coverage.

Further oops, pylint doesn't understand apidoc comments and can't be
used as the linter in this case, either.

You can edit the stubs to remove these directives, but these stubs are
generated -- and it doesn't appear like there's a command line option to
change this behavior. ...Hmm.

And either way, the coverage tool only generates a report and not
something with an error code that I could use to gate the build. Same
goes for the general build: if I remove the :undoc-members: parameter,
there's nothing in the autodoc module that appears to throw warnings
when it encounters undocumented parameters or members.

That seems disappointing, because it's hard to keep docstrings up to
date unless they are checked conclusively at build time.


Conclusions:

- the autodoc documentation page doesn't seem to document examples of
how you're expected to write meaningful docstrings for the tool to extract.

- autodoc fools the coverage tool into reporting 100% coverage.

- autodoc can be configured to omit non-documented members to allow the
coverage tool to work, but the configuration is auto-generated and
defaults to always generating documentation for these entities.

- coverage tool doesn't appear like it can be used for gating the build
natively for missing python docs; it only generates a report.

- Even if we script to block on a non-empty report, the coverage tool
only works at the function/class level and does not understand the
concept of missing parameter or return value tags.

- It would seem that it would be the Autodoc module's job to be
responsible for understanding incomplete documentation, but doesn't
appear to. The :param name: syntax is just a ReST "field list" and isn't
parsed semantically by autodoc, sadly.


It looks to me, at a glance, that there's nothing in Sphinx that knows
how to look for and warn about undocumented parameters, exception types,
return values, etc. Hopefully I've missed something and it is possible.

--js

Re: [Qemu-devel] Exploring Sphinx, autodoc, apidoc, and coverage tools for python/qemu

[Peter Maydell]

On Wed, 24 Jul 2019 at 22:06, John Snow <jsnow@redhat.com> wrote:
> And then you can edit e.g. the top-level index.rst TOC in docs/index.rst
> to look like this:
>
> ```
> .. toctree::
>    :maxdepth: 2
>    :caption: Contents:
>
>    interop/index
>    devel/index
>    specs/index
>    modules
> ```

This is obviously just prototyping, but you don't want to put
anything new in this top level index.rst -- it's only used
for by-hand full-tree docs builds. Builds from the makefile
rules will just build our separate manuals (interop, devel,
specs) separately. You want to put your new documentation into
whichever manual is best suited (probably devel/, but possibly
interop/ in some cases?)

(Will read the rest of this email later.)

thanks
-- PMM

Re: [Qemu-devel] Exploring Sphinx, autodoc, apidoc, and coverage tools for python/qemu

[John Snow]

On 7/25/19 5:02 AM, Peter Maydell wrote:
> On Wed, 24 Jul 2019 at 22:06, John Snow <jsnow@redhat.com> wrote:
>> And then you can edit e.g. the top-level index.rst TOC in docs/index.rst
>> to look like this:
>>
>> ```
>> .. toctree::
>>    :maxdepth: 2
>>    :caption: Contents:
>>
>>    interop/index
>>    devel/index
>>    specs/index
>>    modules
>> ```
> 
> This is obviously just prototyping, but you don't want to put
> anything new in this top level index.rst -- it's only used
> for by-hand full-tree docs builds. Builds from the makefile
> rules will just build our separate manuals (interop, devel,
> specs) separately. You want to put your new documentation into
> whichever manual is best suited (probably devel/, but possibly
> interop/ in some cases?)
> 

Yes, understood -- I was prototyping in a "fresh" / empty repository, so
I was keeping the instructions reasonably comparable to what
sphinx-quickstart might produce if people wanted to explore this outside
of the complexity of the QEMU tree.

If the experiment had gone better, I'd have wanted to either model this
as a Developer sub-manual, or a new top-level manual under "Python
Library" or some such.

(It's hard to say which the QMP library is: it WAS internal developer
tooling, but I'm prototyping turning qmp-shell into something that could
be considered a reference implementation for interop that non-developers
might have a genuine interest in using for non-libvirt scenarios.)

> (Will read the rest of this email later.)
> 

TLDR: I wanted to document for the mailing list that Autodoc seems to
have some shortcomings that doesn't make it very attractive for
documenting our python utilities in a rigorous way.

One of the benefits, in my mind, of using doc generation utilities for
documenting API is the ability to fail the build when the documentation
has observable omissions/mistakes.

Autodoc doesn't seem capable of providing that; though it still may be
more useful than nothing.

--js

> thanks
> -- PMM
> 

Re: [Qemu-devel] Exploring Sphinx, autodoc, apidoc, and coverage tools for python/qemu

[Eduardo Habkost]

CCing Cleber and Gabriel.  Comments at the "conclusions" section
below:

On Wed, Jul 24, 2019 at 05:06:41PM -0400, John Snow wrote:
> Has anyone on this list experimented with these tools?
> 
> I was hoping to use them to document things like the python/machine.py
> and python/qmp.py modules to help demonstrate some of our internal
> tooling API (for test writers, GSoC/Outreachy interns, folks who want to
> script QEMU at a level between writing a CLI driver and using libvirt.)
> 
> What follows below is my process trying to enable this and some of the
> problems I'm still stuck with, summarized below at the end of this more
> exploratory text.
> 
> 
> Enabling autodoc:
> 
> First, it appears as if enabling the "sphinx-autodoc" tool is not
> sufficient for actually generating anything at all when you invoke the
> sphinx-generated "make html" target. It just enables understanding
> certain directives.
> 
> So apparently you need to generate module "stubs" using sphinx-autodoc.
> Sphinx uses the sphinx-autodoc extension to understand how to consume
> the directives in these stubs.
> 
> That strikes me as odd, because these stubs might need to be changed
> frequently as code comes and goes; it seems strange that it isn't
> integrated at the top level. (Do I have the wrong idea on how these
> tools should be used?)
> 
> So you need to run:
> > sphinx-apidoc --separate --module-first -o docs/ python/qemu/
> 
> which generates stubs to docs:
> 
> Creating file docs/qemu.machine.rst.
> Creating file docs/qemu.qmp.rst.
> Creating file docs/qemu.qtest.rst.
> Creating file docs/qemu.rst.
> Creating file docs/modules.rst.
> 
> And then you can edit e.g. the top-level index.rst TOC in docs/index.rst
> to look like this:
> 
> ```
> .. toctree::
>    :maxdepth: 2
>    :caption: Contents:
> 
>    interop/index
>    devel/index
>    specs/index
>    modules
> ```
> 
> And then finally generating the build; manually removing the -W option
> from the Makefile: there are a lot of warnings in here.
> 
> > sphinx-build -n -b html -D version=4.0.92 -D release="4.0.92
> (v4.1.0-rc2-34-g160802eb07-dirty)" -d .doctrees/
> /home/bos/jhuston/src/qemu/docs/ docs/
> 
> Great! that will generate output to docs/index.html which indeed shows
> APIdoc comments generated from our Python files. Good.
> 
> However, where this gets a little strange is if you look at the
> generated stubs. For instance, qemu.machine.rst looks like this:
> 
> ```
> .. automodule:: qemu.machine
>     :members:
>     :undoc-members:
>     :show-inheritance:
> ```
> 
> :undoc-members: says that we want to "document" any members that don't
> have a matching apidoc comment by generating a stub.
> 
> Oops, but the presence of that stub will cause the sphinx coverage tool
> to happily report 100% coverage.
> 
> Further oops, pylint doesn't understand apidoc comments and can't be
> used as the linter in this case, either.
> 
> You can edit the stubs to remove these directives, but these stubs are
> generated -- and it doesn't appear like there's a command line option to
> change this behavior. ...Hmm.
> 
> And either way, the coverage tool only generates a report and not
> something with an error code that I could use to gate the build. Same
> goes for the general build: if I remove the :undoc-members: parameter,
> there's nothing in the autodoc module that appears to throw warnings
> when it encounters undocumented parameters or members.
> 
> That seems disappointing, because it's hard to keep docstrings up to
> date unless they are checked conclusively at build time.
> 
> 
> Conclusions:
> 
> - the autodoc documentation page doesn't seem to document examples of
> how you're expected to write meaningful docstrings for the tool to extract.

I had the same impression when I read sphinx/autodoc
documentation.

> 
> - autodoc fools the coverage tool into reporting 100% coverage.
> 
> - autodoc can be configured to omit non-documented members to allow the
> coverage tool to work, but the configuration is auto-generated and
> defaults to always generating documentation for these entities.
> 
> - coverage tool doesn't appear like it can be used for gating the build
> natively for missing python docs; it only generates a report.
> 
> - Even if we script to block on a non-empty report, the coverage tool
> only works at the function/class level and does not understand the
> concept of missing parameter or return value tags.
> 
> - It would seem that it would be the Autodoc module's job to be
> responsible for understanding incomplete documentation, but doesn't
> appear to. The :param name: syntax is just a ReST "field list" and isn't
> parsed semantically by autodoc, sadly.

I wonder if there are other Python documentation coverage tools
outside Sphinx.  Googling for [python docstring coverage]
resulted in a few different projects, but I don't know which ones
would understand Sphinx-compatible rST docstrings.

> 
> 
> It looks to me, at a glance, that there's nothing in Sphinx that knows
> how to look for and warn about undocumented parameters, exception types,
> return values, etc. Hopefully I've missed something and it is possible.

Personally, my first priority would be generating reasonable
documentation from docstrings, preferably integrated with the
rest of the QEMU docs.  Gating based on docstring coverage checks
would be just nice to have.

-- 
Eduardo

Re: [Qemu-devel] Exploring Sphinx, autodoc, apidoc, and coverage tools for python/qemu

[John Snow]

On 7/26/19 5:16 PM, Eduardo Habkost wrote:
> CCing Cleber and Gabriel.  Comments at the "conclusions" section
> below:
> 
> On Wed, Jul 24, 2019 at 05:06:41PM -0400, John Snow wrote:
>> Has anyone on this list experimented with these tools?
>>
>> I was hoping to use them to document things like the python/machine.py
>> and python/qmp.py modules to help demonstrate some of our internal
>> tooling API (for test writers, GSoC/Outreachy interns, folks who want to
>> script QEMU at a level between writing a CLI driver and using libvirt.)
>>
>> What follows below is my process trying to enable this and some of the
>> problems I'm still stuck with, summarized below at the end of this more
>> exploratory text.
>>
>>
>> Enabling autodoc:
>>
>> First, it appears as if enabling the "sphinx-autodoc" tool is not
>> sufficient for actually generating anything at all when you invoke the
>> sphinx-generated "make html" target. It just enables understanding
>> certain directives.
>>
>> So apparently you need to generate module "stubs" using sphinx-autodoc.
>> Sphinx uses the sphinx-autodoc extension to understand how to consume
>> the directives in these stubs.
>>
>> That strikes me as odd, because these stubs might need to be changed
>> frequently as code comes and goes; it seems strange that it isn't
>> integrated at the top level. (Do I have the wrong idea on how these
>> tools should be used?)
>>
>> So you need to run:
>>> sphinx-apidoc --separate --module-first -o docs/ python/qemu/
>>
>> which generates stubs to docs:
>>
>> Creating file docs/qemu.machine.rst.
>> Creating file docs/qemu.qmp.rst.
>> Creating file docs/qemu.qtest.rst.
>> Creating file docs/qemu.rst.
>> Creating file docs/modules.rst.
>>
>> And then you can edit e.g. the top-level index.rst TOC in docs/index.rst
>> to look like this:
>>
>> ```
>> .. toctree::
>>    :maxdepth: 2
>>    :caption: Contents:
>>
>>    interop/index
>>    devel/index
>>    specs/index
>>    modules
>> ```
>>
>> And then finally generating the build; manually removing the -W option
>> from the Makefile: there are a lot of warnings in here.
>>
>>> sphinx-build -n -b html -D version=4.0.92 -D release="4.0.92
>> (v4.1.0-rc2-34-g160802eb07-dirty)" -d .doctrees/
>> /home/bos/jhuston/src/qemu/docs/ docs/
>>
>> Great! that will generate output to docs/index.html which indeed shows
>> APIdoc comments generated from our Python files. Good.
>>
>> However, where this gets a little strange is if you look at the
>> generated stubs. For instance, qemu.machine.rst looks like this:
>>
>> ```
>> .. automodule:: qemu.machine
>>     :members:
>>     :undoc-members:
>>     :show-inheritance:
>> ```
>>
>> :undoc-members: says that we want to "document" any members that don't
>> have a matching apidoc comment by generating a stub.
>>
>> Oops, but the presence of that stub will cause the sphinx coverage tool
>> to happily report 100% coverage.
>>
>> Further oops, pylint doesn't understand apidoc comments and can't be
>> used as the linter in this case, either.
>>
>> You can edit the stubs to remove these directives, but these stubs are
>> generated -- and it doesn't appear like there's a command line option to
>> change this behavior. ...Hmm.
>>
>> And either way, the coverage tool only generates a report and not
>> something with an error code that I could use to gate the build. Same
>> goes for the general build: if I remove the :undoc-members: parameter,
>> there's nothing in the autodoc module that appears to throw warnings
>> when it encounters undocumented parameters or members.
>>
>> That seems disappointing, because it's hard to keep docstrings up to
>> date unless they are checked conclusively at build time.
>>
>>
>> Conclusions:
>>
>> - the autodoc documentation page doesn't seem to document examples of
>> how you're expected to write meaningful docstrings for the tool to extract.
> 
> I had the same impression when I read sphinx/autodoc
> documentation.
> 
>>
>> - autodoc fools the coverage tool into reporting 100% coverage.
>>
>> - autodoc can be configured to omit non-documented members to allow the
>> coverage tool to work, but the configuration is auto-generated and
>> defaults to always generating documentation for these entities.
>>
>> - coverage tool doesn't appear like it can be used for gating the build
>> natively for missing python docs; it only generates a report.
>>
>> - Even if we script to block on a non-empty report, the coverage tool
>> only works at the function/class level and does not understand the
>> concept of missing parameter or return value tags.
>>
>> - It would seem that it would be the Autodoc module's job to be
>> responsible for understanding incomplete documentation, but doesn't
>> appear to. The :param name: syntax is just a ReST "field list" and isn't
>> parsed semantically by autodoc, sadly.
> 
> I wonder if there are other Python documentation coverage tools
> outside Sphinx.  Googling for [python docstring coverage]
> resulted in a few different projects, but I don't know which ones
> would understand Sphinx-compatible rST docstrings.
> 
>>
>>
>> It looks to me, at a glance, that there's nothing in Sphinx that knows
>> how to look for and warn about undocumented parameters, exception types,
>> return values, etc. Hopefully I've missed something and it is possible.
> 
> Personally, my first priority would be generating reasonable
> documentation from docstrings, preferably integrated with the
> rest of the QEMU docs.  Gating based on docstring coverage checks
> would be just nice to have.
> 

Right, and for THIS, we can simply do a pylint check. we can probably
target some pylint subset when we drop python2 support entirely and
start keeping some minimum code quality metric.

--js

Re: Exploring Sphinx, autodoc, apidoc, and coverage tools for python/qemu

[John Snow]

On 7/24/19 5:06 PM, John Snow wrote:
> Has anyone on this list experimented with these tools?
> 
> I was hoping to use them to document things like the python/machine.py
> and python/qmp.py modules to help demonstrate some of our internal
> tooling API (for test writers, GSoC/Outreachy interns, folks who want to
> script QEMU at a level between writing a CLI driver and using libvirt.)
> 
> What follows below is my process trying to enable this and some of the
> problems I'm still stuck with, summarized below at the end of this more
> exploratory text.
> 
> 
> Enabling autodoc:
> 
> First, it appears as if enabling the "sphinx-autodoc" tool is not
> sufficient for actually generating anything at all when you invoke the
> sphinx-generated "make html" target. It just enables understanding
> certain directives.
> 
> So apparently you need to generate module "stubs" using sphinx-autodoc.
> Sphinx uses the sphinx-autodoc extension to understand how to consume
> the directives in these stubs.
> 
> That strikes me as odd, because these stubs might need to be changed
> frequently as code comes and goes; it seems strange that it isn't
> integrated at the top level. (Do I have the wrong idea on how these
> tools should be used?)
> 
> So you need to run:
>> sphinx-apidoc --separate --module-first -o docs/ python/qemu/
> 
> which generates stubs to docs:
> 
> Creating file docs/qemu.machine.rst.
> Creating file docs/qemu.qmp.rst.
> Creating file docs/qemu.qtest.rst.
> Creating file docs/qemu.rst.
> Creating file docs/modules.rst.
> 
> And then you can edit e.g. the top-level index.rst TOC in docs/index.rst
> to look like this:
> 
> ```
> .. toctree::
>    :maxdepth: 2
>    :caption: Contents:
> 
>    interop/index
>    devel/index
>    specs/index
>    modules
> ```
> 
> And then finally generating the build; manually removing the -W option
> from the Makefile: there are a lot of warnings in here.
> 
>> sphinx-build -n -b html -D version=4.0.92 -D release="4.0.92
> (v4.1.0-rc2-34-g160802eb07-dirty)" -d .doctrees/
> /home/bos/jhuston/src/qemu/docs/ docs/
> 
> Great! that will generate output to docs/index.html which indeed shows
> APIdoc comments generated from our Python files. Good.
> 
> However, where this gets a little strange is if you look at the
> generated stubs. For instance, qemu.machine.rst looks like this:
> 
> ```
> .. automodule:: qemu.machine
>     :members:
>     :undoc-members:
>     :show-inheritance:
> ```
> 
> :undoc-members: says that we want to "document" any members that don't
> have a matching apidoc comment by generating a stub.
> 
> Oops, but the presence of that stub will cause the sphinx coverage tool
> to happily report 100% coverage.
> 
> Further oops, pylint doesn't understand apidoc comments and can't be
> used as the linter in this case, either.
> 
> You can edit the stubs to remove these directives, but these stubs are
> generated -- and it doesn't appear like there's a command line option to
> change this behavior. ...Hmm.
> 

Update: there is indeed a way to change this behavior, through
environment variables.

https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html

SPHINX_APIDOC_OPTIONS
A comma-separated list of option to append to generated automodule
directives. Defaults to members,undoc-members,show-inheritance.

You just have to set it and remove 'undoc-members'.

> And either way, the coverage tool only generates a report and not
> something with an error code that I could use to gate the build. Same
> goes for the general build: if I remove the :undoc-members: parameter,
> there's nothing in the autodoc module that appears to throw warnings
> when it encounters undocumented parameters or members.
> 

This is still a problem, though. Nothing gates on undocumented members
at all.

> That seems disappointing, because it's hard to keep docstrings up to
> date unless they are checked conclusively at build time.
> 
> 
> Conclusions:
> 
> - the autodoc documentation page doesn't seem to document examples of
> how you're expected to write meaningful docstrings for the tool to extract.
> 
> - autodoc fools the coverage tool into reporting 100% coverage.
> 
> - autodoc can be configured to omit non-documented members to allow the
> coverage tool to work, but the configuration is auto-generated and
> defaults to always generating documentation for these entities.
> 

apidoc (the stub generator for autodoc) can indeed be configured to omit
non-documented members through an environment variable now, so this
point is not true.

> - coverage tool doesn't appear like it can be used for gating the build
> natively for missing python docs; it only generates a report.
> 

Still seems true, though there are other tools that can gate on such
things. Specifically we can use pylint and require that docstrings are
present.

Notably, pylint only checks that there is a docstring at all, and
doesn't try to perform any semantic validation of what's in it.

To my knowledge, there are no standalone tools that do such work -- but
the pycharm built-in linter appears to have semantic support for certain
docstring formats.

> - Even if we script to block on a non-empty report, the coverage tool
> only works at the function/class level and does not understand the
> concept of missing parameter or return value tags.
> 

Still a problem based on the above; there is just no agreed upon
semantic format in the python world for documenting parameters.

> - It would seem that it would be the Autodoc module's job to be
> responsible for understanding incomplete documentation, but doesn't
> appear to. The :param name: syntax is just a ReST "field list" and isn't
> parsed semantically by autodoc, sadly.
> 

Might be something I investigate based on some discussions I've had in
the Python community. Could be useful to have a canonical docstring
format that sphinx is able to give warnings against.

> 
> It looks to me, at a glance, that there's nothing in Sphinx that knows
> how to look for and warn about undocumented parameters, exception types,
> return values, etc. Hopefully I've missed something and it is possible.
> 
> --js
>