[Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Peter Maydell]

Hi; I had a play on Friday afternoon with Sphinx, which is the
restructured-text documentation processor that the Linux kernel is
just switching to for their documentation. My first impression is
that it would probably be useful for us too (and if the kernel
is using it this also suggests that it's a fairly reasonable
tool to rely on that isn't going to stagnate or vanish on us).

In particular I think we could:
 * set up a framework for our in-tree docs/ which gives us a
   place to put new docs (both for-users and for-developers) --
   I think having someplace to put things will reduce the barrier
   to people writing useful new docs
 * gradually convert the existing docs to rst
 * use the sphinx extension features to pull in the doc-comments
   we have been fairly consistently writing over the last few years
   (for instance a converted version of docs/memory.txt could pull
   in doc comments from memory.h; or we can just write simple
   wrapper files like a "Bitmap operations" document that
   displays the doc comments from bitops.h)
 * eventually replace our current use of texinfo, since Sphinx
   can output PDF and manpages as well as HTML

Because Sphinx only parses doc-comments in files when a .rst
document in the tree specifically asks for them, this means we
can do a gradual transition. (Tools like gtkdoc want to parse
every file in the tree at once which gives a nasty big-bang
requirement to fix a lot of doc comment syntax errors all at once.)

This does raise the question though of what format our doc
comments actually are. Because we haven't been running any
kind of document tool on the code there's been no syntax checking,
so I suspect what we have in practice is a bit of a mishmash.
The original intent I think was that they were gtkdoc format,
but we also have some kernel-doc format comments (where we've
borrowed headers from the kernel, and then subsequently
copied that style in other doc comments), and probably some
which aren't any formal syntax at all.
We have as for as I can see just 3 gtkdoc "SECTION:"
markers for doc-comments which aren't just documenting a
particular function, type or structure.

Since the doc-comment parsing is handled by a Sphinx extension
(which is in Python), we could in theory pick any syntax we
liked. However if we said "use kernel-doc" we would be able
to just borrow the kernel's kernel-doc extension without
having to do a load of hacking on it (instead we'd need to
fix up doc comments when we added documentation files that
read in those source files, but we will need to do some of
that anyway to fix accidental syntax errors).

Does anybody have strong opinions on doc-comment formats?
Anybody with experience with the kernel-doc format and
views on any shortcomings?

Sphinx is packaged in Debian stable so hopefully for most
developers it would not be "bleeding edge have to have a
custom install of this" tooling.

For an idea of how the docs tend to come out, see
https://readthedocs.org/ which collects formatted docs
for a large number of Sphinx-using projects.
There are a couple of LWN articles about Sphinx for the kernel
if you want more detail: https://lwn.net/Articles/692704/

With a little luck I may be able to put something up
on Monday as a sort of minimal-demonstration of how
this would look in QEMU.

thanks
-- PMM

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Daniel P. Berrange]

On Sat, Nov 05, 2016 at 06:42:23PM +0000, Peter Maydell wrote:
> Hi; I had a play on Friday afternoon with Sphinx, which is the
> restructured-text documentation processor that the Linux kernel is
> just switching to for their documentation. My first impression is
> that it would probably be useful for us too (and if the kernel
> is using it this also suggests that it's a fairly reasonable
> tool to rely on that isn't going to stagnate or vanish on us).
> 
> In particular I think we could:
>  * set up a framework for our in-tree docs/ which gives us a
>    place to put new docs (both for-users and for-developers) --
>    I think having someplace to put things will reduce the barrier
>    to people writing useful new docs
>  * gradually convert the existing docs to rst
>  * use the sphinx extension features to pull in the doc-comments
>    we have been fairly consistently writing over the last few years
>    (for instance a converted version of docs/memory.txt could pull
>    in doc comments from memory.h; or we can just write simple
>    wrapper files like a "Bitmap operations" document that
>    displays the doc comments from bitops.h)
>  * eventually replace our current use of texinfo, since Sphinx
>    can output PDF and manpages as well as HTML
> 
> Because Sphinx only parses doc-comments in files when a .rst
> document in the tree specifically asks for them, this means we
> can do a gradual transition. (Tools like gtkdoc want to parse
> every file in the tree at once which gives a nasty big-bang
> requirement to fix a lot of doc comment syntax errors all at once.)

FWIW, gtk-doc doesn't have to be a big-big - you simply pass
it the list of file names you want it to process - it doesn't
recursively find all itself.

That said, I've done some experiments with gtk-doc and QEMU
a couple of weeks back and it didn't work out well. In particular
it has an peculiar hardcoded assumption about the way typedefs &
enums are declared, which is different from common practice in
QEMU. As a result, gtk-doc simply looses / discards a lot of
type info in QEMU source causing it to get very confused. Having
looke at the source, there's no way to workaround this bug, and
I don't think we want to change QEMU source to match gkt-docs'
preferred style. So IMHO that rules out GTK-DOC for QEMU.

I've also did a few experiments using Doxygen, and while it
managed to cope with consuming QEMU's source code fine, the
way it lays out the resulting HTML pages is just awful IMHO.

> This does raise the question though of what format our doc
> comments actually are. Because we haven't been running any
> kind of document tool on the code there's been no syntax checking,
> so I suspect what we have in practice is a bit of a mishmash.
> The original intent I think was that they were gtkdoc format,
> but we also have some kernel-doc format comments (where we've
> borrowed headers from the kernel, and then subsequently
> copied that style in other doc comments), and probably some
> which aren't any formal syntax at all.
> We have as for as I can see just 3 gtkdoc "SECTION:"
> markers for doc-comments which aren't just documenting a
> particular function, type or structure.

Yep, no matter which tool we use, we're going to need to clean
up existing API docs comments to be consistent.

> Since the doc-comment parsing is handled by a Sphinx extension
> (which is in Python), we could in theory pick any syntax we
> liked. However if we said "use kernel-doc" we would be able
> to just borrow the kernel's kernel-doc extension without
> having to do a load of hacking on it (instead we'd need to
> fix up doc comments when we added documentation files that
> read in those source files, but we will need to do some of
> that anyway to fix accidental syntax errors).

Yes, it'd be nice to not have to write our own extension
if possible.

> Does anybody have strong opinions on doc-comment formats?
> Anybody with experience with the kernel-doc format and
> views on any shortcomings?
> 
> Sphinx is packaged in Debian stable so hopefully for most
> developers it would not be "bleeding edge have to have a
> custom install of this" tooling.
> 
> For an idea of how the docs tend to come out, see
> https://readthedocs.org/ which collects formatted docs
> for a large number of Sphinx-using projects.
> There are a couple of LWN articles about Sphinx for the kernel
> if you want more detail: https://lwn.net/Articles/692704/
> 
> With a little luck I may be able to put something up
> on Monday as a sort of minimal-demonstration of how
> this would look in QEMU.

Sounds like a worthwhile idea to me overall.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1444 bytes --]

On Sat, Nov 05, 2016 at 06:42:23PM +0000, Peter Maydell wrote:
> In particular I think we could:
>  * set up a framework for our in-tree docs/ which gives us a
>    place to put new docs (both for-users and for-developers) --
>    I think having someplace to put things will reduce the barrier
>    to people writing useful new docs
>  * gradually convert the existing docs to rst
>  * use the sphinx extension features to pull in the doc-comments
>    we have been fairly consistently writing over the last few years
>    (for instance a converted version of docs/memory.txt could pull
>    in doc comments from memory.h; or we can just write simple
>    wrapper files like a "Bitmap operations" document that
>    displays the doc comments from bitops.h)

You are suggesting Sphinx for two different purposes:

1. Formatting docs/ in HTML, PDF, etc.

2. API documentation from doc comments.

It's a good idea for #1 since we can then publish automated builds of
the docs.  They will be easy to view and link to in a web browser.

I'm not a fan of #2.  QEMU is not a C library that people develop
against and our APIs are not stable.  There is no incentive for pretty
doc comments.  It might be cool to set it up once but things will
deterioate again quickly because we don't actually need external API
docs.

Instead of #2 we should focus on generating nice external QMP docs for
libvirt and other clients.  That has a clear benefit.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Daniel P. Berrange]

On Mon, Nov 07, 2016 at 01:30:45PM +0000, Stefan Hajnoczi wrote:
> On Sat, Nov 05, 2016 at 06:42:23PM +0000, Peter Maydell wrote:
> > In particular I think we could:
> >  * set up a framework for our in-tree docs/ which gives us a
> >    place to put new docs (both for-users and for-developers) --
> >    I think having someplace to put things will reduce the barrier
> >    to people writing useful new docs
> >  * gradually convert the existing docs to rst
> >  * use the sphinx extension features to pull in the doc-comments
> >    we have been fairly consistently writing over the last few years
> >    (for instance a converted version of docs/memory.txt could pull
> >    in doc comments from memory.h; or we can just write simple
> >    wrapper files like a "Bitmap operations" document that
> >    displays the doc comments from bitops.h)
> 
> You are suggesting Sphinx for two different purposes:
> 
> 1. Formatting docs/ in HTML, PDF, etc.
> 
> 2. API documentation from doc comments.
> 
> It's a good idea for #1 since we can then publish automated builds of
> the docs.  They will be easy to view and link to in a web browser.
> 
> I'm not a fan of #2.  QEMU is not a C library that people develop
> against and our APIs are not stable.  There is no incentive for pretty
> doc comments.  It might be cool to set it up once but things will
> deterioate again quickly because we don't actually need external API
> docs.

For shared internal infrastructure code I very much disagree. If I'm
writing something (like a new block driver), I'm relying on a bunch
of existing code that I'm calling. Some of the methods I'm consuming
may be in a library like GLib, other methods may be QEMU internal
infrastructure (like util/*, io/*, cryto/*). Regardless of whether
those methods are internal or from a library, the API docs are very
important / valuable in ensuring I'm understanding the required usage
contract of the method. The lack of API docs for the QEMU block layer
was a major cause of pain for me when writing the LUKS driver.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Peter Maydell]

On 7 November 2016 at 13:30, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> You are suggesting Sphinx for two different purposes:
>
> 1. Formatting docs/ in HTML, PDF, etc.
>
> 2. API documentation from doc comments.
>
> It's a good idea for #1 since we can then publish automated builds of
> the docs.  They will be easy to view and link to in a web browser.
>
> I'm not a fan of #2.  QEMU is not a C library that people develop
> against and our APIs are not stable.  There is no incentive for pretty
> doc comments.  It might be cool to set it up once but things will
> deterioate again quickly because we don't actually need external API
> docs.

I think my main argument here is that although it's fine
for experienced developers to go wading through the header
files for information, it's a bit of a wall for people who
are new to QEMU and have no idea what kinds of APIs we have
or which headers they might be in. We do see people appearing
on the mailing lists who'd like to add a device model or whatever
to QEMU and have no idea where to start or what the APIs they
need to use for it do. I'd like it to be easier for people to
get started with QEMU development.

A lot of docs/ is also internal technical documentation, which
you could argue equally tends to become out of date. I think
having the two integrated helps: the general overview info
can then link to or include the specifics on the functions.

Actually doing something with our docs comments would mean that
we notice things like "you changed this function but forgot
to update the docs with the new parameter" because they'd
generate a build error. I think it's lack of any automated
process of checking the doc comments that means they've
deteriorated.

thanks
-- PMM

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Peter Maydell]

On 5 November 2016 at 18:42, Peter Maydell <peter.maydell@linaro.org> wrote:
> With a little luck I may be able to put something up
> on Monday as a sort of minimal-demonstration of how
> this would look in QEMU.

Generated documentation:
  http://people.linaro.org/~peter.maydell/sphinx/index.html
Git branch with the patches needed to produce that:
  https://git.linaro.org/people/peter.maydell/qemu-arm.git sphinx-docs
Pointy-clicky interface to git branch:
  https://git.linaro.org/people/peter.maydell/qemu-arm.git/log/?h=sphinx-docs

I didn't bother to write the makefile changes to tie it into
the main build process, so to regenerate the docs locally you'll
need to run
 sphinx-build -b html docs my-build-dir/docs
from the QEMU source tree root, which will put the output into
my-build-dir/docs, which you can then point your web browser at.

The overall organisation structure needs some thought --
I think we should at least separate into user/ for user
docs and dev/ for internals docs (and only install the
user/ docs). The branch above just puts the two example
docs directly into the index.rst for demo purposes.

Conclusions from this exercise:
1) conversion isn't all that difficult, and the results
   look pretty nice
2) some of the doc-comment format differences are irritating:
   . "function - short description" not "function: short description"
   . "&struct.fieldname" not ".@fieldname"
   . "&typename" not "#typename"
3) the most awkward part of kernel-doc syntax is that it bakes
   in the kernel's style choice of always using "struct foo"
   for types -- I don't think there's any way to document
   'MemoryRegion' and 'AddressSpace' without the 'struct'
   coming out in the documentation output.

We could fix (2) by loosening the kernel-doc script's
parsing if we were happy to carry around a forked version
of it. Fixing (3) requires more serious surgery on kernel-doc
I suspect.

I think this is probably sufficient for us to figure out
whether this is a path we want to proceed down, anyway.

thanks
-- PMM

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Paolo Bonzini]

On 07/11/2016 16:03, Peter Maydell wrote:
> On 5 November 2016 at 18:42, Peter Maydell <peter.maydell@linaro.org> wrote:
>> With a little luck I may be able to put something up
>> on Monday as a sort of minimal-demonstration of how
>> this would look in QEMU.
> 
> Generated documentation:
>   http://people.linaro.org/~peter.maydell/sphinx/index.html
> Git branch with the patches needed to produce that:
>   https://git.linaro.org/people/peter.maydell/qemu-arm.git sphinx-docs
> Pointy-clicky interface to git branch:
>   https://git.linaro.org/people/peter.maydell/qemu-arm.git/log/?h=sphinx-docs
> 
> I didn't bother to write the makefile changes to tie it into
> the main build process, so to regenerate the docs locally you'll
> need to run
>  sphinx-build -b html docs my-build-dir/docs
> from the QEMU source tree root, which will put the output into
> my-build-dir/docs, which you can then point your web browser at.
> 
> The overall organisation structure needs some thought --
> I think we should at least separate into user/ for user
> docs and dev/ for internals docs (and only install the
> user/ docs). The branch above just puts the two example
> docs directly into the index.rst for demo purposes.

Yes, the complicated part is establishing a structure for the
documentation (this should be done collaboratively on the wiki, I think).

Ultimately we should have three manuals: user, developer and hardware
specifications, but docs/ is currently a hodge-podge of the first two.
Once we have that, it is possible to add bits from docs/ into
Sphinx-built manuals, and/or disintegrate qemu-doc into docs/ files.

For what it's worth, I played with building QEMU's texinfo documentation
with Sphinx.  The result is at
http://people.redhat.com/pbonzini/qemu-test-doc/ and it works by
converting Texinfo->docbook (XML) with makeinfo, and then using a custom
parser (a Sphinx extension) to process the output.  It should be easy to
modify it so that Python spawns makeinfo and then pipes the output back
into the parser.

Personally, I prefer Texinfo (aka "LaTeX with @ instead of \" :)) to
either restructuredText and Markdown, but I understand that the more
nonstandard pieces you add to the toolchain has disadvantages, the more
it becomes a maintenance hassle.  However, I'll note that Marc-André's
work on automatically-generated QAPI documentation also used Texinfo.

I guess it also depends on whether we want to fork kerneldoc or not,
and/or whether the kernel guys are okay with adding another backend to
kerneldoc.

> Conclusions from this exercise:
> 1) conversion isn't all that difficult, and the results
>    look pretty nice
> 2) some of the doc-comment format differences are irritating:
>    . "function - short description" not "function: short description"
>    . "&struct.fieldname" not ".@fieldname"
>    . "&typename" not "#typename"
> 3) the most awkward part of kernel-doc syntax is that it bakes
>    in the kernel's style choice of always using "struct foo"
>    for types -- I don't think there's any way to document
>    'MemoryRegion' and 'AddressSpace' without the 'struct'
>    coming out in the documentation output.

I actually like having struct in the name, even if the code then doesn't
use it.

Paolo

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Peter Maydell]

On 7 November 2016 at 17:04, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 07/11/2016 16:03, Peter Maydell wrote:
>> The overall organisation structure needs some thought --
>> I think we should at least separate into user/ for user
>> docs and dev/ for internals docs

> Yes, the complicated part is establishing a structure for the
> documentation (this should be done collaboratively on the wiki, I think).
>
> Ultimately we should have three manuals: user, developer and hardware
> specifications, but docs/ is currently a hodge-podge of the first two.

User and developer, sure, but what's "hardware specifications" for?

>> 3) the most awkward part of kernel-doc syntax is that it bakes
>>    in the kernel's style choice of always using "struct foo"
>>    for types -- I don't think there's any way to document
>>    'MemoryRegion' and 'AddressSpace' without the 'struct'
>>    coming out in the documentation output.
>
> I actually like having struct in the name, even if the code then
> doesn't use it.

I think it would be good to at least be able to have '&MemoryRegion'
in a doc comment hyperlink to the documentation of the type --
currently that only works for '&struct MemoryRegion'.

Also it seems a bit odd for our coding style and documentation
style to be divergent, since it suggests to new developers
that they should be using 'struct' in their code.

thanks
-- PMM

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Paolo Bonzini]

On 07/11/2016 18:20, Peter Maydell wrote:
> On 7 November 2016 at 17:04, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> On 07/11/2016 16:03, Peter Maydell wrote:
>>> The overall organisation structure needs some thought --
>>> I think we should at least separate into user/ for user
>>> docs and dev/ for internals docs
> 
>> Yes, the complicated part is establishing a structure for the
>> documentation (this should be done collaboratively on the wiki, I think).
>>
>> Ultimately we should have three manuals: user, developer and hardware
>> specifications, but docs/ is currently a hodge-podge of the first two.
> 
> User and developer, sure, but what's "hardware specifications" for?

It's docs/specs.

>>> 3) the most awkward part of kernel-doc syntax is that it bakes
>>>    in the kernel's style choice of always using "struct foo"
>>>    for types -- I don't think there's any way to document
>>>    'MemoryRegion' and 'AddressSpace' without the 'struct'
>>>    coming out in the documentation output.
>>
>> I actually like having struct in the name, even if the code then
>> doesn't use it.
> 
> I think it would be good to at least be able to have '&MemoryRegion'
> in a doc comment hyperlink to the documentation of the type --
> currently that only works for '&struct MemoryRegion'.

Oh, got it now.  Yes, that would be more than just "nice to have".  What
do you think about requiring &struct in the doc comment, but then
omitting the "struct" in the generated documentation?

In any case, kerneldoc doesn't seem too rough to customize and (apart
from the latest flurry) it is touched very little in Linux.  And it also
includes other formats that Linux doesn't quite use, so perhaps Jon
Corbet would accept a patch for Texinfo.  If our changes were limited to
a bunch of changes $type_* at the top, it would be pretty good already.

Paolo

> Also it seems a bit odd for our coding style and documentation
> style to be divergent, since it suggests to new developers
> that they should be using 'struct' in their code.
> 
> thanks
> -- PMM
> 

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Peter Maydell]

On 7 November 2016 at 17:50, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 07/11/2016 18:20, Peter Maydell wrote:
>> User and developer, sure, but what's "hardware specifications" for?
>
> It's docs/specs.

Those are almost entirely third-party external stuff, unless
I'm confused about what you have in mind. The best we can do
is provide links to them, and even there those are likely to
break from time to time, drift out of date, etc...

thanks
-- PMM

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Paolo Bonzini]

On 07/11/2016 19:23, Peter Maydell wrote:
> On 7 November 2016 at 17:50, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> On 07/11/2016 18:20, Peter Maydell wrote:
>>> User and developer, sure, but what's "hardware specifications" for?
>>
>> It's docs/specs.
> 
> Those are almost entirely third-party external stuff,

No, they are defined by QEMU: fw_cfg, pvpanic, ivshmem, various
test devices.  The ACPI stuff refers to QEMU's implementation of
hotplug, too, and there is the registry of PCI ids defined by
QEMU.  It's not all hardware actually---there is stuff like
vhost-user, qcow2, etc.---but generally it's for interoperability.

It's not quite user documentation and definitely not developer
documentation.

Paolo

> unless
> I'm confused about what you have in mind. The best we can do
> is provide links to them, and even there those are likely to
> break from time to time, drift out of date, etc...
> 
> thanks
> -- PMM
> 
> 

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Peter Maydell]

On 7 November 2016 at 19:43, Paolo Bonzini <pbonzini@redhat.com> wrote:
> No, they are defined by QEMU: fw_cfg, pvpanic, ivshmem, various
> test devices.  The ACPI stuff refers to QEMU's implementation of
> hotplug, too, and there is the registry of PCI ids defined by
> QEMU.  It's not all hardware actually---there is stuff like
> vhost-user, qcow2, etc.---but generally it's for interoperability.
>
> It's not quite user documentation and definitely not developer
> documentation.

Oh, I see, that kind of guest-facing ABI. Yes, that
is a third section of its own.

thanks
-- PMM

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[John Snow]

On 11/07/2016 08:30 AM, Stefan Hajnoczi wrote:
> On Sat, Nov 05, 2016 at 06:42:23PM +0000, Peter Maydell wrote:
>> In particular I think we could:
>>  * set up a framework for our in-tree docs/ which gives us a
>>    place to put new docs (both for-users and for-developers) --
>>    I think having someplace to put things will reduce the barrier
>>    to people writing useful new docs
>>  * gradually convert the existing docs to rst
>>  * use the sphinx extension features to pull in the doc-comments
>>    we have been fairly consistently writing over the last few years
>>    (for instance a converted version of docs/memory.txt could pull
>>    in doc comments from memory.h; or we can just write simple
>>    wrapper files like a "Bitmap operations" document that
>>    displays the doc comments from bitops.h)
>
> You are suggesting Sphinx for two different purposes:
>
> 1. Formatting docs/ in HTML, PDF, etc.
>
> 2. API documentation from doc comments.
>
> It's a good idea for #1 since we can then publish automated builds of
> the docs.  They will be easy to view and link to in a web browser.
>
> I'm not a fan of #2.  QEMU is not a C library that people develop
> against and our APIs are not stable.  There is no incentive for pretty
> doc comments.  It might be cool to set it up once but things will
> deterioate again quickly because we don't actually need external API
> docs.
>
> Instead of #2 we should focus on generating nice external QMP docs for
> libvirt and other clients.  That has a clear benefit.
>
> Stefan
>

I think that designating certain interfaces within QEMU as "Internal 
API" has some merit and are worth documenting for the sake of 
device/format authors like Peter suggests.

Things may be in flux often, but if we can generate the docs from source 
code comments, I don't think it's unjust or unreasonable to request that 
patches keep these docs up to date.

It's error prone, of course, but certainly more manageable if we have a 
build tool doing some robotic checking of doc completeness for select 
interfaces. I think it's not possible to be more error prone than our 
current solution of "Random GTK-doc-like comments strewn about that may 
or may not be accurate, that we don't actually check or verify or even 
use for any doc-building purposes."

I'm also a fan of unifying our internal code documentation formats 
because it helps make the code look more consistent, but may also open 
up some parsing options for enhanced IDE support which could be nice for 
some.

I think at a minimum, having _A_ standard approach cannot possibly be 
*any* worse than _NO_ standard approach.

I'm a fan of the concept, but have no particular feelings on Sphinx yet.

--js

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Emilio G. Cota]

On Mon, Nov 07, 2016 at 15:03:23 +0000, Peter Maydell wrote:
> On 5 November 2016 at 18:42, Peter Maydell <peter.maydell@linaro.org> wrote:
> > With a little luck I may be able to put something up
> > on Monday as a sort of minimal-demonstration of how
> > this would look in QEMU.
> 
> Generated documentation:
>   http://people.linaro.org/~peter.maydell/sphinx/index.html
> Git branch with the patches needed to produce that:
>   https://git.linaro.org/people/peter.maydell/qemu-arm.git sphinx-docs
> Pointy-clicky interface to git branch:
>   https://git.linaro.org/people/peter.maydell/qemu-arm.git/log/?h=sphinx-docs
> 
> I didn't bother to write the makefile changes to tie it into
> the main build process, so to regenerate the docs locally you'll
> need to run
>  sphinx-build -b html docs my-build-dir/docs
> from the QEMU source tree root, which will put the output into
> my-build-dir/docs, which you can then point your web browser at.

I moved qht's documentation to this to see how hard it was.
Was trivial to do! The result looks very nice. 

Patches here:
- Web:	https://github.com/cota/qemu/tree/sphinx-docs
- Git:	https://github.com/cota/qemu.git sphinx-docs

> The overall organisation structure needs some thought --
> I think we should at least separate into user/ for user
> docs and dev/ for internals docs (and only install the
> user/ docs).

Agreed.

> The branch above just puts the two example
> docs directly into the index.rst for demo purposes.
> 
> Conclusions from this exercise:
> 1) conversion isn't all that difficult, and the results
>    look pretty nice
> 2) some of the doc-comment format differences are irritating:
>    . "function - short description" not "function: short description"
>    . "&struct.fieldname" not ".@fieldname"
>    . "&typename" not "#typename"
> 3) the most awkward part of kernel-doc syntax is that it bakes
>    in the kernel's style choice of always using "struct foo"
>    for types -- I don't think there's any way to document
>    'MemoryRegion' and 'AddressSpace' without the 'struct'
>    coming out in the documentation output.
> 
> We could fix (2) by loosening the kernel-doc script's
> parsing if we were happy to carry around a forked version
> of it. Fixing (3) requires more serious surgery on kernel-doc
> I suspect.

FWIW I'd prefer to strictly adhere to kerneldoc as is. Converting
the existing kerneldocs will require some supervision, anyway.

		E.

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2836 bytes --]

On Mon, Nov 07, 2016 at 05:52:37PM -0500, John Snow wrote:
> On 11/07/2016 08:30 AM, Stefan Hajnoczi wrote:
> > On Sat, Nov 05, 2016 at 06:42:23PM +0000, Peter Maydell wrote:
> > > In particular I think we could:
> > >  * set up a framework for our in-tree docs/ which gives us a
> > >    place to put new docs (both for-users and for-developers) --
> > >    I think having someplace to put things will reduce the barrier
> > >    to people writing useful new docs
> > >  * gradually convert the existing docs to rst
> > >  * use the sphinx extension features to pull in the doc-comments
> > >    we have been fairly consistently writing over the last few years
> > >    (for instance a converted version of docs/memory.txt could pull
> > >    in doc comments from memory.h; or we can just write simple
> > >    wrapper files like a "Bitmap operations" document that
> > >    displays the doc comments from bitops.h)
> > 
> > You are suggesting Sphinx for two different purposes:
> > 
> > 1. Formatting docs/ in HTML, PDF, etc.
> > 
> > 2. API documentation from doc comments.
> > 
> > It's a good idea for #1 since we can then publish automated builds of
> > the docs.  They will be easy to view and link to in a web browser.
> > 
> > I'm not a fan of #2.  QEMU is not a C library that people develop
> > against and our APIs are not stable.  There is no incentive for pretty
> > doc comments.  It might be cool to set it up once but things will
> > deterioate again quickly because we don't actually need external API
> > docs.
> > 
> > Instead of #2 we should focus on generating nice external QMP docs for
> > libvirt and other clients.  That has a clear benefit.
> > 
> > Stefan
> > 
> 
> I think that designating certain interfaces within QEMU as "Internal API"
> has some merit and are worth documenting for the sake of device/format
> authors like Peter suggests.

To be clear, I'm not saying QEMU doesn't need doc comments.  Every new
function in include/*.h must have doc comments and many .c internal
functions should too.

I'm just not enthusiastic about an effort to reformat doc comments and
make them render to HTML, PDF, etc in a nice way because I don't think
there's much payoff from doing that or maintaining it.

> I think at a minimum, having _A_ standard approach cannot possibly be *any*
> worse than _NO_ standard approach.

People don't follow the standard format and markup syntax since that
requires rendering and checking that the HTML, PDF, etc output looks
correct before submitting patches.

I guess one solution is to extend checkpatch.pl to enforce that all doc
comments follow a standard format.  It still cannot check that @, #, etc
are used in the right places but at least it can make sure that some
standard layout is followed.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[John Snow]

On 11/08/2016 11:20 AM, Stefan Hajnoczi wrote:
> On Mon, Nov 07, 2016 at 05:52:37PM -0500, John Snow wrote:
>> On 11/07/2016 08:30 AM, Stefan Hajnoczi wrote:
>>> On Sat, Nov 05, 2016 at 06:42:23PM +0000, Peter Maydell wrote:
>>>> In particular I think we could:
>>>>  * set up a framework for our in-tree docs/ which gives us a
>>>>    place to put new docs (both for-users and for-developers) --
>>>>    I think having someplace to put things will reduce the barrier
>>>>    to people writing useful new docs
>>>>  * gradually convert the existing docs to rst
>>>>  * use the sphinx extension features to pull in the doc-comments
>>>>    we have been fairly consistently writing over the last few years
>>>>    (for instance a converted version of docs/memory.txt could pull
>>>>    in doc comments from memory.h; or we can just write simple
>>>>    wrapper files like a "Bitmap operations" document that
>>>>    displays the doc comments from bitops.h)
>>>
>>> You are suggesting Sphinx for two different purposes:
>>>
>>> 1. Formatting docs/ in HTML, PDF, etc.
>>>
>>> 2. API documentation from doc comments.
>>>
>>> It's a good idea for #1 since we can then publish automated builds of
>>> the docs.  They will be easy to view and link to in a web browser.
>>>
>>> I'm not a fan of #2.  QEMU is not a C library that people develop
>>> against and our APIs are not stable.  There is no incentive for pretty
>>> doc comments.  It might be cool to set it up once but things will
>>> deterioate again quickly because we don't actually need external API
>>> docs.
>>>
>>> Instead of #2 we should focus on generating nice external QMP docs for
>>> libvirt and other clients.  That has a clear benefit.
>>>
>>> Stefan
>>>
>>
>> I think that designating certain interfaces within QEMU as "Internal API"
>> has some merit and are worth documenting for the sake of device/format
>> authors like Peter suggests.
>
> To be clear, I'm not saying QEMU doesn't need doc comments.  Every new
> function in include/*.h must have doc comments and many .c internal
> functions should too.
>
> I'm just not enthusiastic about an effort to reformat doc comments and
> make them render to HTML, PDF, etc in a nice way because I don't think
> there's much payoff from doing that or maintaining it.
>

OK, understood -- but if we are using a tool to syntax check the comment 
formats, which helps us keep consistency and our docs up to date, we get 
the PDF/html outputs "for free."

I agree they're not particularly useful, but I consider them a harmless 
side effect. They might also help prove to new contributors that we're 
serious about making QEMU easier to contribute to.

>> I think at a minimum, having _A_ standard approach cannot possibly be *any*
>> worse than _NO_ standard approach.
>
> People don't follow the standard format and markup syntax since that
> requires rendering and checking that the HTML, PDF, etc output looks
> correct before submitting patches.
>

My only experience is with Doxygen, but that at least does have warnings 
for a great number of things. As long as you're passing the generation 
checks, I think there's not much need to actually check the html/pdf 
outputs except periodically.

> I guess one solution is to extend checkpatch.pl to enforce that all doc
> comments follow a standard format.  It still cannot check that @, #, etc
> are used in the right places but at least it can make sure that some
> standard layout is followed.
>

This is the part that I'm hoping the generation tool can fulfill, 
assuming it has generation warnings like Doxygen does.

> Stefan
>

-- 
—js

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Paolo Bonzini]

On 07/11/2016 14:30, Stefan Hajnoczi wrote:
> On Sat, Nov 05, 2016 at 06:42:23PM +0000, Peter Maydell wrote:
>> In particular I think we could:
>>  * set up a framework for our in-tree docs/ which gives us a
>>    place to put new docs (both for-users and for-developers) --
>>    I think having someplace to put things will reduce the barrier
>>    to people writing useful new docs
>>  * gradually convert the existing docs to rst
>>  * use the sphinx extension features to pull in the doc-comments
>>    we have been fairly consistently writing over the last few years
>>    (for instance a converted version of docs/memory.txt could pull
>>    in doc comments from memory.h; or we can just write simple
>>    wrapper files like a "Bitmap operations" document that
>>    displays the doc comments from bitops.h)
> 
> You are suggesting Sphinx for two different purposes:
> 
> 1. Formatting docs/ in HTML, PDF, etc.
> 
> 2. API documentation from doc comments.
> 
> It's a good idea for #1 since we can then publish automated builds of
> the docs.  They will be easy to view and link to in a web browser.
> 
> I'm not a fan of #2.  QEMU is not a C library that people develop
> against and our APIs are not stable.  There is no incentive for pretty
> doc comments.  It might be cool to set it up once but things will
> deterioate again quickly because we don't actually need external API
> docs.

I don't think pretty doc comments matter, but accurate doc comments do.
If we cannot have accurate doc comments, we might not have them at all,
but this is actually not the case.  There are some areas where we
actually go to great(er) lengths to have up-to-date documentation and
up-to-date doc comments, and it's a pity to only provide half of the
information in an easily consumable format.

It doesn't really have to be perfect, but it's a nice thing to have.
I'm not entirely sure that it's interesting to format bitops.h's doc
comments, but for memory.h or aio.h I'm pretty sure it's worth it.

Paolo

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4344 bytes --]

On Tue, Nov 08, 2016 at 11:24:21AM -0500, John Snow wrote:
> 
> 
> On 11/08/2016 11:20 AM, Stefan Hajnoczi wrote:
> > On Mon, Nov 07, 2016 at 05:52:37PM -0500, John Snow wrote:
> > > On 11/07/2016 08:30 AM, Stefan Hajnoczi wrote:
> > > > On Sat, Nov 05, 2016 at 06:42:23PM +0000, Peter Maydell wrote:
> > > > > In particular I think we could:
> > > > >  * set up a framework for our in-tree docs/ which gives us a
> > > > >    place to put new docs (both for-users and for-developers) --
> > > > >    I think having someplace to put things will reduce the barrier
> > > > >    to people writing useful new docs
> > > > >  * gradually convert the existing docs to rst
> > > > >  * use the sphinx extension features to pull in the doc-comments
> > > > >    we have been fairly consistently writing over the last few years
> > > > >    (for instance a converted version of docs/memory.txt could pull
> > > > >    in doc comments from memory.h; or we can just write simple
> > > > >    wrapper files like a "Bitmap operations" document that
> > > > >    displays the doc comments from bitops.h)
> > > > 
> > > > You are suggesting Sphinx for two different purposes:
> > > > 
> > > > 1. Formatting docs/ in HTML, PDF, etc.
> > > > 
> > > > 2. API documentation from doc comments.
> > > > 
> > > > It's a good idea for #1 since we can then publish automated builds of
> > > > the docs.  They will be easy to view and link to in a web browser.
> > > > 
> > > > I'm not a fan of #2.  QEMU is not a C library that people develop
> > > > against and our APIs are not stable.  There is no incentive for pretty
> > > > doc comments.  It might be cool to set it up once but things will
> > > > deterioate again quickly because we don't actually need external API
> > > > docs.
> > > > 
> > > > Instead of #2 we should focus on generating nice external QMP docs for
> > > > libvirt and other clients.  That has a clear benefit.
> > > > 
> > > > Stefan
> > > > 
> > > 
> > > I think that designating certain interfaces within QEMU as "Internal API"
> > > has some merit and are worth documenting for the sake of device/format
> > > authors like Peter suggests.
> > 
> > To be clear, I'm not saying QEMU doesn't need doc comments.  Every new
> > function in include/*.h must have doc comments and many .c internal
> > functions should too.
> > 
> > I'm just not enthusiastic about an effort to reformat doc comments and
> > make them render to HTML, PDF, etc in a nice way because I don't think
> > there's much payoff from doing that or maintaining it.
> > 
> 
> OK, understood -- but if we are using a tool to syntax check the comment
> formats, which helps us keep consistency and our docs up to date, we get the
> PDF/html outputs "for free."
> 
> I agree they're not particularly useful, but I consider them a harmless side
> effect. They might also help prove to new contributors that we're serious
> about making QEMU easier to contribute to.
> 
> > > I think at a minimum, having _A_ standard approach cannot possibly be *any*
> > > worse than _NO_ standard approach.
> > 
> > People don't follow the standard format and markup syntax since that
> > requires rendering and checking that the HTML, PDF, etc output looks
> > correct before submitting patches.
> > 
> 
> My only experience is with Doxygen, but that at least does have warnings for
> a great number of things. As long as you're passing the generation checks, I
> think there's not much need to actually check the html/pdf outputs except
> periodically.
> 
> > I guess one solution is to extend checkpatch.pl to enforce that all doc
> > comments follow a standard format.  It still cannot check that @, #, etc
> > are used in the right places but at least it can make sure that some
> > standard layout is followed.
> > 
> 
> This is the part that I'm hoping the generation tool can fulfill, assuming
> it has generation warnings like Doxygen does.

If we can check that new functions have doc comments and the comments
follow the right format, then I'm happy.

No doc comments -> error.  Comments but not in doc format -> error.

The check needs to be run before submitting patches and also by patchew.
Otherwise those people who don't run the doc generator can fly under the
radar ;-).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Fam Zheng]

On Wed, 11/09 11:32, Stefan Hajnoczi wrote:
> No doc comments -> error.

I'm not sure that is a good idea. For example all .bdrv_co_flush_to_disk
implementations have the same semantics and signature, requiring doc comments
everywhere might be too much.

Fam

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 365 bytes --]

On Thu, Nov 10, 2016 at 11:39:14AM +0800, Fam Zheng wrote:
> On Wed, 11/09 11:32, Stefan Hajnoczi wrote:
> > No doc comments -> error.
> 
> I'm not sure that is a good idea. For example all .bdrv_co_flush_to_disk
> implementations have the same semantics and signature, requiring doc comments
> everywhere might be too much.

I mean in include/.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Paolo Bonzini]

----- Original Message -----
> From: "Fam Zheng" <famz@redhat.com>
> To: "Stefan Hajnoczi" <stefanha@gmail.com>
> Cc: "John Snow" <jsnow@redhat.com>, "Peter Maydell" <peter.maydell@linaro.org>, "QEMU Developers"
> <qemu-devel@nongnu.org>, "Stefan Hajnoczi" <stefanha@redhat.com>, "Paolo Bonzini" <pbonzini@redhat.com>
> Sent: Thursday, November 10, 2016 4:39:14 AM
> Subject: Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)
> 
> On Wed, 11/09 11:32, Stefan Hajnoczi wrote:
> > No doc comments -> error.
> 
> I'm not sure that is a good idea. For example all .bdrv_co_flush_to_disk
> implementations have the same semantics and signature, requiring doc comments
> everywhere might be too much.

The check would be only on specific files.  However, I find it hard to
implement it if we place doc comments for types in headers and those for
functions in .c files (with automatically generated docs, most of the
advantages of doc comments in headers go away).

Paolo

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Daniel P. Berrange]

On Thu, Nov 10, 2016 at 05:08:03AM -0500, Paolo Bonzini wrote:
> 
> 
> ----- Original Message -----
> > From: "Fam Zheng" <famz@redhat.com>
> > To: "Stefan Hajnoczi" <stefanha@gmail.com>
> > Cc: "John Snow" <jsnow@redhat.com>, "Peter Maydell" <peter.maydell@linaro.org>, "QEMU Developers"
> > <qemu-devel@nongnu.org>, "Stefan Hajnoczi" <stefanha@redhat.com>, "Paolo Bonzini" <pbonzini@redhat.com>
> > Sent: Thursday, November 10, 2016 4:39:14 AM
> > Subject: Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)
> > 
> > On Wed, 11/09 11:32, Stefan Hajnoczi wrote:
> > > No doc comments -> error.
> > 
> > I'm not sure that is a good idea. For example all .bdrv_co_flush_to_disk
> > implementations have the same semantics and signature, requiring doc comments
> > everywhere might be too much.
> 
> The check would be only on specific files.  However, I find it hard to
> implement it if we place doc comments for types in headers and those for
> functions in .c files (with automatically generated docs, most of the
> advantages of doc comments in headers go away).

Another reason for using .h file for all API docs - we have APIs
declared for crypto impls in .h files, but there are 3 separate
implementation files chosen between at build time, so no single .c
file is "right" for holding the docs.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] Sphinx for QEMU docs? (and a doc-comment format question)

[Paolo Bonzini]

On 07/11/2016 16:03, Peter Maydell wrote:
> 2) some of the doc-comment format differences are irritating:
>    . "function - short description" not "function: short description"
>    . "&struct.fieldname" not ".@fieldname"
>    . "&typename" not "#typename"
> 3) the most awkward part of kernel-doc syntax is that it bakes
>    in the kernel's style choice of always using "struct foo"
>    for types -- I don't think there's any way to document
>    'MemoryRegion' and 'AddressSpace' without the 'struct'
>    coming out in the documentation output.
> 
> We could fix (2) by loosening the kernel-doc script's
> parsing if we were happy to carry around a forked version
> of it. Fixing (3) requires more serious surgery on kernel-doc
> I suspect.

I've sent some changes to kernel-doc to simplify the implementation of
these changes (http://www.spinics.net/lists/linux-doc/msg42354.html) and
they were accepted.  So with 4.10 + those patches, the local changes to
kernel-doc for QEMU would be limited to the following:

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 4c9ada36fe6b..c43ac038398d 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -215,18 +215,18 @@ my $type_func = '(\w+)\(\)';
 my $type_param = '\@(\w+(\.\.\.)?)';
 my $type_fp_param = '\@(\w+)\(\)';  # Special RST handling for func ptr params
 my $type_env = '(\$\w+)';
-my $type_enum = '\&(enum\s*([_\w]+))';
-my $type_struct = '\&(struct\s*([_\w]+))';
-my $type_typedef = '\&(typedef\s*([_\w]+))';
-my $type_union = '\&(union\s*([_\w]+))';
-my $type_member = '\&([_\w]+)(\.|->)([_\w]+)';
-my $type_fallback = '\&([_\w]+)';
-my $type_enum_xml = '\&(enum\s*([_\w]+))';
-my $type_struct_xml = '\&(struct\s*([_\w]+))';
-my $type_typedef_xml = '\&(typedef\s*([_\w]+))';
-my $type_union_xml = '\&(union\s*([_\w]+))';
-my $type_member_xml = '\&([_\w]+)(\.|-\>)([_\w]+)';
-my $type_fallback_xml = '\&amp([_\w]+)';
+my $type_enum = '#(enum\s*([_\w]+))';
+my $type_struct = '#(struct\s*([_\w]+))';
+my $type_typedef = '#(([A-Z][_\w]*))';
+my $type_union = '#(union\s*([_\w]+))';
+my $type_member = '#([_\w]+)(\.|->)([_\w]+)';
+my $type_fallback = '(?!)';    # this never matches
+my $type_enum_xml = $type_enum;
+my $type_struct_xml = $type_struct;
+my $type_typedef_xml = $type_typedef;
+my $type_union_xml = $type_union;
+my $type_member_xml = $type_member;
+my $type_fallback_xml = $type_fallback;
 my $type_member_func = $type_member . '\(\)';
 
 # Output conversion substitutions.
@@ -2143,6 +2143,14 @@ sub output_blockhead {
 sub dump_declaration($$) {
     no strict 'refs';
     my ($prototype, $file) = @_;
+    if ($decl_type eq 'type name') {
+	if ($prototype =~ /^(enum|struct|union)\s+/) {
+	    $decl_type = $1;
+        } else {
+	    return;
+	}
+    }
+
     my $func = "dump_" . $decl_type;
     &$func(@_);
 }
@@ -2893,7 +2901,7 @@ sub process_file($) {
 	    }
 	    elsif (/$doc_decl/o) {
 		$identifier = $1;
-		if (/\s*([\w\s]+?)\s*-/) {
+		if (/\s*([\w\s]+?)(\s*-|:)/) {
 		    $identifier = $1;
 		}
 
@@ -2903,7 +2911,7 @@ sub process_file($) {
 		$contents = "";
 		$section = $section_default;
 		$new_start_line = $. + 1;
-		if (/-(.*)/) {
+		if (/[-:](.*)/) {
 		    # strip leading/trailing/multiple spaces
 		    $descr= $1;
 		    $descr =~ s/^\s*//;
@@ -2921,7 +2929,9 @@ sub process_file($) {
 			++$warnings;
 		}
 
-		if ($identifier =~ m/^struct/) {
+		if ($identifier =~ m/^[A-Z]/) {
+		    $decl_type = 'type name';
+	        } elsif ($identifier =~ m/^struct/) {
 		    $decl_type = 'struct';
 		} elsif ($identifier =~ m/^union/) {
 		    $decl_type = 'union';

which should be maintainable as a fork of Linux's kernel-doc.

I also worked a bit on support for Texinfo manuals in Sphinx.  My
current attempt is at http://people.redhat.com/pbonzini/qemu-test-doc/_build/.
Because this uses a Texinfo->Docbook->Sphinx pipeline, I also tried some
tools with native Docbook support (Publican), but despite Sphinx's quirks
the output was less usable, and the tools were slower and harder to use.

http://wiki.qemu-project.org/Features/Documentation is another place to
brainstorm ideas on this.

Paolo