Proposal for handling .hx files with Sphinx

Proposal for handling .hx files with Sphinx

[Peter Maydell]

Currently our manual creation includes some .texi files which
are autogenerated from .hx files by running scripts/hxtool.
.hx files are a simple format, where where a line is either a
directive or literal text to be output:

HXCOMM
 -- comment lines, ignored
STEXI/ETEXI
 -- mark start/end of chunks of text to put into the texinfo output only
DEFHEADING/ARCHHEADING
 -- appear in the .h file output verbatim (they are defined as C macros);
    for texi output they are parsed to add in header sections

For Sphinx, rather than creating a file to include, the most
natural way to handle this is to have a small custom Sphinx
extension which will read the .hx file and process it. So
instead of "makefile produces foo.texi from foo.hx, qemu-doc.texi
says '@include foo.texi'", we have "qemu-doc.rst says
'hxtool-doc:: foo.hx', the Sphinx extension for hxtool has
code that runs to handle that Sphinx directive, it reads the .hx
file and emits the appropriate documentation contents". (This is
pretty much the same way the kerneldoc extension works right now.
It also has the advantage that it should work for third-party
services like readthedocs that expect to build the docs directly
with sphinx rather than by invoking our makefiles.)

We'll need to update what the markup is to handle having rST
fragments in it. A very minimalist approach to this would
simply define a new pair of SRST/ERST directives marking the
start/end of chunks of rST text to go into the rST only.
(We might be able to do better than that later, as there's
some repetition currently going on. But we'll probably get
a better idea of how easy it is to avoid the repetition if
we start with a simple conversion.)

Here's what we do with hx files at the moment. We have four:

 hmp-commands.hx
   -- defines monitor commands used by monitor.c; generates
      qemu-monitor.texi, used by qemu-doc.texi
 hmp-commands-info.hx
   -- ditto, for the "info" command's subcommand;
      generates qemu-monitor-info.texi, used by qemu-doc.texi

These two use only the "put this in the texi or in the .h file"
functionality, alternating "raw C code defining an entry for the
monitor command array" with "lump of raw texi for the docs".

 qemu-img-cmds.hx
   -- defines options for qemu-img, used by qemu-img.texi

This uses the STEXI/ETEXI directives to alternate C and texi.
In the for-the-h-file section the only content is always a DEF()
macro invocation defining the option; in the texi is only the
synopsis of the command. This means there's a lot of repetition,
as the DEF macro includes an argument giving the text of the
option synopsis, and then the texi also has that synopsis with
some extra markup. Finally the main qemu-img.texi repeats the
marked-up synopsis later on when it has the actual main documentation
of each command.

 qemu-options.hx
   -- options for qemu proper, used by qemu-doc.texi

This uses only the DEF, DEFHEADING, ARCHHEADING macros
in the for-the-h-file sections (and the DEFHEADING/ARCHHEADING
are read also for texi generation). This also repeats the
synopsis in the DEF macro and in the texi fragment.

So I think my current view is that we should do the very
simple "add SRST/ERST directives" to start with:
 * scripts/hxtool needs to recognize them and just ignore
   the text inside them
 * write the hxtool sphinx extension (shouldn't be too hard)
 * conversion of any particular .hx file then involves
   replacing the STEXI ...texi stuff... ETEXI sections with
   SRST ...rst stuff... ERST. There's no need for any
   particular .hx file to support both texi and rst output
   at the same time

I would initially start with qemu-img-cmds.hx, since that's
pulled in by qemu-img.texi, which we can convert in the
same way I've been doing qemu-nbd and other standalone-ish
manpages. The others are part of the big fat qemu-doc.texi,
which is probably going to be the very last thing we convert...

thanks
-- PMM

Re: Proposal for handling .hx files with Sphinx

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 658 bytes --]

On Fri, Jan 17, 2020 at 05:30:43PM +0000, Peter Maydell wrote:
> So I think my current view is that we should do the very
> simple "add SRST/ERST directives" to start with:
>  * scripts/hxtool needs to recognize them and just ignore
>    the text inside them
>  * write the hxtool sphinx extension (shouldn't be too hard)
>  * conversion of any particular .hx file then involves
>    replacing the STEXI ...texi stuff... ETEXI sections with
>    SRST ...rst stuff... ERST. There's no need for any
>    particular .hx file to support both texi and rst output
>    at the same time

Nice idea, I was wondering how we were going to deal with .hx files.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Proposal for handling .hx files with Sphinx

[John Snow]

On 1/17/20 12:30 PM, Peter Maydell wrote:
> Currently our manual creation includes some .texi files which
> are autogenerated from .hx files by running scripts/hxtool.
> .hx files are a simple format, where where a line is either a
> directive or literal text to be output:
> 
> HXCOMM
>  -- comment lines, ignored
> STEXI/ETEXI
>  -- mark start/end of chunks of text to put into the texinfo output only
> DEFHEADING/ARCHHEADING
>  -- appear in the .h file output verbatim (they are defined as C macros);
>     for texi output they are parsed to add in header sections
> 
> For Sphinx, rather than creating a file to include, the most
> natural way to handle this is to have a small custom Sphinx
> extension which will read the .hx file and process it. So
> instead of "makefile produces foo.texi from foo.hx, qemu-doc.texi
> says '@include foo.texi'", we have "qemu-doc.rst says
> 'hxtool-doc:: foo.hx', the Sphinx extension for hxtool has
> code that runs to handle that Sphinx directive, it reads the .hx
> file and emits the appropriate documentation contents". (This is
> pretty much the same way the kerneldoc extension works right now.
> It also has the advantage that it should work for third-party
> services like readthedocs that expect to build the docs directly
> with sphinx rather than by invoking our makefiles.)
> 
> We'll need to update what the markup is to handle having rST
> fragments in it. A very minimalist approach to this would
> simply define a new pair of SRST/ERST directives marking the
> start/end of chunks of rST text to go into the rST only.
> (We might be able to do better than that later, as there's
> some repetition currently going on. But we'll probably get
> a better idea of how easy it is to avoid the repetition if
> we start with a simple conversion.)
> 
> Here's what we do with hx files at the moment. We have four:
> 
>  hmp-commands.hx
>    -- defines monitor commands used by monitor.c; generates
>       qemu-monitor.texi, used by qemu-doc.texi
>  hmp-commands-info.hx
>    -- ditto, for the "info" command's subcommand;
>       generates qemu-monitor-info.texi, used by qemu-doc.texi
> 
> These two use only the "put this in the texi or in the .h file"
> functionality, alternating "raw C code defining an entry for the
> monitor command array" with "lump of raw texi for the docs".
> 
>  qemu-img-cmds.hx
>    -- defines options for qemu-img, used by qemu-img.texi
> 
> This uses the STEXI/ETEXI directives to alternate C and texi.
> In the for-the-h-file section the only content is always a DEF()
> macro invocation defining the option; in the texi is only the
> synopsis of the command. This means there's a lot of repetition,
> as the DEF macro includes an argument giving the text of the
> option synopsis, and then the texi also has that synopsis with
> some extra markup. Finally the main qemu-img.texi repeats the
> marked-up synopsis later on when it has the actual main documentation
> of each command.
> 
>  qemu-options.hx
>    -- options for qemu proper, used by qemu-doc.texi
> 
> This uses only the DEF, DEFHEADING, ARCHHEADING macros
> in the for-the-h-file sections (and the DEFHEADING/ARCHHEADING
> are read also for texi generation). This also repeats the
> synopsis in the DEF macro and in the texi fragment.
> 
> So I think my current view is that we should do the very
> simple "add SRST/ERST directives" to start with:
>  * scripts/hxtool needs to recognize them and just ignore
>    the text inside them
>  * write the hxtool sphinx extension (shouldn't be too hard)
>  * conversion of any particular .hx file then involves
>    replacing the STEXI ...texi stuff... ETEXI sections with
>    SRST ...rst stuff... ERST. There's no need for any
>    particular .hx file to support both texi and rst output
>    at the same time
> 
> I would initially start with qemu-img-cmds.hx, since that's
> pulled in by qemu-img.texi, which we can convert in the
> same way I've been doing qemu-nbd and other standalone-ish
> manpages. The others are part of the big fat qemu-doc.texi,
> which is probably going to be the very last thing we convert...
> 

At one point I did a quick mockup of turning qemu-img-cmds.hx into json
and wrote a small tool I called "pxtool" that was used for generating
all the rest of the subsequent information -- an attempt at getting rid
of .hx files *entirely*.

The idea at heart was: "Can we remove .hx files and describe everything
in terms of the QAPI schema instead?"

I'm still a bit partial to that idea, but realize there are some nasty
complexities when it comes to describing the QEMU CLI as a schema. One
of those is that I doubt we even have a full understanding of what the
CLI syntax is at all.

Still, I do want to ask: Are we sure we want to double-down on keeping
the .hx files around instead of trying to move to a more generic data
format?

Re: Proposal for handling .hx files with Sphinx

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 1/17/20 12:30 PM, Peter Maydell wrote:
>> Currently our manual creation includes some .texi files which
>> are autogenerated from .hx files by running scripts/hxtool.
>> .hx files are a simple format, where where a line is either a
>> directive or literal text to be output:
>> 
>> HXCOMM
>>  -- comment lines, ignored
>> STEXI/ETEXI
>>  -- mark start/end of chunks of text to put into the texinfo output only
>> DEFHEADING/ARCHHEADING
>>  -- appear in the .h file output verbatim (they are defined as C macros);
>>     for texi output they are parsed to add in header sections
>> 
>> For Sphinx, rather than creating a file to include, the most
>> natural way to handle this is to have a small custom Sphinx
>> extension which will read the .hx file and process it. So
>> instead of "makefile produces foo.texi from foo.hx, qemu-doc.texi
>> says '@include foo.texi'", we have "qemu-doc.rst says
>> 'hxtool-doc:: foo.hx', the Sphinx extension for hxtool has
>> code that runs to handle that Sphinx directive, it reads the .hx
>> file and emits the appropriate documentation contents". (This is
>> pretty much the same way the kerneldoc extension works right now.
>> It also has the advantage that it should work for third-party
>> services like readthedocs that expect to build the docs directly
>> with sphinx rather than by invoking our makefiles.)
>> 
>> We'll need to update what the markup is to handle having rST
>> fragments in it. A very minimalist approach to this would
>> simply define a new pair of SRST/ERST directives marking the
>> start/end of chunks of rST text to go into the rST only.
>> (We might be able to do better than that later, as there's
>> some repetition currently going on. But we'll probably get
>> a better idea of how easy it is to avoid the repetition if
>> we start with a simple conversion.)
>> 
>> Here's what we do with hx files at the moment. We have four:
>> 
>>  hmp-commands.hx
>>    -- defines monitor commands used by monitor.c; generates
>>       qemu-monitor.texi, used by qemu-doc.texi
>>  hmp-commands-info.hx
>>    -- ditto, for the "info" command's subcommand;
>>       generates qemu-monitor-info.texi, used by qemu-doc.texi
>> 
>> These two use only the "put this in the texi or in the .h file"
>> functionality, alternating "raw C code defining an entry for the
>> monitor command array" with "lump of raw texi for the docs".
>> 
>>  qemu-img-cmds.hx
>>    -- defines options for qemu-img, used by qemu-img.texi
>> 
>> This uses the STEXI/ETEXI directives to alternate C and texi.
>> In the for-the-h-file section the only content is always a DEF()
>> macro invocation defining the option; in the texi is only the
>> synopsis of the command. This means there's a lot of repetition,
>> as the DEF macro includes an argument giving the text of the
>> option synopsis, and then the texi also has that synopsis with
>> some extra markup. Finally the main qemu-img.texi repeats the
>> marked-up synopsis later on when it has the actual main documentation
>> of each command.
>> 
>>  qemu-options.hx
>>    -- options for qemu proper, used by qemu-doc.texi
>> 
>> This uses only the DEF, DEFHEADING, ARCHHEADING macros
>> in the for-the-h-file sections (and the DEFHEADING/ARCHHEADING
>> are read also for texi generation). This also repeats the
>> synopsis in the DEF macro and in the texi fragment.
>> 
>> So I think my current view is that we should do the very
>> simple "add SRST/ERST directives" to start with:
>>  * scripts/hxtool needs to recognize them and just ignore
>>    the text inside them
>>  * write the hxtool sphinx extension (shouldn't be too hard)
>>  * conversion of any particular .hx file then involves
>>    replacing the STEXI ...texi stuff... ETEXI sections with
>>    SRST ...rst stuff... ERST. There's no need for any
>>    particular .hx file to support both texi and rst output
>>    at the same time
>> 
>> I would initially start with qemu-img-cmds.hx, since that's
>> pulled in by qemu-img.texi, which we can convert in the
>> same way I've been doing qemu-nbd and other standalone-ish
>> manpages. The others are part of the big fat qemu-doc.texi,
>> which is probably going to be the very last thing we convert...
>> 
>
> At one point I did a quick mockup of turning qemu-img-cmds.hx into json
> and wrote a small tool I called "pxtool" that was used for generating
> all the rest of the subsequent information -- an attempt at getting rid
> of .hx files *entirely*.
>
> The idea at heart was: "Can we remove .hx files and describe everything
> in terms of the QAPI schema instead?"
>
> I'm still a bit partial to that idea, but realize there are some nasty
> complexities when it comes to describing the QEMU CLI as a schema. One
> of those is that I doubt we even have a full understanding of what the
> CLI syntax is at all.

My CLI QAPIfication prototype[*] gets rid of qemu-options.hx.  Three
more are left: hmp-commands.hx, hmp-commands-info.hx, qemu-img-cmds.hx.
No idea whether these could and should be QAPIfied.

Going beyond prototype is hard, not least for the reason you mentioned.

> Still, I do want to ask: Are we sure we want to double-down on keeping
> the .hx files around instead of trying to move to a more generic data
> format?

One the one hand, I'd prefer to invest as little as practical into .hx.
On the other hand, adding more hard dependencies on QAPIfication is not
a good idea.

What's the stupidest solution that could possibly work now?  Is it the
one Peter sketched?



[*] https://lists.nongnu.org/archive/html/qemu-devel/2017-10/msg00209.html
Message-Id: <20171002152552.27999-1-armbru@redhat.com>
https://repo.or.cz/qemu/armbru.git/shortlog/refs/heads/qapi-cmdline

Re: Proposal for handling .hx files with Sphinx

[Peter Maydell]

On Tue, 21 Jan 2020 at 06:40, Markus Armbruster <armbru@redhat.com> wrote:
> John Snow <jsnow@redhat.com> writes:
> > Still, I do want to ask: Are we sure we want to double-down on keeping
> > the .hx files around instead of trying to move to a more generic data
> > format?
>
> One the one hand, I'd prefer to invest as little as practical into .hx.
> On the other hand, adding more hard dependencies on QAPIfication is not
> a good idea.
>
> What's the stupidest solution that could possibly work now?  Is it the
> one Peter sketched?

FWIW, I wrote some code for the Sphinx extension approach yesterday,
along the 'simplest possible thing' lines. It's less than 200 lines
of Python (though I still need to put in the support for DEFHEADING
and ARCHHEADING). The actual texinfo fragments in the various .hx
files of course would also need to be hand-converted to rST, same
as the hand-written manual .texi file contents.

(This has been easier than my last half-attempt at
updating qapi-gen to support output of rST format...)

thanks
-- PMM

Re: Proposal for handling .hx files with Sphinx

[Peter Maydell]

On Tue, 21 Jan 2020 at 11:12, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Tue, 21 Jan 2020 at 06:40, Markus Armbruster <armbru@redhat.com> wrote:
> > John Snow <jsnow@redhat.com> writes:
> > > Still, I do want to ask: Are we sure we want to double-down on keeping
> > > the .hx files around instead of trying to move to a more generic data
> > > format?
> >
> > One the one hand, I'd prefer to invest as little as practical into .hx.
> > On the other hand, adding more hard dependencies on QAPIfication is not
> > a good idea.
> >
> > What's the stupidest solution that could possibly work now?  Is it the
> > one Peter sketched?
>
> FWIW, I wrote some code for the Sphinx extension approach yesterday,
> along the 'simplest possible thing' lines. It's less than 200 lines
> of Python (though I still need to put in the support for DEFHEADING
> and ARCHHEADING). The actual texinfo fragments in the various .hx
> files of course would also need to be hand-converted to rST, same
> as the hand-written manual .texi file contents.

Incidentally, I am definitely coming to the conclusion that the
best way to do generation of docs to go in Sphinx manuals is
to use/write a Sphinx extension -- this lets you properly create
doctree nodes, for instance and it fits the flow better. So a
in potential future where we were generating these docs from
json, I think we'd want to have a Sphinx extension driving the
'parse the json for docs', rather than a separate script that
spit out rst-format-text to include.

thanks
-- PMM

Re: Proposal for handling .hx files with Sphinx

[Markus Armbruster]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Tue, 21 Jan 2020 at 11:12, Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> On Tue, 21 Jan 2020 at 06:40, Markus Armbruster <armbru@redhat.com> wrote:
>> > John Snow <jsnow@redhat.com> writes:
>> > > Still, I do want to ask: Are we sure we want to double-down on keeping
>> > > the .hx files around instead of trying to move to a more generic data
>> > > format?
>> >
>> > One the one hand, I'd prefer to invest as little as practical into .hx.
>> > On the other hand, adding more hard dependencies on QAPIfication is not
>> > a good idea.
>> >
>> > What's the stupidest solution that could possibly work now?  Is it the
>> > one Peter sketched?
>>
>> FWIW, I wrote some code for the Sphinx extension approach yesterday,
>> along the 'simplest possible thing' lines. It's less than 200 lines
>> of Python (though I still need to put in the support for DEFHEADING
>> and ARCHHEADING). The actual texinfo fragments in the various .hx
>> files of course would also need to be hand-converted to rST, same
>> as the hand-written manual .texi file contents.
>
> Incidentally, I am definitely coming to the conclusion that the
> best way to do generation of docs to go in Sphinx manuals is
> to use/write a Sphinx extension -- this lets you properly create
> doctree nodes, for instance and it fits the flow better. So a
> in potential future where we were generating these docs from
> json, I think we'd want to have a Sphinx extension driving the
> 'parse the json for docs', rather than a separate script that
> spit out rst-format-text to include.

I trust your judgement.

Since the Sphinx extension is in Python, we can try to reuse the QAPI
frontend there.  We'll see.

I'd like to encourage you to not feel bound to the existing doc comment
language.  When Marc-André created the doc generator, he chose to fit
the doc comment language to the existing doc comments, to avoid churn.
That was probably the right decision then; getting the job done was
tough enough as it was.  The resulting language was basically defined by
its (ad hoc!) parser.  I cleaned up the parser some, and
retro-documented the syntax.  It's still an idiosyncratic mess, to be
honest.