Sphinx and the :any: role

Sphinx and the :any: role

[John Snow]

For the purposes of my QAPI generator cleanup (adding docstrings and 
mypy static types to ./scripts/qapi), I have a series of patches I've 
been carrying off-list that changes the behavior of how Sphinx 
cross-references work.

Presently, `foo` in our Sphinx docs is interpreted as the "content" 
role, and has "no special meaning".

I experimented with what would happen if we change this to the "any" 
role, which interprets `foo` as a cross-reference to *something* named 
"foo". If no reference is found, it emits a build error.

(Many existing usages of `foo` must be changed to ``foo`` to enable this 
to work.)

I found this useful for testing and writing Python docstrings, because 
it keeps the syntax cleaner than having to write :py:func:`foo` or 
:py:meth:`foo` and so on. It keeps the docstrings less syntax-heavy when 
reading them outside of the Sphinx rendered output.

I would have liked to propose we actually adopt this change, however 
I've noticed a problem with this in our CI builds.

Sphinx 3.2.1 (The current version we seem to test against in our Fedora 
container) has a race condition (?) where occasionally kernel-doc will 
crash outright:

Exception occurred:
   File "/usr/lib/python3.9/site-packages/sphinx/domains/c.py", line 
3751, in resolve_any_xref
     return [('c:' + self.role_for_objtype(objtype), retnode)]
TypeError: can only concatenate str (not "NoneType") to str

The full traceback has been saved in /tmp/sphinx-err-v6iqwtf9.log, if 
you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error 
message can be provided next time.
A bug report can be filed in the tracker at 
<https://github.com/sphinx-doc/sphinx/issues>. Thanks!


It's not actually clear WHICH thing Sphinx is choking on here, exactly 
-- it's just some bug in Sphinx's C domain handling of resolving 
cross-references.

This is fixed in 3.3.1, I believe closed with one of these issues:

https://github.com/sphinx-doc/sphinx/issues/8364
https://github.com/sphinx-doc/sphinx/issues/8388


I'm not sure if there's anything I can do here -- we probably can't 
require such a modern version of Sphinx just yet, I haven't done the 
legwork to check. I also haven't been able to diagnose the exact 
invocation that seems to trigger these failures, so it's not clear if 
there's a workaround just yet.

I'm still interested in making our cross-referencing a bit more robust, 
but maybe my hands are tied on that front for now.

--js

Re: Sphinx and the :any: role

[John Snow]

On 5/11/21 1:31 PM, John Snow wrote:
> For the purposes of my QAPI generator cleanup (adding docstrings and 
> mypy static types to ./scripts/qapi), I have a series of patches I've 
> been carrying off-list that changes the behavior of how Sphinx 
> cross-references work.
> 
> Presently, `foo` in our Sphinx docs is interpreted as the "content" 
> role, and has "no special meaning".
> 
> I experimented with what would happen if we change this to the "any" 
> role, which interprets `foo` as a cross-reference to *something* named 
> "foo". If no reference is found, it emits a build error.
> 
> (Many existing usages of `foo` must be changed to ``foo`` to enable this 
> to work.)
> 
> I found this useful for testing and writing Python docstrings, because 
> it keeps the syntax cleaner than having to write :py:func:`foo` or 
> :py:meth:`foo` and so on. It keeps the docstrings less syntax-heavy when 
> reading them outside of the Sphinx rendered output.
> 
> I would have liked to propose we actually adopt this change, however 
> I've noticed a problem with this in our CI builds.
> 
> Sphinx 3.2.1 (The current version we seem to test against in our Fedora 
> container) has a race condition (?) where occasionally kernel-doc will 
> crash outright:
> 
> Exception occurred:
>    File "/usr/lib/python3.9/site-packages/sphinx/domains/c.py", line 
> 3751, in resolve_any_xref
>      return [('c:' + self.role_for_objtype(objtype), retnode)]
> TypeError: can only concatenate str (not "NoneType") to str
> 
> The full traceback has been saved in /tmp/sphinx-err-v6iqwtf9.log, if 
> you want to report the issue to the developers.
> Please also report this if it was a user error, so that a better error 
> message can be provided next time.
> A bug report can be filed in the tracker at 
> <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
> 
> 
> It's not actually clear WHICH thing Sphinx is choking on here, exactly 
> -- it's just some bug in Sphinx's C domain handling of resolving 
> cross-references.
> 
> This is fixed in 3.3.1, I believe closed with one of these issues:
> 
> https://github.com/sphinx-doc/sphinx/issues/8364
> https://github.com/sphinx-doc/sphinx/issues/8388
> 
> 
> I'm not sure if there's anything I can do here -- we probably can't 
> require such a modern version of Sphinx just yet, I haven't done the 
> legwork to check. I also haven't been able to diagnose the exact 
> invocation that seems to trigger these failures, so it's not clear if 
> there's a workaround just yet.
> 
> I'm still interested in making our cross-referencing a bit more robust, 
> but maybe my hands are tied on that front for now.
> 
> --js

Update:

This is partly caused by a mere typo in exec/memory.h -- fixed here: 
https://gitlab.com/jsnow/qemu/-/pipelines/301171495

Kerneldoc generates an output that Sphinx *misinterprets* as a 
`cross-reference`. With that fixed, everything seems good for now.

The problem with v3.3.0 exists with or without changing the default 
role, and we will likely find that this version causes terse errors if 
it is used even without any of my changes:

Warning, treated as error:
kernel-doc 'perl /home/jsnow/src/qemu/docs/../scripts/kernel-doc -rst 
-enable-lineno -sphinx-version 3.3.0 
/home/jsnow/src/qemu/docs/../include/qemu/bitops.h' processing failed with:
ninja: build stopped: subcommand failed.
make: *** [Makefile:152: run-ninja] Error 1


We don't need to require >= 3.3.1, but possibly we want to add a check 
in conf.py that specifically disallows 3.3.0 with a friendlier error 
message.

That's all, then!

--js