[PATCH 0/3] docs: build an index page for the HTML docs

[PATCH 0/3] docs: build an index page for the HTML docs

[Stefan Hajnoczi]

This patch series switches to a single invocation of sphinx that builds
docs/index.rst.  This global index page links to all HTML documentation
produced by "make html", even the non-rST manuals.

This index page will be used as the entrypoint for documentation published on
the qemu.org website.

Stefan Hajnoczi (3):
  docs: fix rst syntax errors in unbuilt docs
  docs: build a global index page
  docs: install CSS and Javascript files for HTML docs

 Makefile                     | 14 +++++---------
 docs/arm-cpu-features.rst    |  6 +++---
 docs/index.rst               | 27 ++++++++++++++++++++++++++-
 docs/virtio-net-failover.rst |  4 ++--
 docs/virtio-pmem.rst         | 19 ++++++++++---------
 5 files changed, 46 insertions(+), 24 deletions(-)

-- 
2.23.0

[PATCH 1/3] docs: fix rst syntax errors in unbuilt docs

[Stefan Hajnoczi]

The .rst files outside docs/{devel,interop,specs} aren't built yet and
therefore a few syntax errors have slipped through.  Fix them.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 docs/arm-cpu-features.rst    |  6 +++---
 docs/virtio-net-failover.rst |  4 ++--
 docs/virtio-pmem.rst         | 19 ++++++++++---------
 3 files changed, 15 insertions(+), 14 deletions(-)

diff --git a/docs/arm-cpu-features.rst b/docs/arm-cpu-features.rst
index 1b367e22e1..9b537a75e6 100644
--- a/docs/arm-cpu-features.rst
+++ b/docs/arm-cpu-features.rst
@@ -41,9 +41,9 @@ CPU type is possible with the `query-cpu-model-expansion` QMP command.
 Below are some examples where `scripts/qmp/qmp-shell` (see the top comment
 block in the script for usage) is used to issue the QMP commands.
 
-(1) Determine which CPU features are available for the `max` CPU type
-    (Note, we started QEMU with qemu-system-aarch64, so `max` is
-     implementing the ARMv8-A reference manual in this case)::
+1. Determine which CPU features are available for the `max` CPU type
+   (Note, we started QEMU with qemu-system-aarch64, so `max` is
+   implementing the ARMv8-A reference manual in this case)::
 
       (QEMU) query-cpu-model-expansion type=full model={"name":"max"}
       { "return": {
diff --git a/docs/virtio-net-failover.rst b/docs/virtio-net-failover.rst
index 22f64c7bc8..6002dc5d96 100644
--- a/docs/virtio-net-failover.rst
+++ b/docs/virtio-net-failover.rst
@@ -1,6 +1,6 @@
-========================
+======================================
 QEMU virtio-net standby (net_failover)
-========================
+======================================
 
 This document explains the setup and usage of virtio-net standby feature which
 is used to create a net_failover pair of devices.
diff --git a/docs/virtio-pmem.rst b/docs/virtio-pmem.rst
index e77881b26f..4bf5d00443 100644
--- a/docs/virtio-pmem.rst
+++ b/docs/virtio-pmem.rst
@@ -27,17 +27,18 @@ virtio pmem usage
 -----------------
 
   A virtio pmem device backed by a memory-backend-file can be created on
-  the QEMU command line as in the following example:
+  the QEMU command line as in the following example::
 
-  -object memory-backend-file,id=mem1,share,mem-path=./virtio_pmem.img,size=4G
-  -device virtio-pmem-pci,memdev=mem1,id=nv1
+    -object memory-backend-file,id=mem1,share,mem-path=./virtio_pmem.img,size=4G
+    -device virtio-pmem-pci,memdev=mem1,id=nv1
 
-   where:
-   - "object memory-backend-file,id=mem1,share,mem-path=<image>, size=<image size>"
-     creates a backend file with the specified size.
+  where:
 
-   - "device virtio-pmem-pci,id=nvdimm1,memdev=mem1" creates a virtio pmem
-     pci device whose storage is provided by above memory backend device.
+  - "object memory-backend-file,id=mem1,share,mem-path=<image>, size=<image size>"
+    creates a backend file with the specified size.
+
+  - "device virtio-pmem-pci,id=nvdimm1,memdev=mem1" creates a virtio pmem
+    pci device whose storage is provided by above memory backend device.
 
   Multiple virtio pmem devices can be created if multiple pairs of "-object"
   and "-device" are provided.
@@ -50,7 +51,7 @@ memory backing has to be added via 'object_add'; afterwards, the virtio
 pmem device can be added via 'device_add'.
 
 For example, the following commands add another 4GB virtio pmem device to
-the guest:
+the guest::
 
  (qemu) object_add memory-backend-file,id=mem2,share=on,mem-path=virtio_pmem2.img,size=4G
  (qemu) device_add virtio-pmem-pci,id=virtio_pmem2,memdev=mem2
-- 
2.23.0

[PATCH 2/3] docs: build a global index page

[Stefan Hajnoczi]

Build docs/ in a single sphinx invocation instead of treating
docs/{devel,interop,specs} separately.  This allows us to build a global
index page that links to documentation across the different manuals.

Some documentation is built outside of sphinx and is not formatted as
reStructuredText.  Link directly to the .html files for the time being.
If they are converted to .rst files in the future they can be included
more elegantly.

Sphinx wants to build all .rst files and complains if they are not
listed in the table of contents.  We have not yet reviewed and
categorized some of our .rst files.  Hide these files so they are always
built (and syntax-checked from now on!) but not visible in the table of
contents.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 Makefile       | 13 ++++---------
 docs/index.rst | 27 ++++++++++++++++++++++++++-
 2 files changed, 30 insertions(+), 10 deletions(-)

diff --git a/Makefile b/Makefile
index aa9d1a42aa..9487a06bed 100644
--- a/Makefile
+++ b/Makefile
@@ -815,6 +815,7 @@ endef
 install-sphinxdocs: sphinxdocs
 	$(call install-manual,interop)
 	$(call install-manual,specs)
+	$(INSTALL_DATA) "$(MANUAL_BUILDDIR)/index.html" "$(DESTDIR)$(qemu_docdir)/index.html"
 
 install-doc: $(DOCS) install-sphinxdocs
 	$(INSTALL_DIR) "$(DESTDIR)$(qemu_docdir)"
@@ -1001,7 +1002,7 @@ docs/version.texi: $(SRC_PATH)/VERSION config-host.mak
 # and handles "don't rebuild things unless necessary" itself.
 # The '.doctrees' files are cached information to speed this up.
 .PHONY: sphinxdocs
-sphinxdocs: $(MANUAL_BUILDDIR)/devel/index.html $(MANUAL_BUILDDIR)/interop/index.html $(MANUAL_BUILDDIR)/specs/index.html
+sphinxdocs: $(MANUAL_BUILDDIR)/index.html
 
 # Canned command to build a single manual
 # Arguments: $1 = manual name, $2 = Sphinx builder ('html' or 'man')
@@ -1012,14 +1013,8 @@ build-manual = $(call quiet-command,CONFDIR="$(qemu_confdir)" sphinx-build $(if
 # We assume all RST files in the manual's directory are used in it
 manual-deps = $(wildcard $(SRC_PATH)/docs/$1/*.rst) $(SRC_PATH)/docs/$1/conf.py $(SRC_PATH)/docs/conf.py
 
-$(MANUAL_BUILDDIR)/devel/index.html: $(call manual-deps,devel)
-	$(call build-manual,devel,html)
-
-$(MANUAL_BUILDDIR)/interop/index.html: $(call manual-deps,interop)
-	$(call build-manual,interop,html)
-
-$(MANUAL_BUILDDIR)/specs/index.html: $(call manual-deps,specs)
-	$(call build-manual,specs,html)
+$(MANUAL_BUILDDIR)/index.html: $(call manual-deps,)
+	$(call build-manual,,html)
 
 $(MANUAL_BUILDDIR)/interop/qemu-ga.8: $(call manual-deps,interop)
 	$(call build-manual,interop,man)
diff --git a/docs/index.rst b/docs/index.rst
index baa5791c17..9d6d239561 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -6,11 +6,36 @@
 Welcome to QEMU's documentation!
 ================================
 
+.. Non-rst documentation
+
+`QEMU User Documentation <qemu-doc.html>`_
+
+`QEMU QMP Reference Manual <qemu-doc.html>`_
+
+`QEMU Guest Agent Protocol Reference <qemu-doc.html>`_
+
 .. toctree::
    :maxdepth: 2
    :caption: Contents:
 
    interop/index
-   devel/index
    specs/index
 
+.. The QEMU Developer's Guide is not included in the main documentation because
+   users don't need it.
+.. toctree::
+   :hidden:
+
+   devel/index
+
+.. Hidden documents that still need to be reviewed and moved to the appropriate
+   section of the documentation.
+.. toctree::
+   :hidden:
+
+   arm-cpu-features
+   cpu-hotplug
+   microvm
+   pr-manager
+   virtio-net-failover
+   virtio-pmem
-- 
2.23.0

[PATCH 3/3] docs: install CSS and Javascript files for HTML docs

[Stefan Hajnoczi]

Install the sphinx CSS/Javascript support files needed by the HTML
documentation.  The documentation looks ugly without this.

The previous patch switched to only invoking sphinx once so there is
only one _static/ directory that needs to be installed across all manual
sections.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index 9487a06bed..dd60787d4c 100644
--- a/Makefile
+++ b/Makefile
@@ -813,6 +813,7 @@ endef
 # for QEMU developers, and not interesting to our users.
 .PHONY: install-sphinxdocs
 install-sphinxdocs: sphinxdocs
+	$(call install-manual,_static)
 	$(call install-manual,interop)
 	$(call install-manual,specs)
 	$(INSTALL_DATA) "$(MANUAL_BUILDDIR)/index.html" "$(DESTDIR)$(qemu_docdir)/index.html"
-- 
2.23.0

Re: [PATCH 1/3] docs: fix rst syntax errors in unbuilt docs

[Daniel P. Berrangé]

On Fri, Nov 08, 2019 at 10:59:40AM +0100, Stefan Hajnoczi wrote:
> The .rst files outside docs/{devel,interop,specs} aren't built yet and
> therefore a few syntax errors have slipped through.  Fix them.
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  docs/arm-cpu-features.rst    |  6 +++---
>  docs/virtio-net-failover.rst |  4 ++--
>  docs/virtio-pmem.rst         | 19 ++++++++++---------
>  3 files changed, 15 insertions(+), 14 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/3] docs: build a global index page

[Daniel P. Berrangé]

On Fri, Nov 08, 2019 at 10:59:41AM +0100, Stefan Hajnoczi wrote:
> Build docs/ in a single sphinx invocation instead of treating
> docs/{devel,interop,specs} separately.  This allows us to build a global
> index page that links to documentation across the different manuals.
> 
> Some documentation is built outside of sphinx and is not formatted as
> reStructuredText.  Link directly to the .html files for the time being.
> If they are converted to .rst files in the future they can be included
> more elegantly.
> 
> Sphinx wants to build all .rst files and complains if they are not
> listed in the table of contents.  We have not yet reviewed and
> categorized some of our .rst files.  Hide these files so they are always
> built (and syntax-checked from now on!) but not visible in the table of
> contents.

Ah, nice trick.

> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  Makefile       | 13 ++++---------
>  docs/index.rst | 27 ++++++++++++++++++++++++++-
>  2 files changed, 30 insertions(+), 10 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 3/3] docs: install CSS and Javascript files for HTML docs

[Daniel P. Berrangé]

On Fri, Nov 08, 2019 at 10:59:42AM +0100, Stefan Hajnoczi wrote:
> Install the sphinx CSS/Javascript support files needed by the HTML
> documentation.  The documentation looks ugly without this.
> 
> The previous patch switched to only invoking sphinx once so there is
> only one _static/ directory that needs to be installed across all manual
> sections.
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  Makefile | 1 +
>  1 file changed, 1 insertion(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/3] docs: build a global index page

[Peter Maydell]

On Fri, 8 Nov 2019 at 09:59, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> Build docs/ in a single sphinx invocation instead of treating
> docs/{devel,interop,specs} separately.  This allows us to build a global
> index page that links to documentation across the different manuals.
>
> Some documentation is built outside of sphinx and is not formatted as
> reStructuredText.  Link directly to the .html files for the time being.
> If they are converted to .rst files in the future they can be included
> more elegantly.
>
> Sphinx wants to build all .rst files and complains if they are not
> listed in the table of contents.  We have not yet reviewed and
> categorized some of our .rst files.  Hide these files so they are always
> built (and syntax-checked from now on!) but not visible in the table of
> contents.

So the reason I went for the odd "run sphinx multiple times"
approach was because we don't want to ship 'devel' to users,
and as far as I could tell there was no way to have a
single-invocation build of the docs not include it in
eg the index/search (which would then be broken when
we don't install devel/ as part of 'make install').
Does this patchset result in a set of installed docs
that don't have dangling references ?

thanks
-- PMM

Re: [PATCH 3/3] docs: install CSS and Javascript files for HTML docs

[Peter Maydell]

On Fri, 8 Nov 2019 at 10:00, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> Install the sphinx CSS/Javascript support files needed by the HTML
> documentation.  The documentation looks ugly without this.
>
> The previous patch switched to only invoking sphinx once so there is
> only one _static/ directory that needs to be installed across all manual
> sections.
>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  Makefile | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/Makefile b/Makefile
> index 9487a06bed..dd60787d4c 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -813,6 +813,7 @@ endef
>  # for QEMU developers, and not interesting to our users.
>  .PHONY: install-sphinxdocs
>  install-sphinxdocs: sphinxdocs
> +       $(call install-manual,_static)
>         $(call install-manual,interop)
>         $(call install-manual,specs)
>         $(INSTALL_DATA) "$(MANUAL_BUILDDIR)/index.html" "$(DESTDIR)$(qemu_docdir)/index.html"

'install-manual' does some complicated stuff to
(a) handle subdirectories and (b) skip things we don't
want to install. It's intended for installing manual
directories (specs, interop, etc). _static is just
a simple single directory with no underlying files,
and it's not a manual, so it seems a bit odd to use
install-manual for it.

Also, this is only needed because we're now building
the docs in a single run (with the 'build manuals
one at a time' approach you get a separate specs/_static,
interop/_static, etc, which are installed by the
relevant install-manual calls for each manual). So
it seems like it ought to be squashed into the commit
that switches to doing the docs build in one run of
sphinx.

thanks
-- PMM

Re: [PATCH 2/3] docs: build a global index page

[Stefan Hajnoczi]

On Fri, Nov 8, 2019 at 11:52 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> On Fri, 8 Nov 2019 at 09:59, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > Build docs/ in a single sphinx invocation instead of treating
> > docs/{devel,interop,specs} separately.  This allows us to build a global
> > index page that links to documentation across the different manuals.
> >
> > Some documentation is built outside of sphinx and is not formatted as
> > reStructuredText.  Link directly to the .html files for the time being.
> > If they are converted to .rst files in the future they can be included
> > more elegantly.
> >
> > Sphinx wants to build all .rst files and complains if they are not
> > listed in the table of contents.  We have not yet reviewed and
> > categorized some of our .rst files.  Hide these files so they are always
> > built (and syntax-checked from now on!) but not visible in the table of
> > contents.
>
> So the reason I went for the odd "run sphinx multiple times"
> approach was because we don't want to ship 'devel' to users,
> and as far as I could tell there was no way to have a
> single-invocation build of the docs not include it in
> eg the index/search (which would then be broken when
> we don't install devel/ as part of 'make install').
> Does this patchset result in a set of installed docs
> that don't have dangling references ?

You are right:
 * The hidden documents are included in the navigation bar (different
from the table of contents).
 * The search index (which install-doc omits!) includes content from
the hidden documents.

I have asked on #sphinx-doc.  Let's see if there is a solution.

It might be possible to hack docs/config.py to exclude devel/ when run
from make install-sphinxdocs
(https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-exclude_patterns).
This requires building the docs again at install time but the
advantage is we get a single searchable set of documentation.

Stefan

Re: [PATCH 2/3] docs: build a global index page

[Peter Maydell]

On Fri, 8 Nov 2019 at 11:39, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Fri, Nov 8, 2019 at 11:52 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> > So the reason I went for the odd "run sphinx multiple times"
> > approach was because we don't want to ship 'devel' to users,
> > and as far as I could tell there was no way to have a
> > single-invocation build of the docs not include it in
> > eg the index/search (which would then be broken when
> > we don't install devel/ as part of 'make install').
> > Does this patchset result in a set of installed docs
> > that don't have dangling references ?
>
> You are right:
>  * The hidden documents are included in the navigation bar (different
> from the table of contents).
>  * The search index (which install-doc omits!) includes content from
> the hidden documents.
>
> I have asked on #sphinx-doc.  Let's see if there is a solution.

FWIW, this is the thread where I asked on the sphinx-users
list about the best way to handle "multiple manuals but we
only ship a subset":

https://www.mail-archive.com/sphinx-users@googlegroups.com/msg03224.html

> It might be possible to hack docs/config.py to exclude devel/ when run
> from make install-sphinxdocs
> (https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-exclude_patterns).
> This requires building the docs again at install time but the
> advantage is we get a single searchable set of documentation.

Yeah, if you're willing to build the docs twice (once for local
and once for installed) that also works. I'd prefer not to
do build work at 'make install' time, though -- if we want
to do this we should build them twice at 'make' time and
install just the right one.

thanks
-- PMM

Re: [PATCH 2/3] docs: build a global index page

[Peter Maydell]

On Fri, 8 Nov 2019 at 11:39, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> You are right:
>  * The hidden documents are included in the navigation bar (different
> from the table of contents).
>  * The search index (which install-doc omits!) includes content from
> the hidden documents.

What is install-doc failing to install? I just did a test
'make install' into a tempdir, and the search seems to work
in that set of installed docs.

thanks
-- PMM