[Qemu-devel] [PATCH v3 0/3] Add QMP lexer for annotated JSON to Sphinx

[Qemu-devel] [PATCH v3 0/3] Add QMP lexer for annotated JSON to Sphinx

[John Snow]

QMP isn't json. It turns out Pygments cares about this.
Teach Pygments to go with the annotated flow.

V3:
 - Also annotate elision indicators '...', which cause
   problems in Sphinx 2.1+
 - Adjust setup.py to cope with future versions of Sphinx
   that change the add_lexer interface

V2:
 - Change GPL2 to GPLv2+ (Peter Maydell)
 - Change "highlights" to "lexes" in docstrings
 - Fix typo ("Sphinx()" => "Sphinx") in docstring

Reported-by: Aarushi Mehta <mehta.aaru20@gmail.com>

John Snow (3):
  docs/interop/bitmaps.rst: Fix typos
  sphinx: add qmp_lexer
  docs/bitmaps: use QMP lexer instead of json

 docs/conf.py             |  4 +--
 docs/interop/bitmaps.rst | 58 ++++++++++++++++++++--------------------
 docs/sphinx/qmp_lexer.py | 43 +++++++++++++++++++++++++++++
 3 files changed, 74 insertions(+), 31 deletions(-)
 create mode 100644 docs/sphinx/qmp_lexer.py

-- 
2.20.1

[Qemu-devel] [PATCH v3 1/3] docs/interop/bitmaps.rst: Fix typos

[John Snow]

Pygments and Sphinx get pickier all the time; Sphinx 2.1+ now catches
these errors.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/interop/bitmaps.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/interop/bitmaps.rst b/docs/interop/bitmaps.rst
index 510e8809a9..c29ac4a854 100644
--- a/docs/interop/bitmaps.rst
+++ b/docs/interop/bitmaps.rst
@@ -399,7 +399,7 @@ in any one source bitmap, the target bitmap will mark that segment dirty.
        "arguments": {
          "node": "drive0",
          "target": "new_bitmap",
-         "bitmaps: [ "bitmap0" ]
+         "bitmaps": [ "bitmap0" ]
        }
      }
 
@@ -1437,7 +1437,7 @@ applied:
    .. code:: json
 
     <- {
-         "timestamp": {...}
+         "timestamp": {...},
          "data": {
            "device": "drive0",
            "type": "backup",
-- 
2.20.1

[Qemu-devel] [PATCH v3 2/3] sphinx: add qmp_lexer

[John Snow]

Sphinx, through Pygments, does not like annotated json examples very
much. In some versions of Sphinx (1.7), it will render the non-json
portions of code blocks in red, but in newer versions (2.0) it will
throw an exception and not highlight the block at all. Though we can
suppress this warning, it doesn't bring back highlighting on non-strict
json blocks.

We can alleviate this by creating a custom lexer for QMP examples that
allows us to properly highlight these examples in a robust way, keeping
our directionality and elision notations.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/conf.py             |  4 ++--
 docs/sphinx/qmp_lexer.py | 43 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+), 2 deletions(-)
 create mode 100644 docs/sphinx/qmp_lexer.py

diff --git a/docs/conf.py b/docs/conf.py
index befbcc6c3e..e46b299b71 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -41,7 +41,7 @@ except NameError:
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use an absolute path starting from qemu_docdir.
 #
-# sys.path.insert(0, os.path.join(qemu_docdir, "my_subdir"))
+sys.path.insert(0, os.path.join(qemu_docdir, "sphinx"))
 
 
 # -- General configuration ------------------------------------------------
@@ -54,7 +54,7 @@ needs_sphinx = '1.3'
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = []
+extensions = ['qmp_lexer']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py
new file mode 100644
index 0000000000..f7e4c0e198
--- /dev/null
+++ b/docs/sphinx/qmp_lexer.py
@@ -0,0 +1,43 @@
+# QEMU Monitor Protocol Lexer Extension
+#
+# Copyright (C) 2019, Red Hat Inc.
+#
+# Authors:
+#  Eduardo Habkost <ehabkost@redhat.com>
+#  John Snow <jsnow@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPLv2 or later.
+# See the COPYING file in the top-level directory.
+"""qmp_lexer is a Sphinx extension that provides a QMP lexer for code blocks."""
+
+from pygments.lexer import RegexLexer, DelegatingLexer
+from pygments.lexers.data import JsonLexer
+from pygments import token
+from sphinx import errors
+
+class QMPExampleMarkersLexer(RegexLexer):
+    """
+    QMPExampleMarkersLexer lexes QMP example annotations.
+    This lexer adds support for directionality flow and elision indicators.
+    """
+    tokens = {
+        'root': [
+            (r'-> ', token.Generic.Prompt),
+            (r'<- ', token.Generic.Prompt),
+            (r' ?\.{3} ?', token.Generic.Prompt),
+        ]
+    }
+
+class QMPExampleLexer(DelegatingLexer):
+    """QMPExampleLexer lexes annotated QMP examples."""
+    def __init__(self, **options):
+        super(QMPExampleLexer, self).__init__(JsonLexer, QMPExampleMarkersLexer,
+                                              token.Error, **options)
+
+def setup(sphinx):
+    """For use by the Sphinx extensions API."""
+    try:
+        sphinx.require_sphinx('2.1')
+        sphinx.add_lexer('QMP', QMPExampleLexer)
+    except errors.VersionRequirementError:
+        sphinx.add_lexer('QMP', QMPExampleLexer())
-- 
2.20.1

[Qemu-devel] [PATCH v3 3/3] docs/bitmaps: use QMP lexer instead of json

[John Snow]

The annotated style json we use in QMP documentation is not strict json
and depending on the version of Sphinx (2.0+) or Pygments installed,
might cause the build to fail.

Use the new QMP lexer.

Further, some versions of Sphinx can not apply custom lexers to "code"
directives and require the use of "code-block" directives instead, so
make that change at this time as well.

Tested under:
- Sphinx 1.3.6 and Pygments 2.4
- Sphinx 1.7.6 and Pygments 2.2 (Fedora 29 packages)
- Sphinx 2.0.1 and Pygments 2.4
- Sphinx 3.0.0+/f396b3a783 and Pygments 2.4 (From Sphinx git c4f44bdd)

Reported-by: Aarushi Mehta <mehta.aaru20@gmail.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
---
 docs/interop/bitmaps.rst | 54 ++++++++++++++++++++--------------------
 1 file changed, 27 insertions(+), 27 deletions(-)

diff --git a/docs/interop/bitmaps.rst b/docs/interop/bitmaps.rst
index c29ac4a854..c20bd37a79 100644
--- a/docs/interop/bitmaps.rst
+++ b/docs/interop/bitmaps.rst
@@ -199,7 +199,7 @@ persistence, and recording state can be adjusted at creation time.
 
  to create a new, actively recording persistent bitmap:
 
- .. code:: json
+ .. code-block:: QMP
 
   -> { "execute": "block-dirty-bitmap-add",
        "arguments": {
@@ -220,7 +220,7 @@ persistence, and recording state can be adjusted at creation time.
  To create a new, disabled (``-recording``), transient bitmap that tracks
  changes in 32KiB segments:
 
- .. code:: json
+ .. code-block:: QMP
 
   -> { "execute": "block-dirty-bitmap-add",
        "arguments": {
@@ -254,7 +254,7 @@ Deletes a bitmap. Bitmaps that are ``+busy`` cannot be removed.
 
  Remove a bitmap named ``bitmap0`` from node ``drive0``:
 
- .. code:: json
+ .. code-block:: QMP
 
   -> { "execute": "block-dirty-bitmap-remove",
        "arguments": {
@@ -280,7 +280,7 @@ Clears all dirty bits from a bitmap. ``+busy`` bitmaps cannot be cleared.
 
  Clear all dirty bits from bitmap ``bitmap0`` on node ``drive0``:
 
- .. code:: json
+ .. code-block:: QMP
 
   -> { "execute": "block-dirty-bitmap-clear",
        "arguments": {
@@ -309,7 +309,7 @@ begin being recorded. ``+busy`` bitmaps cannot be enabled.
 
  To set ``+recording`` on bitmap ``bitmap0`` on node ``drive0``:
 
- .. code:: json
+ .. code-block:: QMP
 
   -> { "execute": "block-dirty-bitmap-enable",
        "arguments": {
@@ -347,7 +347,7 @@ writes to begin being ignored. ``+busy`` bitmaps cannot be disabled.
 
  To set ``-recording`` on bitmap ``bitmap0`` on node ``drive0``:
 
- .. code:: json
+ .. code-block:: QMP
 
   -> { "execute": "block-dirty-bitmap-disable",
        "arguments": {
@@ -393,7 +393,7 @@ in any one source bitmap, the target bitmap will mark that segment dirty.
  ``drive0``. If ``new_bitmap`` was empty prior to this command, this achieves
  a copy.
 
- .. code:: json
+ .. code-block:: QMP
 
   -> { "execute": "block-dirty-bitmap-merge",
        "arguments": {
@@ -424,7 +424,7 @@ attached to nodes serving as the root for guest devices.
  API. This result highlights a bitmap ``bitmap0`` attached to the root node of
  device ``drive0``.
 
- .. code:: json
+ .. code-block:: QMP
 
   -> {
        "execute": "query-block",
@@ -562,7 +562,7 @@ new, empty bitmap that records writes from this point in time forward.
           destination. These writes will be recorded in the bitmap
           accordingly.
 
-.. code:: json
+.. code-block:: QMP
 
   -> {
        "execute": "transaction",
@@ -650,7 +650,7 @@ Example: Resetting an Incremental Backup Anchor Point
 If we want to start a new backup chain with an existing bitmap, we can also
 use a transaction to reset the bitmap while making a new full backup:
 
-.. code:: json
+.. code-block:: QMP
 
   -> {
        "execute": "transaction",
@@ -730,7 +730,7 @@ Example: First Incremental Backup
 
 #. Issue an incremental backup command:
 
-   .. code:: json
+   .. code-block:: QMP
 
     -> {
          "execute": "drive-backup",
@@ -788,7 +788,7 @@ Example: Second Incremental Backup
 #. Issue a new incremental backup command. The only difference here is that we
    have changed the target image below.
 
-   .. code:: json
+   .. code-block:: QMP
 
     -> {
          "execute": "drive-backup",
@@ -869,7 +869,7 @@ image:
 #. Issue a new incremental backup command. Apart from the new destination
    image, there is no difference from the last two examples.
 
-   .. code:: json
+   .. code-block:: QMP
 
     -> {
          "execute": "drive-backup",
@@ -932,7 +932,7 @@ point in time.
 
 #. Create a full (anchor) backup for each drive, with accompanying bitmaps:
 
-   .. code:: json
+   .. code-block:: QMP
 
     -> {
          "execute": "transaction",
@@ -1018,7 +1018,7 @@ point in time.
 
 #. Issue a multi-drive incremental push backup transaction:
 
-   .. code:: json
+   .. code-block:: QMP
 
     -> {
          "execute": "transaction",
@@ -1121,7 +1121,7 @@ described above. This example demonstrates the single-job failure case:
 
 #. Attempt to create an incremental backup via QMP:
 
-   .. code:: json
+   .. code-block:: QMP
 
     -> {
          "execute": "drive-backup",
@@ -1139,7 +1139,7 @@ described above. This example demonstrates the single-job failure case:
 
 #. Receive a pair of events indicating failure:
 
-   .. code:: json
+   .. code-block:: QMP
 
     <- {
          "timestamp": {...},
@@ -1175,7 +1175,7 @@ described above. This example demonstrates the single-job failure case:
 #. Retry the command after fixing the underlying problem, such as
    freeing up space on the backup volume:
 
-   .. code:: json
+   .. code-block:: QMP
 
     -> {
          "execute": "drive-backup",
@@ -1193,7 +1193,7 @@ described above. This example demonstrates the single-job failure case:
 
 #. Receive confirmation that the job completed successfully:
 
-   .. code:: json
+   .. code-block:: QMP
 
     <- {
          "timestamp": {...},
@@ -1233,7 +1233,7 @@ and one succeeds:
 
 #. Issue the transaction to start a backup of both drives.
 
-   .. code:: json
+   .. code-block:: QMP
 
     -> {
          "execute": "transaction",
@@ -1267,13 +1267,13 @@ and one succeeds:
 #. Receive notice that the Transaction was accepted, and jobs were
    launched:
 
-   .. code:: json
+   .. code-block:: QMP
 
     <- { "return": {} }
 
 #. Receive notice that the first job has completed:
 
-   .. code:: json
+   .. code-block:: QMP
 
     <- {
          "timestamp": {...},
@@ -1289,7 +1289,7 @@ and one succeeds:
 
 #. Receive notice that the second job has failed:
 
-   .. code:: json
+   .. code-block:: QMP
 
     <- {
          "timestamp": {...},
@@ -1365,7 +1365,7 @@ applied:
 
 #. Issue the multi-drive incremental backup transaction:
 
-   .. code:: json
+   .. code-block:: QMP
 
     -> {
          "execute": "transaction",
@@ -1401,13 +1401,13 @@ applied:
 
 #. Receive notice that the Transaction was accepted, and jobs were launched:
 
-   .. code:: json
+   .. code-block:: QMP
 
     <- { "return": {} }
 
 #. Receive notification that the backup job for ``drive1`` has failed:
 
-   .. code:: json
+   .. code-block:: QMP
 
     <- {
          "timestamp": {...},
@@ -1434,7 +1434,7 @@ applied:
 
 #. Receive notification that the job for ``drive0`` has been cancelled:
 
-   .. code:: json
+   .. code-block:: QMP
 
     <- {
          "timestamp": {...},
-- 
2.20.1

Re: [Qemu-devel] [PATCH v3 1/3] docs/interop/bitmaps.rst: Fix typos

[Vladimir Sementsov-Ogievskiy]

04.06.2019 0:46, John Snow wrote:
> Pygments and Sphinx get pickier all the time; Sphinx 2.1+ now catches
> these errors.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

> ---
>   docs/interop/bitmaps.rst | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/docs/interop/bitmaps.rst b/docs/interop/bitmaps.rst
> index 510e8809a9..c29ac4a854 100644
> --- a/docs/interop/bitmaps.rst
> +++ b/docs/interop/bitmaps.rst
> @@ -399,7 +399,7 @@ in any one source bitmap, the target bitmap will mark that segment dirty.
>          "arguments": {
>            "node": "drive0",
>            "target": "new_bitmap",
> -         "bitmaps: [ "bitmap0" ]
> +         "bitmaps": [ "bitmap0" ]
>          }
>        }
>   
> @@ -1437,7 +1437,7 @@ applied:
>      .. code:: json
>   
>       <- {
> -         "timestamp": {...}
> +         "timestamp": {...},
>            "data": {
>              "device": "drive0",
>              "type": "backup",
> 


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH v3 2/3] sphinx: add qmp_lexer

[Peter Maydell]

On Mon, 3 Jun 2019 at 22:47, John Snow <jsnow@redhat.com> wrote:
>
> Sphinx, through Pygments, does not like annotated json examples very
> much. In some versions of Sphinx (1.7), it will render the non-json
> portions of code blocks in red, but in newer versions (2.0) it will
> throw an exception and not highlight the block at all. Though we can
> suppress this warning, it doesn't bring back highlighting on non-strict
> json blocks.
>
> We can alleviate this by creating a custom lexer for QMP examples that
> allows us to properly highlight these examples in a robust way, keeping
> our directionality and elision notations.
>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> Signed-off-by: John Snow <jsnow@redhat.com>

Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
(mostly reviewing the sphinx config bits -- I trust that
the actual lexer does what it says on the tin :-))

thanks
-- PMM