[PATCH 0/6] more kernel-doc patches

[PATCH 0/6] more kernel-doc patches

[Daniel Vetter]

Hi all,

Finally gotten around to polish and flush these out. I think Markus'
kfigure/kimage/krender support is now simplified a lot (at least as much as I
managed with my python/sphinx noob-ness), and also rebase the doc patch about
the light markup approach we've discussed a while ago.

Follow-up patches are all for drm to show how awesome the inline DOT stuff is
and how much I want it. To get them in I'd need a topic branch.

I think the rst syntax itself is solid (I liked it a lot in the DRM patches), so
if more polish is needed for the script I'd like if we can get the current
version landed, to unblock the DRM doc work - I orignally wrote these for 4.10.

Cheers, Daniel

Daniel Vetter (5):
  doc: Explain light-handed markup preference a bit better
  drm/doc: Add KMS overview graphs
  drm/doc: Consistent kerneldoc include order
  drm/doc: diagram for mode objects and properties
  drm/doc: atomic overview, with graph

Markus Heiser (1):
  docs-rst: automatically convert Graphviz and SVG images

 Documentation/conf.py                 |   2 +-
 Documentation/doc-guide/hello.dot     |   3 +
 Documentation/doc-guide/sphinx.rst    | 107 +++++++-
 Documentation/doc-guide/svg_image.svg |  10 +
 Documentation/gpu/drm-internals.rst   |   6 +-
 Documentation/gpu/drm-kms-helpers.rst |  28 ++-
 Documentation/gpu/drm-kms.rst         | 263 +++++++++++++++++++-
 Documentation/gpu/drm-mm.rst          |  24 +-
 Documentation/process/changes.rst     |   7 +-
 Documentation/sphinx/kfigure.py       | 442 ++++++++++++++++++++++++++++++++++
 10 files changed, 846 insertions(+), 46 deletions(-)
 create mode 100644 Documentation/doc-guide/hello.dot
 create mode 100644 Documentation/doc-guide/svg_image.svg
 create mode 100644 Documentation/sphinx/kfigure.py

-- 
2.11.0

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 1/6] doc: Explain light-handed markup preference a bit better

[Daniel Vetter]

We already had a super-short blurb, but worth extending it I think:
We're still pretty far away from anything like a consensus, but
there's clearly a lot of people who prefer an as-light as possible
approach to converting existing .txt files to .rst. Make sure this is
properly taken into account and clear.

Motivated by discussions with Peter and Christoph and others.

v2:
- Mention that existing headings should be kept when converting
  existing .txt files (Mauro).
- Explain that we prefer :: for quoting code, it's easier on the
  eyes (Mauro).
- Explain that blindly converting outdated docs is harmful. Motived
  by comments Peter did in our discussion.

v3: Make the explanations around fixed-width quoting more concise
(Jani).

v4:
- Rebase onto docs-4.10.
- Go with the more terse recommendation from Jani, defer to the much
  more detailed conversion guide Mauro is working on for details.

Cc: Jonathan Corbet <corbet@lwn.net>
Cc: linux-doc@vger.kernel.org
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Jani Nikula <jani.nikula@linux.intel.com>
Cc: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
---
 Documentation/doc-guide/sphinx.rst | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst
index 96fe7ccb2c67..532d65b70500 100644
--- a/Documentation/doc-guide/sphinx.rst
+++ b/Documentation/doc-guide/sphinx.rst
@@ -73,7 +73,16 @@ Specific guidelines for the kernel documentation
 
 Here are some specific guidelines for the kernel documentation:
 
-* Please don't go overboard with reStructuredText markup. Keep it simple.
+* Please don't go overboard with reStructuredText markup. Keep it
+  simple. For the most part the documentation should be plain text with
+  just enough consistency in formatting that it can be converted to
+  other formats.
+
+* Please keep the formatting changes minimal when converting existing
+  documentation to reStructuredText.
+
+* Also update the content, not just the formatting, when converting
+  documentation.
 
 * Please stick to this order of heading adornments:
 
@@ -103,6 +112,12 @@ Here are some specific guidelines for the kernel documentation:
   the order as encountered."), having the higher levels the same overall makes
   it easier to follow the documents.
 
+* For inserting fixed width text blocks (for code examples, use case
+  examples, etc.), use ``::`` for anything that doesn't really benefit
+  from syntax highlighting, especially short snippets. Use
+  ``.. code-block:: <language>`` for longer code blocks that benefit
+  from highlighting.
+
 
 the C domain
 ------------
-- 
2.11.0

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Daniel Vetter]

From: Markus Heiser <markus.heiser@darmarit.de>

This patch brings scalable figure, image handling and a concept to
embed *render* markups:

* DOT (http://www.graphviz.org)
* SVG

For image handling use the 'image' replacement::

    .. kernel-image::  svg_image.svg
       :alt:    simple SVG image

For figure handling use the 'figure' replacement::

    .. kernel-figure::  svg_image.svg
       :alt:    simple SVG image

       SVG image example

Embed *render* markups (or languages) like Graphviz's **DOT** is
provided by the *render* directive.::

  .. kernel-render:: DOT
     :alt: foobar digraph
     :caption: Embedded **DOT** (Graphviz) code.

     digraph foo {
      "bar" -> "baz";
     }

The *render* directive is a concept to integrate *render* markups and
languages, yet supported markups:

* DOT: render embedded Graphviz's **DOT**
* SVG: render embedded Scalable Vector Graphics (**SVG**)

v2: s/DOC/DOT/ in a few places (by Daniel).

v3: Simplify stuff a bit (by Daniel):

- Remove path detection and setup/check code for that. In
  Documentation/media/Makefile we already simply use these tools,
  better to have one consolidated check if we want/need one. Also
  remove the convertsvg support, we require ImageMagick's convert
  already in the doc build, no need for a 2nd fallback.

- Use sphinx for depency tracking, remove hand-rolled version.

- Forward stderr from dot and convert, otherwise debugging issues with
  the diagrams is impossible.

Cc: Jonathan Corbet <corbet@lwn.net>
Cc: linux-doc@vger.kernel.org
Cc: Jani Nikula <jani.nikula@linux.intel.com>
Cc: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de> (v1)
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
---
 Documentation/conf.py                 |   2 +-
 Documentation/doc-guide/hello.dot     |   3 +
 Documentation/doc-guide/sphinx.rst    |  90 ++++++-
 Documentation/doc-guide/svg_image.svg |  10 +
 Documentation/process/changes.rst     |   7 +-
 Documentation/sphinx/kfigure.py       | 442 ++++++++++++++++++++++++++++++++++
 6 files changed, 548 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/doc-guide/hello.dot
 create mode 100644 Documentation/doc-guide/svg_image.svg
 create mode 100644 Documentation/sphinx/kfigure.py

diff --git a/Documentation/conf.py b/Documentation/conf.py
index f6823cf01275..e3f537ce2935 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -34,7 +34,7 @@ from load_config import loadConfig
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain']
+extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure']
 
 # The name of the math extension changed on Sphinx 1.4
 if major == 1 and minor > 3:
diff --git a/Documentation/doc-guide/hello.dot b/Documentation/doc-guide/hello.dot
new file mode 100644
index 000000000000..504621dfc595
--- /dev/null
+++ b/Documentation/doc-guide/hello.dot
@@ -0,0 +1,3 @@
+graph G {
+      Hello -- World
+}
diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst
index 532d65b70500..b902744ce7dd 100644
--- a/Documentation/doc-guide/sphinx.rst
+++ b/Documentation/doc-guide/sphinx.rst
@@ -34,8 +34,10 @@ format-specific subdirectories under ``Documentation/output``.
 
 To generate documentation, Sphinx (``sphinx-build``) must obviously be
 installed. For prettier HTML output, the Read the Docs Sphinx theme
-(``sphinx_rtd_theme``) is used if available. For PDF output, ``rst2pdf`` is also
-needed. All of these are widely available and packaged in distributions.
+(``sphinx_rtd_theme``) is used if available. For PDF output you'll also need
+``XeLaTeX`` and CairoSVG (http://cairosvg.org) or alternatively ``convert(1)``
+from ImageMagick (https://www.imagemagick.org). All of these are widely
+available and packaged in distributions.
 
 To pass extra options to Sphinx, you can use the ``SPHINXOPTS`` make
 variable. For example, use ``make SPHINXOPTS=-v htmldocs`` to get more verbose
@@ -232,3 +234,87 @@ Rendered as:
       * .. _`last row`:
 
         - column 3
+
+
+Figures & Images
+================
+
+If you want to add an image, you should use the ``kernel-figure`` and
+``kernel-image`` directives. E.g. to insert a figure with a scalable
+image format use SVG::
+
+    .. kernel-figure::  svg_image.svg
+       :alt:    simple SVG image
+
+       SVG image example
+
+.. kernel-figure::  svg_image.svg
+   :alt:    simple SVG image
+
+   SVG image example
+
+The kernel figure (and image) directive support **DOT** formated files, see
+
+* DOT: http://graphviz.org/pdf/dotguide.pdf
+* Graphviz: http://www.graphviz.org/content/dot-language
+
+A simple example::
+
+  .. kernel-figure::  hello.dot
+     :alt:    hello world
+
+     DOT's hello world example
+
+.. kernel-figure::  hello.dot
+   :alt:    hello world
+
+   DOT's hello world example
+
+Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the
+``kernel-render`` directives.::
+
+  .. kernel-render:: DOT
+     :alt: foobar digraph
+     :caption: Embedded **DOT** (Graphviz) code.
+
+     digraph foo {
+      "bar" -> "baz";
+     }
+
+How this will be rendered depends on the installed tools. If Graphviz is
+installed, you will see an vector image. If not the raw markup is inserted as
+*literal-block*.
+
+.. kernel-render:: DOT
+   :alt: foobar digraph
+   :caption: Embedded **DOT** (Graphviz) code.
+
+   digraph foo {
+      "bar" -> "baz";
+   }
+
+The *render* directive has all the options known from the *figure* directive,
+plus option ``caption``.  If ``caption`` has a value, a *figure* node is
+inserted. If not, a *image* node is inserted.
+
+Embedded **SVG**::
+
+  .. kernel-render:: SVG
+     :caption: Embedded **SVG** markup.
+     :alt: so-nw-arrow
+
+     <?xml version="1.0" encoding="UTF-8"?>
+     <svg xmlns="http://www.w3.org/2000/svg" version="1.1" ...>
+        ...
+     </svg>
+
+.. kernel-render:: SVG
+   :caption: Embedded **SVG** markup.
+   :alt: so-nw-arrow
+
+   <?xml version="1.0" encoding="UTF-8"?>
+   <svg xmlns="http://www.w3.org/2000/svg"
+     version="1.1" baseProfile="full" width="70px" height="40px" viewBox="0 0 700 400">
+   <line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/>
+   <polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/>
+   </svg>
diff --git a/Documentation/doc-guide/svg_image.svg b/Documentation/doc-guide/svg_image.svg
new file mode 100644
index 000000000000..5405f85b8137
--- /dev/null
+++ b/Documentation/doc-guide/svg_image.svg
@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- originate: https://commons.wikimedia.org/wiki/File:Variable_Resistor.svg -->
+<svg xmlns="http://www.w3.org/2000/svg"
+	version="1.1" baseProfile="full"
+	width="70px" height="40px" viewBox="0 0 700 400">
+	<line x1="0" y1="200" x2="700" y2="200" stroke="black" stroke-width="20px"/>
+	<rect x="100" y="100" width="500" height="200" fill="white" stroke="black" stroke-width="20px"/>
+	<line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/>
+	<polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/>
+</svg>
diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 56ce66114665..e4f25038ef65 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -318,9 +318,10 @@ PDF outputs, it is recommended to use version 1.4.6.
 .. note::
 
   Please notice that, for PDF and LaTeX output, you'll also need ``XeLaTeX``
-  version 3.14159265. Depending on the distribution, you may also need
-  to install a series of ``texlive`` packages that provide the minimal
-  set of functionalities required for ``XeLaTex`` to work.
+  version 3.14159265. Depending on the distribution, you may also need to
+  install a series of ``texlive`` packages that provide the minimal set of
+  functionalities required for ``XeLaTex`` to work. For PDF output you'll also
+  need ``convert(1)`` from ImageMagick (https://www.imagemagick.org).
 
 Other tools
 -----------
diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py
new file mode 100644
index 000000000000..8f749d389dae
--- /dev/null
+++ b/Documentation/sphinx/kfigure.py
@@ -0,0 +1,442 @@
+# -*- coding: utf-8; mode: python -*-
+# pylint: disable=C0103
+u"""
+    scalable figure and image handling
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    Sphinx extension which implements scalable image handling.
+
+    :copyright:  Copyright (C) 2016  Markus Heiser
+    :license:    GPL Version 2, June 1991 see Linux/COPYING for details.
+
+    The build for image formats depence on image's source format and output's
+    destination format. This extension implement methods to simplify image
+    handling from the author's POV. Directives like ``kernel-figure`` implement
+    methods *to* always get the best output-format even if some tools are not
+    installed.For more details take a look at ``convert_image(...)`` which is
+    the core of all conversions.
+
+    * ``.. kernel-image``: for image handling / ``.. image::`` replacement
+
+    * ``.. kernel-figure``: for figure handling / ``.. figure::`` replacement
+
+    * ``.. kernel-render``: for render markup / a concept to embed *render*
+      markups (or languages). Supported markups (see ``RENDER_MARKUP_EXT``)
+
+      + ``DOT``: render embedded Graphviz's **DOC**
+      + ``SVG``: render embedded Scalable Vector Graphics (**SVG**)
+      + ... *developable*
+
+    Used tools:
+
+    * ``dot(1)``: Graphviz (http://www.graphviz.org). If Graphviz is not
+      available, the DOT language is inserted as literal-block.
+
+    * SVG to PDF: To generate PDF, you need at least one of this tools:
+
+      - ``convert(1)``: ImageMagick (https://www.imagemagick.org)
+
+    List of customizations:
+
+    * generate PDF from SVG / used by PDF (LaTeX) builder
+
+    * generate SVG (html-builder) and PDF (latex-builder) from DOT files.
+      DOT: see http://www.graphviz.org/content/dot-language
+
+    """
+
+import os
+from os import path
+import subprocess
+from hashlib import sha1
+import sys
+
+from docutils import nodes
+from docutils.statemachine import ViewList
+from docutils.parsers.rst import directives
+from docutils.parsers.rst.directives import images
+
+from sphinx.directives import patches
+
+__version__  = '1.0'
+
+# simple helper
+# -------------
+
+def mkdir(folder, mode=0o775):
+    if not path.isdir(folder):
+        os.makedirs(folder, mode)
+
+# def debug_handle(self, node):           # pylint: disable=W0613
+#     from linuxdoc.kernel_doc import CONSOLE
+#     CONSOLE()
+
+def pass_handle(self, node):           # pylint: disable=W0613
+    pass
+
+# setup conversion tools and sphinx extension
+# -------------------------------------------
+
+def setup(app):
+    # image handling
+    app.add_directive("kernel-image",  KernelImage)
+    app.add_node(kernel_image,
+                 html    = (visit_kernel_image, pass_handle),
+                 latex   = (visit_kernel_image, pass_handle),
+                 texinfo = (visit_kernel_image, pass_handle),
+                 text    = (visit_kernel_image, pass_handle),
+                 man     = (visit_kernel_image, pass_handle), )
+
+    # figure handling
+    app.add_directive("kernel-figure", KernelFigure)
+    app.add_node(kernel_figure,
+                 html    = (visit_kernel_figure, pass_handle),
+                 latex   = (visit_kernel_figure, pass_handle),
+                 texinfo = (visit_kernel_figure, pass_handle),
+                 text    = (visit_kernel_figure, pass_handle),
+                 man     = (visit_kernel_figure, pass_handle), )
+
+    # render handling
+    app.add_directive('kernel-render', KernelRender)
+    app.add_node(kernel_render,
+                 html    = (visit_kernel_render, pass_handle),
+                 latex   = (visit_kernel_render, pass_handle),
+                 texinfo = (visit_kernel_render, pass_handle),
+                 text    = (visit_kernel_render, pass_handle),
+                 man     = (visit_kernel_render, pass_handle), )
+
+    return dict(
+        version = __version__,
+        parallel_read_safe = True,
+        parallel_write_safe = True
+    )
+
+# integrate conversion tools
+# --------------------------
+
+RENDER_MARKUP_EXT = {
+    # The '.ext' must be handled by convert_image(..) function's *in_ext* input.
+    # <name> : <.ext>
+    'DOT'   : '.dot'
+    , 'SVG' : '.svg'
+}
+
+def convert_image(img_node, translator):  # pylint: disable=R0912
+    """Convert an image node for the builder.
+
+    Different builder prefer different image formats, e.g. *latex* builder
+    prefer PDF while *html* builder prefer SVG format for images.
+
+    This function handles outputs image formats in depence of source the format
+    of the image and the translator's output format. This also means to
+    manipulate/update the *image* dictionary of the builder (``builder.images``)
+
+    """
+    fname, in_ext = path.splitext(path.basename(img_node['uri']))
+    src_fname     = path.join(translator.builder.srcdir, img_node['uri'])
+    src_folder    = path.dirname(img_node['uri'])
+    out_dir       = translator.builder.outdir
+    dst_fname     = None
+
+    # in kernel builds, use 'make SPHINXOPTS=-v' to see verbose messages
+    verbose = translator.builder.app.verbose
+    warn    = translator.builder.warn
+
+    verbose('assert best format for: ' + img_node['uri'])
+
+    if in_ext == '.dot':
+
+        # ----------
+        # handle DOT
+        # ----------
+
+        dst_fname = path.join(out_dir, fname + '.pdf')
+
+        if translator.builder.format == 'html':
+            dst_fname = path.join(out_dir, src_folder, fname + '.svg')
+        else:
+            # all other builder formats will include DOT as raw
+            with open(src_fname, "r") as dot:
+                data = dot.read()
+                node = nodes.literal_block(data, data)
+                img_node.replace_self(node)
+
+
+    elif in_ext == '.svg':
+
+        # ----------
+        # handle SVG
+        # ----------
+
+        if translator.builder.format == 'latex':
+            dst_fname = path.join(out_dir, fname + '.pdf')
+
+    if dst_fname:
+        name = dst_fname[len(out_dir) + 1:]
+        # the builder needs not to copy one more time, so pop it if exists.
+        translator.builder.images.pop(img_node['uri'], None)
+        img_node['uri'] = dst_fname
+        img_node['candidates'] = {'*': dst_fname}
+
+        mkdir(path.dirname(dst_fname))
+
+        if in_ext == '.dot':
+            verbose('convert DOT to: {out}/' + name)
+            dot2format(src_fname, dst_fname)
+
+        elif in_ext == '.svg':
+            verbose('convert SVG to: {out}/' + name)
+            svg2pdf(src_fname, dst_fname)
+
+def dot2format(dot_fname, out_fname):
+    """Converts DOT file to ``out_fname`` using ``dot(1)``.
+
+    * ``dot_fname`` pathname of the input DOT file, including extension ``.dot``
+    * ``out_fname`` pathname of the output file, including format extension
+
+    The *format extension* depends on the ``dot`` command (see ``man dot``
+    option ``-Txxx``). Normally you will use one of the following extensions:
+
+    - ``.ps`` for PostScript,
+    - ``.svg`` or ``svgz`` for Structured Vector Graphics,
+    - ``.fig`` for XFIG graphics and
+    - ``.png`` or ``gif`` for common bitmap graphics.
+
+    """
+    out_format = path.splitext(out_fname)[1][1:]
+    cmd = ['dot', '-T%s' % out_format, dot_fname]
+    exit_code = 42
+    with open(out_fname, "w") as out:
+        p = subprocess.Popen(
+            cmd, stdout = out, stderr = subprocess.PIPE )
+        nil, err = p.communicate()
+
+        sys.stderr.write(err)
+
+        exit_code = p.returncode
+        out.flush()
+    return bool(exit_code == 0)
+
+def svg2pdf(svg_fname, pdf_fname):
+    """Converts SVG to PDF with CairoSVG or ``convert(1)`` command.
+
+    Uses ``convert(1)`` from ImageMagick (https://www.imagemagick.org) for
+    conversion.  Returns ``True`` on success and ``False`` if an error occurred
+    (e.g. none of the conversion tool is available).
+
+    * ``svg_fname`` pathname of the input SVG file with extension (``.svg``)
+    * ``pdf_name``  pathname of the output PDF file with extension (``.pdf``)
+
+    """
+    cmd = [convert_cmd, svg_fname, pdf_fname]
+    p = subprocess.Popen(
+        cmd, stdout = out, stderr = subprocess.PIPE )
+    nil, err = p.communicate()
+
+    sys.stderr.write(err)
+
+    exit_code = p.returncode
+    return bool(exit_code == 0)
+
+
+# image handling
+# ---------------------
+
+def visit_kernel_image(self, node):    # pylint: disable=W0613
+    """Visitor of the ``kernel_image`` Node.
+
+    Handles the ``image`` child-node with the ``convert_image(...)``.
+    """
+    img_node = node[0]
+    convert_image(img_node, self)
+
+class kernel_image(nodes.General, nodes.Element):
+    """Node for ``kernel-image`` directive."""
+    pass
+
+class KernelImage(images.Image):
+    u"""KernelImage directive
+
+    Earns everything from ``.. image::`` directive, except *remote URI* and
+    *glob* pattern. The KernelImage wraps a image node into a
+    kernel_image node. See ``visit_kernel_image``.
+    """
+
+    def run(self):
+        env = self.state.document.settings.env
+
+        uri = self.arguments[0]
+        if uri.endswith('.*') or uri.find('://') != -1:
+            raise self.severe(
+                'Error in "%s: %s": glob pattern and remote images are not allowed'
+                % (self.name, uri))
+
+        # Tell sphinx of the dependency
+        env.note_dependency(os.path.abspath(uri))
+
+        result = images.Image.run(self)
+        if len(result) == 2 or isinstance(result[0], nodes.system_message):
+            return result
+        (image_node,) = result
+        # wrap image node into a kernel_image node / see visitors
+        node = kernel_image('', image_node)
+        return [node]
+
+# figure handling
+# ---------------------
+
+def visit_kernel_figure(self, node):   # pylint: disable=W0613
+    """Visitor of the ``kernel_figure`` Node.
+
+    Handles the ``image`` child-node with the ``convert_image(...)``.
+    """
+    img_node = node[0][0]
+    convert_image(img_node, self)
+
+class kernel_figure(nodes.General, nodes.Element):
+    """Node for ``kernel-figure`` directive."""
+
+class KernelFigure(patches.Figure):
+    u"""KernelImage directive
+
+    Earns everything from ``.. figure::`` directive, except *remote URI* and
+    *glob* pattern.  The KernelFigure wraps a figure node into a kernel_figure
+    node. See ``visit_kernel_figure``.
+    """
+
+    def run(self):
+        env = self.state.document.settings.env
+
+        uri = self.arguments[0]
+        if uri.endswith('.*') or uri.find('://') != -1:
+            raise self.severe(
+                'Error in "%s: %s":'
+                ' glob pattern and remote images are not allowed'
+                % (self.name, uri))
+
+        # Tell sphinx of the dependency
+        env.note_dependency(os.path.abspath(uri))
+
+        result = patches.Figure.run(self)
+        if len(result) == 2 or isinstance(result[0], nodes.system_message):
+            return result
+        (figure_node,) = result
+        # wrap figure node into a kernel_figure node / see visitors
+        node = kernel_figure('', figure_node)
+        return [node]
+
+
+# render handling
+# ---------------------
+
+def visit_kernel_render(self, node):
+    """Visitor of the ``kernel_render`` Node.
+
+    If rendering tools available, save the markup of the ``literal_block`` child
+    node into a file and replace the ``literal_block`` node with a new created
+    ``image`` node, pointing to the saved markup file. Afterwards, handle the
+    image child-node with the ``convert_image(...)``.
+    """
+
+    verbose = self.builder.app.verbose
+    warn    = self.builder.warn
+    srclang = node.get('srclang')
+
+    verbose('visit kernel-render node lang: "%s"' % (srclang))
+
+    tmp_ext = RENDER_MARKUP_EXT.get(srclang, None)
+    if tmp_ext is None:
+        warn('kernel-render: "%s" unknow / include raw.' % (srclang))
+        return
+
+    literal_block = node[0]
+    code = literal_block.astext()
+
+    if tmp_ext:
+        hashobj   = code.encode('utf-8') #  str(node.attributes)
+        fname     = '%s-%s' % (srclang, sha1(hashobj).hexdigest())
+        tmp_fname = path.join(
+            self.builder.outdir, self.builder.imagedir, fname + tmp_ext)
+
+        if not path.isfile(tmp_fname):
+            mkdir(path.dirname(tmp_fname))
+            with open(tmp_fname, "w") as out:
+                out.write(code)
+
+        image_node = nodes.image(node.rawsource, **node.attributes)
+        image_node['uri'] = tmp_fname
+
+        literal_block.replace_self(image_node)
+        convert_image(image_node, self)
+
+
+class kernel_render(nodes.General, nodes.Inline, nodes.Element):
+    """Node for ``kernel-render`` directive."""
+    pass
+
+class KernelRender(patches.Figure):
+    u"""KernelRender directive
+
+    Render content by external tool.  Has all the options known from the
+    *figure*  directive, plus option ``caption``.  If ``caption`` has a
+    value, a figure node with the *caption* is inserted. If not, a image node is
+    inserted.
+
+    The KernelRender directive wraps the text of the directive into a
+    literal_block node and wraps it into a kernel_render node. See
+    ``visit_kernel_render``.
+    """
+    has_content = True
+    required_arguments = 1
+    optional_arguments = 0
+    final_argument_whitespace = False
+
+    # earn options from 'figure'
+    option_spec = patches.Figure.option_spec.copy()
+    option_spec['caption'] = directives.unchanged
+
+    def run(self):
+        return [self.build_node()]
+
+    def build_node(self):
+
+        srclang = self.arguments[0].strip()
+        if srclang not in RENDER_MARKUP_EXT.keys():
+            return [self.state_machine.reporter.warning(
+                'Unknow source language "%s", use one of: %s.' % (
+                    srclang, ",".join(RENDER_MARKUP_EXT.keys())),
+                line=self.lineno)]
+
+        code = '\n'.join(self.content)
+        if not code.strip():
+            return [self.state_machine.reporter.warning(
+                'Ignoring "%s" directive without content.' % (
+                    self.name),
+                line=self.lineno)]
+
+        node = kernel_render()
+        node['alt'] = self.options.get('alt','')
+        node['srclang'] = srclang
+        literal_node = nodes.literal_block(code, code)
+        node += literal_node
+
+        caption = self.options.get('caption')
+        if caption:
+            # parse cation's content
+            parsed = nodes.Element()
+            self.state.nested_parse(
+                ViewList([caption], source=''), self.content_offset, parsed)
+            caption_node = nodes.caption(
+                parsed[0].rawsource, '', *parsed[0].children)
+            caption_node.source = parsed[0].source
+            caption_node.line = parsed[0].line
+
+            figure_node = nodes.figure('', node)
+            for k,v in self.options.items():
+                figure_node[k] = v
+            figure_node += caption_node
+
+            node = figure_node
+
+        return node
+
-- 
2.11.0

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Gabriel Krisman Bertazi]

Daniel Vetter <daniel.vetter@ffwll.ch> writes:

Hi Daniel,

> +    if dst_fname:
> +        name = dst_fname[len(out_dir) + 1:]
> +        # the builder needs not to copy one more time, so pop it if exists.
> +        translator.builder.images.pop(img_node['uri'], None)
> +        img_node['uri'] = dst_fname
> +        img_node['candidates'] = {'*': dst_fname}
> +
> +        mkdir(path.dirname(dst_fname))
> +
> +        if in_ext == '.dot':
> +            verbose('convert DOT to: {out}/' + name)
> +            dot2format(src_fname, dst_fname)
> +
> +        elif in_ext == '.svg':
> +            verbose('convert SVG to: {out}/' + name)
> +            svg2pdf(src_fname, dst_fname)

Small nit, but, shouldn't you add dst_fname to img_node only if
dot2format or svg2pdf was successful?

Thanks,

-- 
Gabriel Krisman Bertazi
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Daniel Vetter]

On Wed, Mar 01, 2017 at 12:56:36PM -0300, Gabriel Krisman Bertazi wrote:
> Daniel Vetter <daniel.vetter@ffwll.ch> writes:
> 
> Hi Daniel,
> 
> > +    if dst_fname:
> > +        name = dst_fname[len(out_dir) + 1:]
> > +        # the builder needs not to copy one more time, so pop it if exists.
> > +        translator.builder.images.pop(img_node['uri'], None)
> > +        img_node['uri'] = dst_fname
> > +        img_node['candidates'] = {'*': dst_fname}
> > +
> > +        mkdir(path.dirname(dst_fname))
> > +
> > +        if in_ext == '.dot':
> > +            verbose('convert DOT to: {out}/' + name)
> > +            dot2format(src_fname, dst_fname)
> > +
> > +        elif in_ext == '.svg':
> > +            verbose('convert SVG to: {out}/' + name)
> > +            svg2pdf(src_fname, dst_fname)
> 
> Small nit, but, shouldn't you add dst_fname to img_node only if
> dot2format or svg2pdf was successful?

I kinda assumed they're always successful, since the media documentation
build already requires them.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Markus Heiser]

Am 02.03.2017 um 08:12 schrieb Daniel Vetter <daniel@ffwll.ch>:

> On Wed, Mar 01, 2017 at 12:56:36PM -0300, Gabriel Krisman Bertazi wrote:
>> Daniel Vetter <daniel.vetter@ffwll.ch> writes:
>> 
>> Hi Daniel,
>> 
>>> +    if dst_fname:
>>> +        name = dst_fname[len(out_dir) + 1:]
>>> +        # the builder needs not to copy one more time, so pop it if exists.
>>> +        translator.builder.images.pop(img_node['uri'], None)
>>> +        img_node['uri'] = dst_fname
>>> +        img_node['candidates'] = {'*': dst_fname}
>>> +
>>> +        mkdir(path.dirname(dst_fname))
>>> +
>>> +        if in_ext == '.dot':
>>> +            verbose('convert DOT to: {out}/' + name)
>>> +            dot2format(src_fname, dst_fname)
>>> +
>>> +        elif in_ext == '.svg':
>>> +            verbose('convert SVG to: {out}/' + name)
>>> +            svg2pdf(src_fname, dst_fname)
>> 
>> Small nit, but, shouldn't you add dst_fname to img_node only if
>> dot2format or svg2pdf was successful?
> 
> I kinda assumed they're always successful, since the media documentation
> build already requires them.

SHORT:

At this time (Phase 1: Reading), the internal doctree is build and
with these two lines:

>>> +        img_node['uri'] = dst_fname
>>> +        img_node['candidates'] = {'*': dst_fname}


the image is added as wildcard "*" to the doctree. 

Later (Phase 4: Writing), when the output is generated from
doctree, the (pdf-/html-) builder will search for the best
existing "*" candidate, e.g. "*.svg" for html and "*.pdf"
for LaTeX.

MORE VERBOSE:

To see/understand the whole play we have to know that Sphinx/docutils
has Build Phases: 

* http://www.sphinx-doc.org/en/stable/extdev/tutorial.html#build-phases

"""
Phase 1: Reading
In Phase 1, all source files (and on subsequent builds, those that are new or changed) are read and parsed. This is the phase where directives and roles are encountered by docutils, and the corresponding code is executed. The output of this phase is a doctree for each source file"""
...

Phase 4: Writing
This phase converts the resolved doctrees to the desired output format, such as HTML or LaTeX. This happens via a so-called docutils writer that visits the individual nodes of each doctree and produces some output in the process.
"""

The visitor for handling kernel_image, kernel_render and kernel_figure
directives are analog, so lets look at the simple one::

+def visit_kernel_image(self, node):    # pylint: disable=W0613
+    """Visitor of the ``kernel_image`` Node.
+
+    Handles the ``image`` child-node with the ``convert_image(...)``.
+    """
+    img_node = node[0]
+    convert_image(img_node, self)

this is the visitor, for building the internal doctree. The visitor
calls convert_image() with img_node as first argument.

The convert_image() first evals a new dst_fname for image
types which are covered by:

+    dst_fname     = None
...
+    if in_ext == '.dot':
+        dst_fname = path.join(out_dir, fname + '.pdf')
....
+
+    elif in_ext == '.svg':
+        if translator.builder.format == 'latex':
+            dst_fname = path.join(out_dir, fname + '.pdf')

Consider that dst_fname is always in the out_dir. So, if this image type
is covered by convert_image(), we have to remove the old image from
builder's image-stack (the stack is outside of the node-tree).

+    if dst_fname:
+        name = dst_fname[len(out_dir) + 1:]
+        # the builder needs not to copy one more time, so pop it if exists.
+        translator.builder.images.pop(img_node['uri'], None)
+        img_node['uri'] = dst_fname
+        img_node['candidates'] = {'*': dst_fname}


So you might already guess, this image handling method is bonded
only to Phase 1 and we do not need to touch any builder (Phase 4).
The central point is the last line:

+        img_node['candidates'] = {'*': dst_fname}

The other important thing is, that generated files (dst_fname) are 
always in the out_dir. This is needed, since kernel-build should not
pollute the src_folder. 


@Daniel: Thanks a lot for your:

v2: s/DOC/DOT/ in a few places (by Daniel).
v3: Simplify stuff a bit (by Daniel):

I haven't had time to test and I can't promise that I will
find the time for. So, for the first, take my Acked-by.

--Markus--
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Laurent Pinchart]

Hi Daniel,

Thank you for the patch.

With this applied, I get

make[1]: Entering directory '/home/laurent/src/iob/renesas/linux64'
  SPHINX  htmldocs --> file:///home/laurent/src/iob/renesas/linux64/Documentation/output
  PARSE   include/uapi/linux/videodev2.h
Running Sphinx v1.3.1

Extension error:
Could not import extension kfigure (exception: cannot import name patches)
make[2]: *** [/home/laurent/src/iob/renesas/linux/Documentation/Makefile.sphinx:70: htmldocs] Error 1
make[1]: *** [/home/laurent/src/iob/renesas/linux/Makefile:1453: htmldocs] Error 2
make[1]: Leaving directory '/home/laurent/src/iob/renesas/linux64'
make: *** [Makefile:152: sub-make] Error 2

sphinx.directive.patches got introduced in Sphinx 1.4. If you want to bump the
minimum required version I think a notice is needed.

On Tuesday 28 Feb 2017 18:13:15 Daniel Vetter wrote:
> From: Markus Heiser <markus.heiser@darmarit.de>
> 
> This patch brings scalable figure, image handling and a concept to
> embed *render* markups:
> 
> * DOT (http://www.graphviz.org)
> * SVG
> 
> For image handling use the 'image' replacement::
> 
>     .. kernel-image::  svg_image.svg
> 
>        :alt:    simple SVG image
> 
> For figure handling use the 'figure' replacement::
> 
>     .. kernel-figure::  svg_image.svg
> 
>        :alt:    simple SVG image
> 
>        SVG image example
> 
> Embed *render* markups (or languages) like Graphviz's **DOT** is
> provided by the *render* directive.::
> 
>   .. kernel-render:: DOT
> 
>      :alt: foobar digraph
>      :caption: Embedded **DOT** (Graphviz) code.
> 
>      digraph foo {
>       "bar" -> "baz";
>      }
> 
> The *render* directive is a concept to integrate *render* markups and
> languages, yet supported markups:
> 
> * DOT: render embedded Graphviz's **DOT**
> * SVG: render embedded Scalable Vector Graphics (**SVG**)
> 
> v2: s/DOC/DOT/ in a few places (by Daniel).
> 
> v3: Simplify stuff a bit (by Daniel):
> 
> - Remove path detection and setup/check code for that. In
>   Documentation/media/Makefile we already simply use these tools,
>   better to have one consolidated check if we want/need one. Also
>   remove the convertsvg support, we require ImageMagick's convert
>   already in the doc build, no need for a 2nd fallback.
> 
> - Use sphinx for depency tracking, remove hand-rolled version.
> 
> - Forward stderr from dot and convert, otherwise debugging issues with
>   the diagrams is impossible.
> 
> Cc: Jonathan Corbet <corbet@lwn.net>
> Cc: linux-doc@vger.kernel.org
> Cc: Jani Nikula <jani.nikula@linux.intel.com>
> Cc: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> Signed-off-by: Markus Heiser <markus.heiser@darmarit.de> (v1)
> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
> ---
>  Documentation/conf.py                 |   2 +-
>  Documentation/doc-guide/hello.dot     |   3 +
>  Documentation/doc-guide/sphinx.rst    |  90 ++++++-
>  Documentation/doc-guide/svg_image.svg |  10 +
>  Documentation/process/changes.rst     |   7 +-
>  Documentation/sphinx/kfigure.py       | 442 ++++++++++++++++++++++++++++++
>  6 files changed, 548 insertions(+), 6 deletions(-)
>  create mode 100644 Documentation/doc-guide/hello.dot
>  create mode 100644 Documentation/doc-guide/svg_image.svg
>  create mode 100644 Documentation/sphinx/kfigure.py

-- 
Regards,

Laurent Pinchart

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Daniel Vetter]

On Thu, Mar 2, 2017 at 1:26 PM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> Hi Daniel,
>
> Thank you for the patch.
>
> With this applied, I get
>
> make[1]: Entering directory '/home/laurent/src/iob/renesas/linux64'
>   SPHINX  htmldocs --> file:///home/laurent/src/iob/renesas/linux64/Documentation/output
>   PARSE   include/uapi/linux/videodev2.h
> Running Sphinx v1.3.1
>
> Extension error:
> Could not import extension kfigure (exception: cannot import name patches)
> make[2]: *** [/home/laurent/src/iob/renesas/linux/Documentation/Makefile.sphinx:70: htmldocs] Error 1
> make[1]: *** [/home/laurent/src/iob/renesas/linux/Makefile:1453: htmldocs] Error 2
> make[1]: Leaving directory '/home/laurent/src/iob/renesas/linux64'
> make: *** [Makefile:152: sub-make] Error 2
>
> sphinx.directive.patches got introduced in Sphinx 1.4. If you want to bump the
> minimum required version I think a notice is needed.

Ugh. But this also goes completely over my head, no idea whether we
must require sphinx 1.4 (it was released Mar 28, 2016), or whether
there's some way to work around this ... Halp?
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Laurent Pinchart]

Hi Daniel,

On Thursday 02 Mar 2017 14:54:32 Daniel Vetter wrote:
> On Thu, Mar 2, 2017 at 1:26 PM, Laurent Pinchart wrote:
> > Hi Daniel,
> > 
> > Thank you for the patch.
> > 
> > With this applied, I get
> > 
> > make[1]: Entering directory '/home/laurent/src/iob/renesas/linux64'
> > 
> >   SPHINX  htmldocs -->
> >   file:///home/laurent/src/iob/renesas/linux64/Documentation/output PARSE
> >     include/uapi/linux/videodev2.h
> > 
> > Running Sphinx v1.3.1
> > 
> > Extension error:
> > Could not import extension kfigure (exception: cannot import name patches)
> > make[2]: ***
> > [/home/laurent/src/iob/renesas/linux/Documentation/Makefile.sphinx:70:
> > htmldocs] Error 1 make[1]: ***
> > [/home/laurent/src/iob/renesas/linux/Makefile:1453: htmldocs] Error 2
> > make[1]: Leaving directory '/home/laurent/src/iob/renesas/linux64' make:
> > *** [Makefile:152: sub-make] Error 2
> > 
> > sphinx.directive.patches got introduced in Sphinx 1.4. If you want to bump
> > the minimum required version I think a notice is needed.
> 
> Ugh. But this also goes completely over my head, no idea whether we
> must require sphinx 1.4 (it was released Mar 28, 2016), or whether
> there's some way to work around this ... Halp?

I'm not a Sphinx expert so I don't know, but what I can tell is that copying 
the patches.py from Sphinx 1.4 to Documentation/sphinx/ and modifying 
kfigure.py to import it from there fixes the build. There's thus no extra 
depencency on Sphinx 1.4 (or newer).

I'm not sure we want to set a precedent by copying part of the Sphinx source 
code to the kernel tree (or inlining the single small function that the module 
provides), and I'll let someone more knowledgeable than me decide how to 
proceed.

-- 
Regards,

Laurent Pinchart

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Markus Heiser]

Hi Daniel, Laurent

Am 02.03.2017 um 15:14 schrieb Laurent Pinchart <laurent.pinchart@ideasonboard.com>:

> Hi Daniel,
> 
> On Thursday 02 Mar 2017 14:54:32 Daniel Vetter wrote:
>> On Thu, Mar 2, 2017 at 1:26 PM, Laurent Pinchart wrote:
>>> Hi Daniel,
>>> 
>>> Thank you for the patch.
>>> 
>>> With this applied, I get
>>> 
>>> make[1]: Entering directory '/home/laurent/src/iob/renesas/linux64'
>>> 
>>>  SPHINX  htmldocs -->
>>>  file:///home/laurent/src/iob/renesas/linux64/Documentation/output PARSE
>>>    include/uapi/linux/videodev2.h
>>> 
>>> Running Sphinx v1.3.1
>>> 
>>> Extension error:
>>> Could not import extension kfigure (exception: cannot import name patches)
>>> make[2]: ***
>>> [/home/laurent/src/iob/renesas/linux/Documentation/Makefile.sphinx:70:
>>> htmldocs] Error 1 make[1]: ***
>>> [/home/laurent/src/iob/renesas/linux/Makefile:1453: htmldocs] Error 2
>>> make[1]: Leaving directory '/home/laurent/src/iob/renesas/linux64' make:
>>> *** [Makefile:152: sub-make] Error 2
>>> 
>>> sphinx.directive.patches got introduced in Sphinx 1.4. If you want to bump
>>> the minimum required version I think a notice is needed.
>> 
>> Ugh. But this also goes completely over my head, no idea whether we
>> must require sphinx 1.4 (it was released Mar 28, 2016), or whether
>> there's some way to work around this ... Halp?
> 
> I'm not a Sphinx expert so I don't know, but what I can tell is that copying 
> the patches.py from Sphinx 1.4 to Documentation/sphinx/ and modifying 
> kfigure.py to import it from there fixes the build. There's thus no extra 
> depencency on Sphinx 1.4 (or newer).
> 
> I'm not sure we want to set a precedent by copying part of the Sphinx source 
> code to the kernel tree (or inlining the single small function that the module 
> provides), and I'll let someone more knowledgeable than me decide how to 
> proceed.


Aargh ... we need virtualenv! For interim something like the following
might help. In file Documentation/sphinx/kfigure.py edit the imports

...
from docutils.parsers.rst.directives import images
try:
    from sphinx.directives.patches import Figure
except ImportError:
    Figure = images.Figure 
...

And fix the class definition, so it use 'Figure' and no
longer 'patch.Figure'::

...
-class KernelFigure(patches.Figure):
+class KernelFigure(Figure):
...

Sorry that I have not yet the time to send you a decent and tested
patch. Do you like to test my suggestion? / thanks!

-- Markus --
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Daniel Vetter]

On Thu, Mar 02, 2017 at 03:58:36PM +0100, Markus Heiser wrote:
> Hi Daniel, Laurent
> 
> Am 02.03.2017 um 15:14 schrieb Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> 
> > Hi Daniel,
> > 
> > On Thursday 02 Mar 2017 14:54:32 Daniel Vetter wrote:
> >> On Thu, Mar 2, 2017 at 1:26 PM, Laurent Pinchart wrote:
> >>> Hi Daniel,
> >>> 
> >>> Thank you for the patch.
> >>> 
> >>> With this applied, I get
> >>> 
> >>> make[1]: Entering directory '/home/laurent/src/iob/renesas/linux64'
> >>> 
> >>>  SPHINX  htmldocs -->
> >>>  file:///home/laurent/src/iob/renesas/linux64/Documentation/output PARSE
> >>>    include/uapi/linux/videodev2.h
> >>> 
> >>> Running Sphinx v1.3.1
> >>> 
> >>> Extension error:
> >>> Could not import extension kfigure (exception: cannot import name patches)
> >>> make[2]: ***
> >>> [/home/laurent/src/iob/renesas/linux/Documentation/Makefile.sphinx:70:
> >>> htmldocs] Error 1 make[1]: ***
> >>> [/home/laurent/src/iob/renesas/linux/Makefile:1453: htmldocs] Error 2
> >>> make[1]: Leaving directory '/home/laurent/src/iob/renesas/linux64' make:
> >>> *** [Makefile:152: sub-make] Error 2
> >>> 
> >>> sphinx.directive.patches got introduced in Sphinx 1.4. If you want to bump
> >>> the minimum required version I think a notice is needed.
> >> 
> >> Ugh. But this also goes completely over my head, no idea whether we
> >> must require sphinx 1.4 (it was released Mar 28, 2016), or whether
> >> there's some way to work around this ... Halp?
> > 
> > I'm not a Sphinx expert so I don't know, but what I can tell is that copying 
> > the patches.py from Sphinx 1.4 to Documentation/sphinx/ and modifying 
> > kfigure.py to import it from there fixes the build. There's thus no extra 
> > depencency on Sphinx 1.4 (or newer).
> > 
> > I'm not sure we want to set a precedent by copying part of the Sphinx source 
> > code to the kernel tree (or inlining the single small function that the module 
> > provides), and I'll let someone more knowledgeable than me decide how to 
> > proceed.
> 
> 
> Aargh ... we need virtualenv! For interim something like the following
> might help. In file Documentation/sphinx/kfigure.py edit the imports
> 
> ...
> from docutils.parsers.rst.directives import images
> try:
>     from sphinx.directives.patches import Figure
> except ImportError:
>     Figure = images.Figure 
> ...
> 
> And fix the class definition, so it use 'Figure' and no
> longer 'patch.Figure'::
> 
> ...
> -class KernelFigure(patches.Figure):
> +class KernelFigure(Figure):
> ...
> 
> Sorry that I have not yet the time to send you a decent and tested
> patch. Do you like to test my suggestion? / thanks!

I'll give it a shot at implementing it, but I can't (easily at least) test
on sphinx 1.3.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Laurent Pinchart]

Hi Daniel and Markus,

On Thursday 02 Mar 2017 16:11:08 Daniel Vetter wrote:
> On Thu, Mar 02, 2017 at 03:58:36PM +0100, Markus Heiser wrote:
> > Am 02.03.2017 um 15:14 schrieb Laurent Pinchart:
> > > On Thursday 02 Mar 2017 14:54:32 Daniel Vetter wrote:
> > >> On Thu, Mar 2, 2017 at 1:26 PM, Laurent Pinchart wrote:
> > >>> Hi Daniel,
> > >>> 
> > >>> Thank you for the patch.
> > >>> 
> > >>> With this applied, I get
> > >>> 
> > >>> make[1]: Entering directory '/home/laurent/src/iob/renesas/linux64'
> > >>> 
> > >>>  SPHINX  htmldocs -->
> > >>>  file:///home/laurent/src/iob/renesas/linux64/Documentation/output
> > >>>  PARSE
> > >>>  
> > >>>    include/uapi/linux/videodev2.h
> > >>> 
> > >>> Running Sphinx v1.3.1
> > >>> 
> > >>> Extension error:
> > >>> Could not import extension kfigure (exception: cannot import name
> > >>> patches)
> > >>> make[2]: ***
> > >>> [/home/laurent/src/iob/renesas/linux/Documentation/Makefile.sphinx:70:
> > >>> htmldocs] Error 1 make[1]: ***
> > >>> [/home/laurent/src/iob/renesas/linux/Makefile:1453: htmldocs] Error 2
> > >>> make[1]: Leaving directory '/home/laurent/src/iob/renesas/linux64'
> > >>> make:
> > >>> *** [Makefile:152: sub-make] Error 2
> > >>> 
> > >>> sphinx.directive.patches got introduced in Sphinx 1.4. If you want to
> > >>> bump
> > >>> the minimum required version I think a notice is needed.
> > >> 
> > >> Ugh. But this also goes completely over my head, no idea whether we
> > >> must require sphinx 1.4 (it was released Mar 28, 2016), or whether
> > >> there's some way to work around this ... Halp?
> > > 
> > > I'm not a Sphinx expert so I don't know, but what I can tell is that
> > > copying the patches.py from Sphinx 1.4 to Documentation/sphinx/ and
> > > modifying kfigure.py to import it from there fixes the build. There's
> > > thus no extra depencency on Sphinx 1.4 (or newer).
> > > 
> > > I'm not sure we want to set a precedent by copying part of the Sphinx
> > > source code to the kernel tree (or inlining the single small function
> > > that the module provides), and I'll let someone more knowledgeable than
> > > me decide how to proceed.
> > 
> > Aargh ... we need virtualenv! For interim something like the following
> > might help. In file Documentation/sphinx/kfigure.py edit the imports
> > 
> > ...
> > from docutils.parsers.rst.directives import images
> > 
> > try:
> >     from sphinx.directives.patches import Figure
> > 
> > except ImportError:
> >     Figure = images.Figure
> > 
> > ...
> > 
> > And fix the class definition, so it use 'Figure' and no
> > longer 'patch.Figure'::
> > 
> > ...
> > -class KernelFigure(patches.Figure):
> > +class KernelFigure(Figure):
> > ...
> > 
> > Sorry that I have not yet the time to send you a decent and tested
> > patch. Do you like to test my suggestion? / thanks!
> 
> I'll give it a shot at implementing it, but I can't (easily at least) test
> on sphinx 1.3.

I can, and it works.

Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Thank you for the quick fix Markus.

-- 
Regards,

Laurent Pinchart

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Markus Heiser]

Am 02.03.2017 um 16:11 schrieb Daniel Vetter <daniel@ffwll.ch>:

> On Thu, Mar 02, 2017 at 03:58:36PM +0100, Markus Heiser wrote:
>> Hi Daniel, Laurent
>> 
>> Am 02.03.2017 um 15:14 schrieb Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
>> 
>>> Hi Daniel,
>>> 
>>> On Thursday 02 Mar 2017 14:54:32 Daniel Vetter wrote:
>>>> On Thu, Mar 2, 2017 at 1:26 PM, Laurent Pinchart wrote:
>>>>> Hi Daniel,
>>>>> 
>>>>> Thank you for the patch.
>>>>> 
>>>>> With this applied, I get
>>>>> 
>>>>> make[1]: Entering directory '/home/laurent/src/iob/renesas/linux64'
>>>>> 
>>>>> SPHINX  htmldocs -->
>>>>> file:///home/laurent/src/iob/renesas/linux64/Documentation/output PARSE
>>>>>   include/uapi/linux/videodev2.h
>>>>> 
>>>>> Running Sphinx v1.3.1
>>>>> 
>>>>> Extension error:
>>>>> Could not import extension kfigure (exception: cannot import name patches)
>>>>> make[2]: ***
>>>>> [/home/laurent/src/iob/renesas/linux/Documentation/Makefile.sphinx:70:
>>>>> htmldocs] Error 1 make[1]: ***
>>>>> [/home/laurent/src/iob/renesas/linux/Makefile:1453: htmldocs] Error 2
>>>>> make[1]: Leaving directory '/home/laurent/src/iob/renesas/linux64' make:
>>>>> *** [Makefile:152: sub-make] Error 2
>>>>> 
>>>>> sphinx.directive.patches got introduced in Sphinx 1.4. If you want to bump
>>>>> the minimum required version I think a notice is needed.
>>>> 
>>>> Ugh. But this also goes completely over my head, no idea whether we
>>>> must require sphinx 1.4 (it was released Mar 28, 2016), or whether
>>>> there's some way to work around this ... Halp?
>>> 
>>> I'm not a Sphinx expert so I don't know, but what I can tell is that copying 
>>> the patches.py from Sphinx 1.4 to Documentation/sphinx/ and modifying 
>>> kfigure.py to import it from there fixes the build. There's thus no extra 
>>> depencency on Sphinx 1.4 (or newer).
>>> 
>>> I'm not sure we want to set a precedent by copying part of the Sphinx source 
>>> code to the kernel tree (or inlining the single small function that the module 
>>> provides), and I'll let someone more knowledgeable than me decide how to 
>>> proceed.
>> 
>> 
>> Aargh ... we need virtualenv! For interim something like the following
>> might help. In file Documentation/sphinx/kfigure.py edit the imports
>> 
>> ...
>> from docutils.parsers.rst.directives import images
>> try:
>>    from sphinx.directives.patches import Figure
>> except ImportError:
>>    Figure = images.Figure 
>> ...
>> 
>> And fix the class definition, so it use 'Figure' and no
>> longer 'patch.Figure'::
>> 
>> ...
>> -class KernelFigure(patches.Figure):
>> +class KernelFigure(Figure):
>> ...
>> 
>> Sorry that I have not yet the time to send you a decent and tested
>> patch. Do you like to test my suggestion? / thanks!
> 
> I'll give it a shot at implementing it, but I can't (easily at least) test
> on sphinx 1.3.

You can test it with virtualenv  https://virtualenv.pypa.io/en/stable/userguide/

In short:

$ cd kernel-src
$ virtualenv myenv
$ source myenv/bin/activate
$ pip install 'Sphinx==1.3.1'
$ make ....

-- Markus --


> -Daniel
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Mauro Carvalho Chehab]

Em Thu, 2 Mar 2017 16:21:33 +0100
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 02.03.2017 um 16:11 schrieb Daniel Vetter <daniel@ffwll.ch>:
> 
> > On Thu, Mar 02, 2017 at 03:58:36PM +0100, Markus Heiser wrote:  
> >> Hi Daniel, Laurent
> >> 
> >> Am 02.03.2017 um 15:14 schrieb Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> >>   
> >>> Hi Daniel,
> >>> 
> >>> On Thursday 02 Mar 2017 14:54:32 Daniel Vetter wrote:  
> >>>> On Thu, Mar 2, 2017 at 1:26 PM, Laurent Pinchart wrote:  
> >>>>> Hi Daniel,
> >>>>> 
> >>>>> Thank you for the patch.
> >>>>> 
> >>>>> With this applied, I get
> >>>>> 
> >>>>> make[1]: Entering directory '/home/laurent/src/iob/renesas/linux64'
> >>>>> 
> >>>>> SPHINX  htmldocs -->
> >>>>> file:///home/laurent/src/iob/renesas/linux64/Documentation/output PARSE
> >>>>>   include/uapi/linux/videodev2.h
> >>>>> 
> >>>>> Running Sphinx v1.3.1
> >>>>> 
> >>>>> Extension error:
> >>>>> Could not import extension kfigure (exception: cannot import name patches)
> >>>>> make[2]: ***
> >>>>> [/home/laurent/src/iob/renesas/linux/Documentation/Makefile.sphinx:70:
> >>>>> htmldocs] Error 1 make[1]: ***
> >>>>> [/home/laurent/src/iob/renesas/linux/Makefile:1453: htmldocs] Error 2
> >>>>> make[1]: Leaving directory '/home/laurent/src/iob/renesas/linux64' make:
> >>>>> *** [Makefile:152: sub-make] Error 2
> >>>>> 
> >>>>> sphinx.directive.patches got introduced in Sphinx 1.4. If you want to bump
> >>>>> the minimum required version I think a notice is needed.  
> >>>> 
> >>>> Ugh. But this also goes completely over my head, no idea whether we
> >>>> must require sphinx 1.4 (it was released Mar 28, 2016), or whether
> >>>> there's some way to work around this ... Halp?  
> >>> 
> >>> I'm not a Sphinx expert so I don't know, but what I can tell is that copying 
> >>> the patches.py from Sphinx 1.4 to Documentation/sphinx/ and modifying 
> >>> kfigure.py to import it from there fixes the build. There's thus no extra 
> >>> depencency on Sphinx 1.4 (or newer).
> >>> 
> >>> I'm not sure we want to set a precedent by copying part of the Sphinx source 
> >>> code to the kernel tree (or inlining the single small function that the module 
> >>> provides), and I'll let someone more knowledgeable than me decide how to 
> >>> proceed.  
> >> 
> >> 
> >> Aargh ... we need virtualenv! For interim something like the following
> >> might help. In file Documentation/sphinx/kfigure.py edit the imports
> >> 
> >> ...
> >> from docutils.parsers.rst.directives import images
> >> try:
> >>    from sphinx.directives.patches import Figure
> >> except ImportError:
> >>    Figure = images.Figure 
> >> ...
> >> 
> >> And fix the class definition, so it use 'Figure' and no
> >> longer 'patch.Figure'::
> >> 
> >> ...
> >> -class KernelFigure(patches.Figure):
> >> +class KernelFigure(Figure):
> >> ...
> >> 
> >> Sorry that I have not yet the time to send you a decent and tested
> >> patch. Do you like to test my suggestion? / thanks!  
> > 
> > I'll give it a shot at implementing it, but I can't (easily at least) test
> > on sphinx 1.3.  
> 
> You can test it with virtualenv  https://virtualenv.pypa.io/en/stable/userguide/
> 
> In short:
> 
> $ cd kernel-src
> $ virtualenv myenv
> $ source myenv/bin/activate
> $ pip install 'Sphinx==1.3.1'
> $ make ....

Hmm... at least here, building docs-next with Sphinx 1.3.1 inside a
virtualenv is broken:

writing output... [ 16%] media/intro                                            
Exception occurred:
  File "/devel/v4l/patchwork/myenv-1.3/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
    assert not self.context, 'len(context) = %s' % len(self.context)
AssertionError: len(context) = 1
The full traceback has been saved in /tmp/sphinx-err-W7CPtT.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Documentation/Makefile.sphinx:69: recipe for target 'htmldocs' failed
make[1]: *** [htmldocs] Error 1
Makefile:1450: recipe for target 'htmldocs' failed
make: *** [htmldocs] Error 2

Perhaps it is time for us to move minimal requirements to 1.4?

Thanks,
Mauro

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Mauro Carvalho Chehab]

Em Thu, 2 Mar 2017 12:45:32 -0300
Mauro Carvalho Chehab <mchehab@s-opensource.com> escreveu:

> Em Thu, 2 Mar 2017 16:21:33 +0100
> Markus Heiser <markus.heiser@darmarit.de> escreveu:
> 
> > Am 02.03.2017 um 16:11 schrieb Daniel Vetter <daniel@ffwll.ch>:
> >   
> > > On Thu, Mar 02, 2017 at 03:58:36PM +0100, Markus Heiser wrote:    
> > >> Hi Daniel, Laurent
> > >> 
> > >> Am 02.03.2017 um 15:14 schrieb Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> > >>     
> > >>> Hi Daniel,
> > >>> 
> > >>> On Thursday 02 Mar 2017 14:54:32 Daniel Vetter wrote:    
> > >>>> On Thu, Mar 2, 2017 at 1:26 PM, Laurent Pinchart wrote:    
> > >>>>> Hi Daniel,
> > >>>>> 
> > >>>>> Thank you for the patch.
> > >>>>> 
> > >>>>> With this applied, I get
> > >>>>> 
> > >>>>> make[1]: Entering directory '/home/laurent/src/iob/renesas/linux64'
> > >>>>> 
> > >>>>> SPHINX  htmldocs -->
> > >>>>> file:///home/laurent/src/iob/renesas/linux64/Documentation/output PARSE
> > >>>>>   include/uapi/linux/videodev2.h
> > >>>>> 
> > >>>>> Running Sphinx v1.3.1
> > >>>>> 
> > >>>>> Extension error:
> > >>>>> Could not import extension kfigure (exception: cannot import name patches)
> > >>>>> make[2]: ***
> > >>>>> [/home/laurent/src/iob/renesas/linux/Documentation/Makefile.sphinx:70:
> > >>>>> htmldocs] Error 1 make[1]: ***
> > >>>>> [/home/laurent/src/iob/renesas/linux/Makefile:1453: htmldocs] Error 2
> > >>>>> make[1]: Leaving directory '/home/laurent/src/iob/renesas/linux64' make:
> > >>>>> *** [Makefile:152: sub-make] Error 2
> > >>>>> 
> > >>>>> sphinx.directive.patches got introduced in Sphinx 1.4. If you want to bump
> > >>>>> the minimum required version I think a notice is needed.    
> > >>>> 
> > >>>> Ugh. But this also goes completely over my head, no idea whether we
> > >>>> must require sphinx 1.4 (it was released Mar 28, 2016), or whether
> > >>>> there's some way to work around this ... Halp?    
> > >>> 
> > >>> I'm not a Sphinx expert so I don't know, but what I can tell is that copying 
> > >>> the patches.py from Sphinx 1.4 to Documentation/sphinx/ and modifying 
> > >>> kfigure.py to import it from there fixes the build. There's thus no extra 
> > >>> depencency on Sphinx 1.4 (or newer).
> > >>> 
> > >>> I'm not sure we want to set a precedent by copying part of the Sphinx source 
> > >>> code to the kernel tree (or inlining the single small function that the module 
> > >>> provides), and I'll let someone more knowledgeable than me decide how to 
> > >>> proceed.    
> > >> 
> > >> 
> > >> Aargh ... we need virtualenv! For interim something like the following
> > >> might help. In file Documentation/sphinx/kfigure.py edit the imports
> > >> 
> > >> ...
> > >> from docutils.parsers.rst.directives import images
> > >> try:
> > >>    from sphinx.directives.patches import Figure
> > >> except ImportError:
> > >>    Figure = images.Figure 
> > >> ...
> > >> 
> > >> And fix the class definition, so it use 'Figure' and no
> > >> longer 'patch.Figure'::
> > >> 
> > >> ...
> > >> -class KernelFigure(patches.Figure):
> > >> +class KernelFigure(Figure):
> > >> ...
> > >> 
> > >> Sorry that I have not yet the time to send you a decent and tested
> > >> patch. Do you like to test my suggestion? / thanks!    
> > > 
> > > I'll give it a shot at implementing it, but I can't (easily at least) test
> > > on sphinx 1.3.    
> > 
> > You can test it with virtualenv  https://virtualenv.pypa.io/en/stable/userguide/
> > 
> > In short:
> > 
> > $ cd kernel-src
> > $ virtualenv myenv
> > $ source myenv/bin/activate
> > $ pip install 'Sphinx==1.3.1'
> > $ make ....  
> 
> Hmm... at least here, building docs-next with Sphinx 1.3.1 inside a
> virtualenv is broken:
> 
> writing output... [ 16%] media/intro                                            
> Exception occurred:
>   File "/devel/v4l/patchwork/myenv-1.3/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
>     assert not self.context, 'len(context) = %s' % len(self.context)
> AssertionError: len(context) = 1
> The full traceback has been saved in /tmp/sphinx-err-W7CPtT.log, if you want to report the issue to the developers.
> Please also report this if it was a user error, so that a better error message can be provided next time.
> A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
> Documentation/Makefile.sphinx:69: recipe for target 'htmldocs' failed
> make[1]: *** [htmldocs] Error 1
> Makefile:1450: recipe for target 'htmldocs' failed
> make: *** [htmldocs] Error 2
> 
> Perhaps it is time for us to move minimal requirements to 1.4?

Hmm... the same happened with 1.4.9 inside virtualenv. It built fine
with 1.5.2 (for htmldocs).

Thanks,
Mauro

-

This is the error log with 1.4:

# Sphinx version: 1.4.9
# Python version: 2.7.13 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.5
# Last messages:
#   writing output... [ 18%] media/dvb-drivers/faq
#   writing output... [ 18%] media/dvb-drivers/index
#   writing output... [ 18%] media/dvb-drivers/intro
#   writing output... [ 18%] media/dvb-drivers/lmedm04
#   writing output... [ 18%] media/dvb-drivers/opera-firmware
#   writing output... [ 18%] media/dvb-drivers/technisat
#   writing output... [ 19%] media/dvb-drivers/ttusb-dec
#   writing output... [ 19%] media/dvb-drivers/udev
#   writing output... [ 19%] media/index
#   writing output... [ 19%] media/intro
# Loaded extensions:
#   kernel_include (1.0) from /devel/v4l/patchwork/Documentation/sphinx/kernel_include.pyc
#   rstFlatTable (1.0) from /devel/v4l/patchwork/Documentation/sphinx/rstFlatTable.pyc
#   cdomain (1.0) from /devel/v4l/patchwork/Documentation/sphinx/cdomain.pyc
#   kerneldoc (1.0) from /devel/v4l/patchwork/Documentation/sphinx/kerneldoc.pyc
#   alabaster (0.7.10) from /devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/alabaster/__init__.pyc
#   sphinx.ext.imgmath (1.4.9) from /devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/sphinx/ext/imgmath.pyc
Traceback (most recent call last):
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/sphinx/cmdline.py", line 244, in main
    app.build(opts.force_all, filenames)
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/sphinx/application.py", line 297, in build
    self.builder.build_update()
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 251, in build_update
    'out of date' % len(to_build))
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 322, in build
    self.write(docnames, list(updated_docnames), method)
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 360, in write
    self._write_serial(sorted(docnames), warnings)
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 368, in _write_serial
    self.write_doc(docname, doctree)
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/sphinx/builders/html.py", line 448, in write_doc
    self.docwriter.write(doctree, destination)
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/writers/__init__.py", line 80, in write
    self.translate()
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/sphinx/writers/html.py", line 47, in translate
    self.document.walkabout(visitor)
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/nodes.py", line 187, in walkabout
    visitor.dispatch_departure(self)
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/nodes.py", line 1895, in dispatch_departure
    return method(node)
  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
    assert not self.context, 'len(context) = %s' % len(self.context)
AssertionError: len(context) = 1

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Markus Heiser]

> Am 02.03.2017 um 16:49 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> 
>>> You can test it with virtualenv  https://virtualenv.pypa.io/en/stable/userguide/
>>> 
>>> In short:
>>> 
>>> $ cd kernel-src
>>> $ virtualenv myenv
>>> $ source myenv/bin/activate
>>> $ pip install 'Sphinx==1.3.1'
>>> $ make ....  
>> 
>> Hmm... at least here, building docs-next with Sphinx 1.3.1 inside a
>> virtualenv is broken:
>> 
>> writing output... [ 16%] media/intro                                            
>> Exception occurred:
>>  File "/devel/v4l/patchwork/myenv-1.3/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
>>    assert not self.context, 'len(context) = %s' % len(self.context)
>> AssertionError: len(context) = 1
>> The full traceback has been saved in /tmp/sphinx-err-W7CPtT.log, if you want to report the issue to the developers.
>> Please also report this if it was a user error, so that a better error message can be provided next time.
>> A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
>> Documentation/Makefile.sphinx:69: recipe for target 'htmldocs' failed
>> make[1]: *** [htmldocs] Error 1
>> Makefile:1450: recipe for target 'htmldocs' failed
>> make: *** [htmldocs] Error 2
>> 
>> Perhaps it is time for us to move minimal requirements to 1.4?
> 
> Hmm... the same happened with 1.4.9 inside virtualenv. It built fine
> with 1.5.2 (for htmldocs).
> 
> Thanks,
> Mauro
> 
> -
> 
> This is the error log with 1.4:
> 
> # Sphinx version: 1.4.9
....
>  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/nodes.py", line 187, in walkabout
>    visitor.dispatch_departure(self)
>  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/nodes.py", line 1895, in dispatch_departure
>    return method(node)
>  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
>    assert not self.context, 'len(context) = %s' % len(self.context)
> AssertionError: len(context) = 1

I guess this is a error from newer docutils, so we have to fix docutils version also.

Sphinx itself specifies "docutils>=0.11"

  https://github.com/sphinx-doc/sphinx/blob/1.3.1/setup.py#L52

So I guess it uses a up to date docutils or the ducutils which are already installed system wide.

— Markus —
 

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Mauro Carvalho Chehab]

Em Thu, 2 Mar 2017 17:13:25 +0100
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> > Am 02.03.2017 um 16:49 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> >   
> >>> You can test it with virtualenv  https://virtualenv.pypa.io/en/stable/userguide/
> >>> 
> >>> In short:
> >>> 
> >>> $ cd kernel-src
> >>> $ virtualenv myenv
> >>> $ source myenv/bin/activate
> >>> $ pip install 'Sphinx==1.3.1'
> >>> $ make ....    
> >> 
> >> Hmm... at least here, building docs-next with Sphinx 1.3.1 inside a
> >> virtualenv is broken:
> >> 
> >> writing output... [ 16%] media/intro                                            
> >> Exception occurred:
> >>  File "/devel/v4l/patchwork/myenv-1.3/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
> >>    assert not self.context, 'len(context) = %s' % len(self.context)
> >> AssertionError: len(context) = 1
> >> The full traceback has been saved in /tmp/sphinx-err-W7CPtT.log, if you want to report the issue to the developers.
> >> Please also report this if it was a user error, so that a better error message can be provided next time.
> >> A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
> >> Documentation/Makefile.sphinx:69: recipe for target 'htmldocs' failed
> >> make[1]: *** [htmldocs] Error 1
> >> Makefile:1450: recipe for target 'htmldocs' failed
> >> make: *** [htmldocs] Error 2
> >> 
> >> Perhaps it is time for us to move minimal requirements to 1.4?  
> > 
> > Hmm... the same happened with 1.4.9 inside virtualenv. It built fine
> > with 1.5.2 (for htmldocs).
> > 
> > Thanks,
> > Mauro
> > 
> > -
> > 
> > This is the error log with 1.4:
> > 
> > # Sphinx version: 1.4.9  
> ....
> >  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/nodes.py", line 187, in walkabout
> >    visitor.dispatch_departure(self)
> >  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/nodes.py", line 1895, in dispatch_departure
> >    return method(node)
> >  File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
> >    assert not self.context, 'len(context) = %s' % len(self.context)
> > AssertionError: len(context) = 1  
> 
> I guess this is a error from newer docutils, so we have to fix docutils version also.
> 
> Sphinx itself specifies "docutils>=0.11"
> 
>   https://github.com/sphinx-doc/sphinx/blob/1.3.1/setup.py#L52
> 
> So I guess it uses a up to date docutils or the ducutils which are already installed system wide.

The system-wide docutils is this one:

	python2-docutils-0.13.1-3.fc25.noarch
	python3-docutils-0.13.1-3.fc25.noarch

Btw, I tested also with virtualenv-3/pip3 and the same issue happens there.


Manually installing version 0.11 makes it to work again.

Considering that Sphinx require a specific docutils package for it to
work, perhaps it is time for us to consider to use the virtenv
enchantments at make docs targets :-p



Thanks,
Mauro

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Markus Heiser]

> Am 02.03.2017 um 17:29 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> 
> Em Thu, 2 Mar 2017 17:13:25 +0100
> Markus Heiser <markus.heiser@darmarit.de> escreveu:
> 
>>> Am 02.03.2017 um 16:49 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
>>> 
>>>>> You can test it with virtualenv  https://virtualenv.pypa.io/en/stable/userguide/
>>>>> 
>>>>> In short:
>>>>> 
>>>>> $ cd kernel-src
>>>>> $ virtualenv myenv
>>>>> $ source myenv/bin/activate
>>>>> $ pip install 'Sphinx==1.3.1'
>>>>> $ make ....    
>>>> 
>>>> Hmm... at least here, building docs-next with Sphinx 1.3.1 inside a
>>>> virtualenv is broken:
>>>> 
>>>> writing output... [ 16%] media/intro                                            
>>>> Exception occurred:
>>>> File "/devel/v4l/patchwork/myenv-1.3/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
>>>>   assert not self.context, 'len(context) = %s' % len(self.context)
>>>> AssertionError: len(context) = 1
>>>> The full traceback has been saved in /tmp/sphinx-err-W7CPtT.log, if you want to report the issue to the developers.
>>>> Please also report this if it was a user error, so that a better error message can be provided next time.
>>>> A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
>>>> Documentation/Makefile.sphinx:69: recipe for target 'htmldocs' failed
>>>> make[1]: *** [htmldocs] Error 1
>>>> Makefile:1450: recipe for target 'htmldocs' failed
>>>> make: *** [htmldocs] Error 2
>>>> 
>>>> Perhaps it is time for us to move minimal requirements to 1.4?  
>>> 
>>> Hmm... the same happened with 1.4.9 inside virtualenv. It built fine
>>> with 1.5.2 (for htmldocs).
>>> 
>>> Thanks,
>>> Mauro
>>> 
>>> -
>>> 
>>> This is the error log with 1.4:
>>> 
>>> # Sphinx version: 1.4.9  
>> ....
>>> File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/nodes.py", line 187, in walkabout
>>>   visitor.dispatch_departure(self)
>>> File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/nodes.py", line 1895, in dispatch_departure
>>>   return method(node)
>>> File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
>>>   assert not self.context, 'len(context) = %s' % len(self.context)
>>> AssertionError: len(context) = 1  
>> 
>> I guess this is a error from newer docutils, so we have to fix docutils version also.
>> 
>> Sphinx itself specifies "docutils>=0.11"
>> 
>>  https://github.com/sphinx-doc/sphinx/blob/1.3.1/setup.py#L52
>> 
>> So I guess it uses a up to date docutils or the ducutils which are already installed system wide.
> 
> The system-wide docutils is this one:
> 
> 	python2-docutils-0.13.1-3.fc25.noarch
> 	python3-docutils-0.13.1-3.fc25.noarch

Sorry my mistake, in the traceback we see

File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
  assert not self.context, 'len(context) = %s' % len(self.context)

... which means: system wide installation does not matter.

> Btw, I tested also with virtualenv-3/pip3 and the same issue happens there.
> 
> Manually installing version 0.11 makes it to work again.

Yes, its only about "docutils>=0.11" in Sphinx 1.3 dependencies. 
In Sphinx 1.5 the error:

  https://github.com/sphinx-doc/sphinx/issues/3212

is fixed:

  https://github.com/tk0miya/sphinx/commit/73663f63672f22304810ce6bb9787490ad250127

But this will never be fixed downwards.

All this is about semantic versioning. If you want to promise your
builds, you have to name which versions of dependencies you support.
I guess this is nothing new for kernel developers ;)

The problem is, that PEP440 defines not only ONE version scheme

 """Some hard to read version identifiers are permitted by
    this scheme in order to better accommodate the wide range
    of versioning practices across existing public and private
    Python projects."""

In practice, the python projects use slightly different schemes
which not follow one rule like <main>.<compatible feature>.<patch> 
From history packaging in Python is the hell, it becomes better, but
the problem with slightly different version schemes still exist.

How can this info help us? Now we know, that we have to stick Sphinx
and docutils by patch-levels we are willing to support. Or with
your words ....

> Considering that Sphinx require a specific docutils package for it to
> work, perhaps it is time for us to consider to use the virtenv
> enchantments at make docs targets :-p


This is very easy, if we use a requiremts.txt file where we 
stick the versions and run the sphinx in this build in a
virtualenv which is build up by this requirements.txt.

  https://pip.pypa.io/en/stable/user_guide/#requirements-files

To summarize, I recommend a Makefile.sphinx cmd which does
something like:

  virtualenv output/myenv
  source output/myenv/bin/activate
  pip install -r requirements.txt
  sphinx-build ...
  
I guess this is something we should discuss with Jon, he
is also familiar with it virtualenv. 

-- Markus --






_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Jonathan Corbet]

On Thu, 2 Mar 2017 19:16:47 +0100
Markus Heiser <markus.heiser@darmarit.de> wrote:

> This is very easy, if we use a requiremts.txt file where we 
> stick the versions and run the sphinx in this build in a
> virtualenv which is build up by this requirements.txt.
> 
>   https://pip.pypa.io/en/stable/user_guide/#requirements-files
> 
> To summarize, I recommend a Makefile.sphinx cmd which does
> something like:
> 
>   virtualenv output/myenv
>   source output/myenv/bin/activate
>   pip install -r requirements.txt
>   sphinx-build ...
>   
> I guess this is something we should discuss with Jon, he
> is also familiar with it virtualenv. 

That would perhaps make the build more reliable, but it would also make
the build dependent on net access to PyPI, and that's not an idea I like a
whole lot.  We should be able to do a build without going out on the
network.

I'm kind of pressed for time, but will try to ponder on this more
shortly...

jon
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Markus Heiser]

> Am 02.03.2017 um 19:20 schrieb Jonathan Corbet <corbet@lwn.net>:
> 
> On Thu, 2 Mar 2017 19:16:47 +0100
> Markus Heiser <markus.heiser@darmarit.de> wrote:
> 
>> This is very easy, if we use a requiremts.txt file where we 
>> stick the versions and run the sphinx in this build in a
>> virtualenv which is build up by this requirements.txt.
>> 
>>  https://pip.pypa.io/en/stable/user_guide/#requirements-files
>> 
>> To summarize, I recommend a Makefile.sphinx cmd which does
>> something like:
>> 
>>  virtualenv output/myenv
>>  source output/myenv/bin/activate
>>  pip install -r requirements.txt
>>  sphinx-build ...
>> 
>> I guess this is something we should discuss with Jon, he
>> is also familiar with it virtualenv. 
> 
> That would perhaps make the build more reliable, but it would also make
> the build dependent on net access to PyPI, and that's not an idea I like a
> whole lot.  We should be able to do a build without going out on the
> network.

Right, there are PROS and CONS. Another point is where to place
the virtualenv. In my example above it is placed in output/ with
I’am not really happy.

> I'm kind of pressed for time, but will try to ponder on this more
> shortly…

My recommendation are mostly from a python developer POV, which is
sometimes diametral to what kernel development needs. You now both
POV in deep so I’am hopeful that you find the right conceptual
answers for those open questions ;)

-- Markus --
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Mauro Carvalho Chehab]

Em Thu, 2 Mar 2017 19:16:47 +0100
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> > Am 02.03.2017 um 17:29 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> > 
> > Em Thu, 2 Mar 2017 17:13:25 +0100
> > Markus Heiser <markus.heiser@darmarit.de> escreveu:
> >   
> >>> Am 02.03.2017 um 16:49 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> >>>   
> >>>>> You can test it with virtualenv  https://virtualenv.pypa.io/en/stable/userguide/
> >>>>> 
> >>>>> In short:
> >>>>> 
> >>>>> $ cd kernel-src
> >>>>> $ virtualenv myenv
> >>>>> $ source myenv/bin/activate
> >>>>> $ pip install 'Sphinx==1.3.1'
> >>>>> $ make ....      
> >>>> 
> >>>> Hmm... at least here, building docs-next with Sphinx 1.3.1 inside a
> >>>> virtualenv is broken:
> >>>> 
> >>>> writing output... [ 16%] media/intro                                            
> >>>> Exception occurred:
> >>>> File "/devel/v4l/patchwork/myenv-1.3/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
> >>>>   assert not self.context, 'len(context) = %s' % len(self.context)
> >>>> AssertionError: len(context) = 1
> >>>> The full traceback has been saved in /tmp/sphinx-err-W7CPtT.log, if you want to report the issue to the developers.
> >>>> Please also report this if it was a user error, so that a better error message can be provided next time.
> >>>> A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
> >>>> Documentation/Makefile.sphinx:69: recipe for target 'htmldocs' failed
> >>>> make[1]: *** [htmldocs] Error 1
> >>>> Makefile:1450: recipe for target 'htmldocs' failed
> >>>> make: *** [htmldocs] Error 2
> >>>> 
> >>>> Perhaps it is time for us to move minimal requirements to 1.4?    
> >>> 
> >>> Hmm... the same happened with 1.4.9 inside virtualenv. It built fine
> >>> with 1.5.2 (for htmldocs).
> >>> 
> >>> Thanks,
> >>> Mauro
> >>> 
> >>> -
> >>> 
> >>> This is the error log with 1.4:
> >>> 
> >>> # Sphinx version: 1.4.9    
> >> ....  
> >>> File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/nodes.py", line 187, in walkabout
> >>>   visitor.dispatch_departure(self)
> >>> File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/nodes.py", line 1895, in dispatch_departure
> >>>   return method(node)
> >>> File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
> >>>   assert not self.context, 'len(context) = %s' % len(self.context)
> >>> AssertionError: len(context) = 1    
> >> 
> >> I guess this is a error from newer docutils, so we have to fix docutils version also.
> >> 
> >> Sphinx itself specifies "docutils>=0.11"
> >> 
> >>  https://github.com/sphinx-doc/sphinx/blob/1.3.1/setup.py#L52
> >> 
> >> So I guess it uses a up to date docutils or the ducutils which are already installed system wide.  
> > 
> > The system-wide docutils is this one:
> > 
> > 	python2-docutils-0.13.1-3.fc25.noarch
> > 	python3-docutils-0.13.1-3.fc25.noarch  
> 
> Sorry my mistake, in the traceback we see
> 
> File "/devel/v4l/patchwork/myenv-1.4/lib/python2.7/site-packages/docutils/writers/_html_base.py", line 671, in depart_document
>   assert not self.context, 'len(context) = %s' % len(self.context)
> 
> ... which means: system wide installation does not matter.
> 
> > Btw, I tested also with virtualenv-3/pip3 and the same issue happens there.
> > 
> > Manually installing version 0.11 makes it to work again.  
> 
> Yes, its only about "docutils>=0.11" in Sphinx 1.3 dependencies. 
> In Sphinx 1.5 the error:
> 
>   https://github.com/sphinx-doc/sphinx/issues/3212
> 
> is fixed:
> 
>   https://github.com/tk0miya/sphinx/commit/73663f63672f22304810ce6bb9787490ad250127
> 
> But this will never be fixed downwards.

Crap. This kind of patch should be backported to Sphinx 1.3/1.4,
or Python's PIP repository should be fixed to require docutils
version to be either 0.11 or 0.12, if one installs version 1.3
or 1.4 of Sphinx.

The way it is, PIP repository is broken!

> All this is about semantic versioning. If you want to promise your
> builds, you have to name which versions of dependencies you support.
> I guess this is nothing new for kernel developers ;)

No. At the Kernel, we do everything possible to not break APIs.

If this were not the case, you wouldn't be able to run Sphinx 1.5 
with legacy Kernel versions ;-)

> The problem is, that PEP440 defines not only ONE version scheme
> 
>  """Some hard to read version identifiers are permitted by
>     this scheme in order to better accommodate the wide range
>     of versioning practices across existing public and private
>     Python projects."""
> 
> In practice, the python projects use slightly different schemes
> which not follow one rule like <main>.<compatible feature>.<patch> 
> From history packaging in Python is the hell, it becomes better, but
> the problem with slightly different version schemes still exist.

IMHO, the way Python and python libraries break compatibility is crazy.

Good packaging sense says that, if APIs can break on every single new
release (with seems to be the case of docutils), then a package
depending on such bad-developed library should require the exact
version(s) it is known to work.

When we're discussing about the docs toolchain, I mentioned that I
was afraid that the Python development model would cause this sort
of issues. Unfortunately, it seems that my concerns were pertinent :-(

> How can this info help us? Now we know, that we have to stick Sphinx
> and docutils by patch-levels we are willing to support. Or with
> your words ....
> 
> > Considering that Sphinx require a specific docutils package for it to
> > work, perhaps it is time for us to consider to use the virtenv
> > enchantments at make docs targets :-p  
> 
> 
> This is very easy, if we use a requiremts.txt file where we 
> stick the versions and run the sphinx in this build in a
> virtualenv which is build up by this requirements.txt.
> 
>   https://pip.pypa.io/en/stable/user_guide/#requirements-files
> 
> To summarize, I recommend a Makefile.sphinx cmd which does
> something like:
> 
>   virtualenv output/myenv
>   source output/myenv/bin/activate
>   pip install -r requirements.txt
>   sphinx-build ...
>   
> I guess this is something we should discuss with Jon, he
> is also familiar with it virtualenv. 

Yeah, making kernel build to depend on network can be a problem.

Maybe one way would be to have a sort of "prepare" script that
would be network-dependent, and will install whatever needed to
build the docs. If called, make *docs will use the virtualenv.
Otherwise, it would print a warning message saying that the doc
build is not reliable, but would try to use whatever installed
on the machine.

Thanks,
Mauro

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Markus Heiser]

> Am 02.03.2017 um 20:09 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
...

>> 
>> Yes, its only about "docutils>=0.11" in Sphinx 1.3 dependencies. 
>> In Sphinx 1.5 the error:
>> 
>>  https://github.com/sphinx-doc/sphinx/issues/3212
>> 
>> is fixed:
>> 
>>  https://github.com/tk0miya/sphinx/commit/73663f63672f22304810ce6bb9787490ad250127
>> 
>> But this will never be fixed downwards.
> 
> Crap. This kind of patch should be backported to Sphinx 1.3/1.4,
> or Python's PIP repository should be fixed to require docutils
> version to be either 0.11 or 0.12, if one installs version 1.3
> or 1.4 of Sphinx.
> 
> The way it is, PIP repository is broken!

I leaved a comment at sphinx-doc project:

  https://github.com/sphinx-doc/sphinx/issues/3212#issuecomment-283756374

>> All this is about semantic versioning. If you want to promise your
>> builds, you have to name which versions of dependencies you support.
>> I guess this is nothing new for kernel developers ;)
> 
> No. At the Kernel, we do everything possible to not break APIs.

Sorry I was not precise. I was talking about third tools dependencies
and that this is well handled by the kernel ;)
 
> If this were not the case, you wouldn't be able to run Sphinx 1.5 
> with legacy Kernel versions ;-)

This is what I mean ;)

> 
>> The problem is, that PEP440 defines not only ONE version scheme
>> 
>> """Some hard to read version identifiers are permitted by
>>    this scheme in order to better accommodate the wide range
>>    of versioning practices across existing public and private
>>    Python projects."""
>> 
>> In practice, the python projects use slightly different schemes
>> which not follow one rule like <main>.<compatible feature>.<patch> 
>> From history packaging in Python is the hell, it becomes better, but
>> the problem with slightly different version schemes still exist.
> 
> IMHO, the way Python and python libraries break compatibility is crazy.
> 
> Good packaging sense says that, if APIs can break on every single new
> release (with seems to be the case of docutils), then a package
> depending on such bad-developed library should require the exact
> version(s) it is known to work.

I can only repeat myself, the main problem is, that PEP440 allows
multiple version schemes. Instead of one scheme 

 <main>.<compatible feature>.<patch>

Every project use a slightly different scheme and others
do not care about any scheme.

> When we're discussing about the docs toolchain, I mentioned that I
> was afraid that the Python development model would cause this sort
> of issues. Unfortunately, it seems that my concerns were pertinent :-(

Not really ;) .. you are tend to mix at least three parts

1. Python
2. Python packaging
3. Sphinx developers who do not stick there depencies

But you are right, when you say that in all parts some confusion
prevail. E.g.

1. Python 2 to 3 movement has been done reckless
--> in the meantime we have https://pythonhosted.org/six/

2. Python packaging is a mess (setup-tools, distutils, pip, easy_install ..) 
--> in the meantime we have PyPA who brings us more structure (https://www.pypa.io/en/latest)

3. Developer using Python
--> we all have a learn curve and making errors all days long. 
    But this should not stop us from continue :)


>> I guess this is something we should discuss with Jon, he
>> is also familiar with it virtualenv. 
> 
> Yeah, making kernel build to depend on network can be a problem.
> 
> Maybe one way would be to have a sort of "prepare" script that
> would be network-dependent, and will install whatever needed to
> build the docs. If called, make *docs will use the virtualenv.
> Otherwise, it would print a warning message saying that the doc
> build is not reliable, but would try to use whatever installed
> on the machine.

Could be a workaround. May Jon has continuative ideas, nothing
we have to solve today ... give Jon some time.
 
-- Markus --
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Mauro Carvalho Chehab]

Em Thu, 2 Mar 2017 21:16:05 +0100
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> > Am 02.03.2017 um 20:09 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:  

> I leaved a comment at sphinx-doc project:
> 
>   https://github.com/sphinx-doc/sphinx/issues/3212#issuecomment-283756374

Thanks!

> > Maybe one way would be to have a sort of "prepare" script that
> > would be network-dependent, and will install whatever needed to
> > build the docs. If called, make *docs will use the virtualenv.
> > Otherwise, it would print a warning message saying that the doc
> > build is not reliable, but would try to use whatever installed
> > on the machine.  
> 
> Could be a workaround. May Jon has continuative ideas, nothing
> we have to solve today ... give Jon some time.

Sure.

Thanks,
Mauro

Re: docutils borkage (was: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images)

[Jonathan Corbet]

On Thu, 2 Mar 2017 16:09:21 -0300
Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:

> IMHO, the way Python and python libraries break compatibility is crazy.
> 
> Good packaging sense says that, if APIs can break on every single new
> release (with seems to be the case of docutils), then a package
> depending on such bad-developed library should require the exact
> version(s) it is known to work.
> 
> When we're discussing about the docs toolchain, I mentioned that I
> was afraid that the Python development model would cause this sort
> of issues. Unfortunately, it seems that my concerns were pertinent :-(

It doesn't have to be that way...  The LWN site is a Python application
that has been running since 2002 without any significant issues of this
type, despite having some significant external dependencies.  There is
nothing inherent in Python that creates this kind of mess, it's really
just a matter of having the same discipline that you need when writing a
library in any other language.

It would appear that this discipline is severely lacking in the docutils
camp in particular.

One way kids these days handle such issues is to just bundle up all the
dependencies they are about and ship their own copies.  This isn't just
Python; see https://lwn.net/Articles/712318/ for some much worse
examples.  But I don't think we want to do that.

There does not appear to be an ideal solution here.  I wonder if we want
do to something like this:

 - Put in a little test program to verify that the build system has
   versions of sphinx and docutils that play well together.

 - Add a script to create a working combination in a virtualenv for
   people who need it.  If the build process sees one of those in place,
   it should use it.

I feel pretty strongly that the build should *not* just go off and do the
virtualenv thing automatically.  We should not be fetching and installing
external software on somebody's system, even in a contained environment,
without an explicit request to do so.

Look for a grumpy article on your favorite news site in the relatively
near future...:)

jon
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: docutils borkage (was: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images)

[Mauro Carvalho Chehab]

Em Sat, 4 Mar 2017 07:56:52 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> On Thu, 2 Mar 2017 16:09:21 -0300
> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> 
> > IMHO, the way Python and python libraries break compatibility is crazy.
> > 
> > Good packaging sense says that, if APIs can break on every single new
> > release (with seems to be the case of docutils), then a package
> > depending on such bad-developed library should require the exact
> > version(s) it is known to work.
> > 
> > When we're discussing about the docs toolchain, I mentioned that I
> > was afraid that the Python development model would cause this sort
> > of issues. Unfortunately, it seems that my concerns were pertinent :-(  
> 
> It doesn't have to be that way...  The LWN site is a Python application
> that has been running since 2002 without any significant issues of this
> type, despite having some significant external dependencies.  There is
> nothing inherent in Python that creates this kind of mess, it's really
> just a matter of having the same discipline that you need when writing a
> library in any other language.
> 
> It would appear that this discipline is severely lacking in the docutils
> camp in particular.

>From the answer for this bug at:
	https://github.com/sphinx-doc/sphinx/issues/3212

It seems that Sphinx maintainers don't care enough to avoid those
issues either.

> One way kids these days handle such issues is to just bundle up all the
> dependencies they are about and ship their own copies.  This isn't just
> Python; see https://lwn.net/Articles/712318/ for some much worse
> examples. 

Yeah, the direction they're taken is, IMHO, a bad idea.

> But I don't think we want to do that.
> 
> There does not appear to be an ideal solution here.  I wonder if we want
> do to something like this:
> 
>  - Put in a little test program to verify that the build system has
>    versions of sphinx and docutils that play well together.

I wrote something like that some time ago, meant to build the media
drivers on legacy Kernels:
	https://git.linuxtv.org/media_build.git/tree/build

Perhaps some of the ideas there could be used on such script. The logic
there identifies the distribution, via either lsb or by reading some
files at /etc:

	$system_release = qx(lsb_release -d) if which("lsb_release");
	$system_release =~ s/Description:\s*// if ($system_release);
	$system_release = catcheck("/etc/system-release") if !$system_release;
	$system_release = catcheck("/etc/redhat-release") if !$system_release;
	$system_release = catcheck("/etc/lsb-release") if !$system_release;
	$system_release = catcheck("/etc/gentoo-release") if !$system_release;
	$system_release = catcheck("/etc/issue") if !$system_release;
	$system_release =~ s/\s+$//;

and presents distro-specific commands and package names:

	sub give_hints()
	{

		# Distro-specific hints
		if ($system_release =~ /Red Hat Enterprise Linux/) {
			give_redhat_hints;
			return;
		}
		if ($system_release =~ /Fedora/) {
			give_redhat_hints;
			return;
		}
		if ($system_release =~ /Ubuntu/) {
			give_ubuntu_hints;
			return;
		}
	...

I guess your idea would be to run such test program if someone calls make
with htmldocs target, right? It will still need to check also for the
LaTex packages needed for Math symbols support.

It could make sense to also run it for pdfdocs, although the LaTex
dependencies for it can be harder for us to identify.

>  - Add a script to create a working combination in a virtualenv for
>    people who need it.  If the build process sees one of those in place,
>    it should use it.

Makes sense. The test program could present both alternatives to the
user (manually install required packages or run the virtualenv
creation script).

> I feel pretty strongly that the build should *not* just go off and do the
> virtualenv thing automatically.  We should not be fetching and installing
> external software on somebody's system, even in a contained environment,
> without an explicit request to do so.

Agreed.

> Look for a grumpy article on your favorite news site in the relatively
> near future...:)

Sounds good :-)


Thanks,
Mauro

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Jonathan Corbet]

On Thu, 2 Mar 2017 16:11:08 +0100
Daniel Vetter <daniel@ffwll.ch> wrote:

> I'll give it a shot at implementing it, but I can't (easily at least) test
> on sphinx 1.3.

Virtualenv makes that kind of thing pretty easy to do.  Maybe we should
add a script to create and populate a suitable venv for this kind of
thing.

Meanwhile, I've been seeing only parts 1 and 2 of what's clearly a bigger
series.  Do you want me to carry just these two?

jon
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Daniel Vetter]

On Thu, Mar 2, 2017 at 4:22 PM, Jonathan Corbet <corbet@lwn.net> wrote:
> On Thu, 2 Mar 2017 16:11:08 +0100
> Daniel Vetter <daniel@ffwll.ch> wrote:
>
>> I'll give it a shot at implementing it, but I can't (easily at least) test
>> on sphinx 1.3.
>
> Virtualenv makes that kind of thing pretty easy to do.  Maybe we should
> add a script to create and populate a suitable venv for this kind of
> thing.

Laurent quickly checked v5 of this patch, looks all good now.

> Meanwhile, I've been seeing only parts 1 and 2 of what's clearly a bigger
> series.  Do you want me to carry just these two?

I submitted the entire series to both linux-doc and dri-devel. But
yeah, probably best if you just pull in the first 2 and send me a
topic pull for drm so I can apply the drm stuff on top in drm-misc.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Mauro Carvalho Chehab]

Em Thu, 2 Mar 2017 16:53:04 +0100
Daniel Vetter <daniel@ffwll.ch> escreveu:

> On Thu, Mar 2, 2017 at 4:22 PM, Jonathan Corbet <corbet@lwn.net> wrote:
> > On Thu, 2 Mar 2017 16:11:08 +0100
> > Daniel Vetter <daniel@ffwll.ch> wrote:
> >  
> >> I'll give it a shot at implementing it, but I can't (easily at least) test
> >> on sphinx 1.3.  
> >
> > Virtualenv makes that kind of thing pretty easy to do.  Maybe we should
> > add a script to create and populate a suitable venv for this kind of
> > thing.  
> 
> Laurent quickly checked v5 of this patch, looks all good now.
> 
> > Meanwhile, I've been seeing only parts 1 and 2 of what's clearly a bigger
> > series.  Do you want me to carry just these two?  
> 
> I submitted the entire series to both linux-doc and dri-devel. 

Sure you sent to linux-doc? I'm not seeing it there.

I'm not seeing it at the mirrors neither:
	https://marc.info/?l=linux-doc&r=1&b=201703&w=2
	https://www.spinics.net/lists/linux-doc/maillist.html

Regards,
Mauro

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Daniel Vetter]

On Thu, Mar 02, 2017 at 01:01:36PM -0300, Mauro Carvalho Chehab wrote:
> Em Thu, 2 Mar 2017 16:53:04 +0100
> Daniel Vetter <daniel@ffwll.ch> escreveu:
> 
> > On Thu, Mar 2, 2017 at 4:22 PM, Jonathan Corbet <corbet@lwn.net> wrote:
> > > On Thu, 2 Mar 2017 16:11:08 +0100
> > > Daniel Vetter <daniel@ffwll.ch> wrote:
> > >  
> > >> I'll give it a shot at implementing it, but I can't (easily at least) test
> > >> on sphinx 1.3.  
> > >
> > > Virtualenv makes that kind of thing pretty easy to do.  Maybe we should
> > > add a script to create and populate a suitable venv for this kind of
> > > thing.  
> > 
> > Laurent quickly checked v5 of this patch, looks all good now.
> > 
> > > Meanwhile, I've been seeing only parts 1 and 2 of what's clearly a bigger
> > > series.  Do you want me to carry just these two?  
> > 
> > I submitted the entire series to both linux-doc and dri-devel. 
> 
> Sure you sent to linux-doc? I'm not seeing it there.
> 
> I'm not seeing it at the mirrors neither:
> 	https://marc.info/?l=linux-doc&r=1&b=201703&w=2
> 	https://www.spinics.net/lists/linux-doc/maillist.html

First one I spotted:

https://marc.info/?l=linux-doc&m=148848308304746&w=2

So maybe just mail server hiccup that's recovering now?
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Daniel Vetter]

On Tue, Feb 28, 2017 at 06:13:15PM +0100, Daniel Vetter wrote:
> From: Markus Heiser <markus.heiser@darmarit.de>
> 
> This patch brings scalable figure, image handling and a concept to
> embed *render* markups:
> 
> * DOT (http://www.graphviz.org)
> * SVG
> 
> For image handling use the 'image' replacement::
> 
>     .. kernel-image::  svg_image.svg
>        :alt:    simple SVG image
> 
> For figure handling use the 'figure' replacement::
> 
>     .. kernel-figure::  svg_image.svg
>        :alt:    simple SVG image
> 
>        SVG image example
> 
> Embed *render* markups (or languages) like Graphviz's **DOT** is
> provided by the *render* directive.::
> 
>   .. kernel-render:: DOT
>      :alt: foobar digraph
>      :caption: Embedded **DOT** (Graphviz) code.
> 
>      digraph foo {
>       "bar" -> "baz";
>      }
> 
> The *render* directive is a concept to integrate *render* markups and
> languages, yet supported markups:
> 
> * DOT: render embedded Graphviz's **DOT**
> * SVG: render embedded Scalable Vector Graphics (**SVG**)
> 
> v2: s/DOC/DOT/ in a few places (by Daniel).
> 
> v3: Simplify stuff a bit (by Daniel):
> 
> - Remove path detection and setup/check code for that. In
>   Documentation/media/Makefile we already simply use these tools,
>   better to have one consolidated check if we want/need one. Also
>   remove the convertsvg support, we require ImageMagick's convert
>   already in the doc build, no need for a 2nd fallback.
> 
> - Use sphinx for depency tracking, remove hand-rolled version.
> 
> - Forward stderr from dot and convert, otherwise debugging issues with
>   the diagrams is impossible.
> 
> Cc: Jonathan Corbet <corbet@lwn.net>
> Cc: linux-doc@vger.kernel.org
> Cc: Jani Nikula <jani.nikula@linux.intel.com>
> Cc: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> Signed-off-by: Markus Heiser <markus.heiser@darmarit.de> (v1)
> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>

Just to avoid confusion: Markus&I chatted a bit in private, and he
volunteered to take over, and fix the few issues that need to be fixed in
his original patch:
- drop the convertsvg python fallback, since we need convert anyway
- relative paths
- stderr fwd
- and the small polish I noticed in docs

I think with my blind hacking I botched the job too much :-)

Anyway, still hoping we could land this for 4.12, I really want the
drm-side doc improvements this provides ...

Thanks, Daniel

> ---
>  Documentation/conf.py                 |   2 +-
>  Documentation/doc-guide/hello.dot     |   3 +
>  Documentation/doc-guide/sphinx.rst    |  90 ++++++-
>  Documentation/doc-guide/svg_image.svg |  10 +
>  Documentation/process/changes.rst     |   7 +-
>  Documentation/sphinx/kfigure.py       | 442 ++++++++++++++++++++++++++++++++++
>  6 files changed, 548 insertions(+), 6 deletions(-)
>  create mode 100644 Documentation/doc-guide/hello.dot
>  create mode 100644 Documentation/doc-guide/svg_image.svg
>  create mode 100644 Documentation/sphinx/kfigure.py
> 
> diff --git a/Documentation/conf.py b/Documentation/conf.py
> index f6823cf01275..e3f537ce2935 100644
> --- a/Documentation/conf.py
> +++ b/Documentation/conf.py
> @@ -34,7 +34,7 @@ from load_config import loadConfig
>  # Add any Sphinx extension module names here, as strings. They can be
>  # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
>  # ones.
> -extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain']
> +extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure']
>  
>  # The name of the math extension changed on Sphinx 1.4
>  if major == 1 and minor > 3:
> diff --git a/Documentation/doc-guide/hello.dot b/Documentation/doc-guide/hello.dot
> new file mode 100644
> index 000000000000..504621dfc595
> --- /dev/null
> +++ b/Documentation/doc-guide/hello.dot
> @@ -0,0 +1,3 @@
> +graph G {
> +      Hello -- World
> +}
> diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst
> index 532d65b70500..b902744ce7dd 100644
> --- a/Documentation/doc-guide/sphinx.rst
> +++ b/Documentation/doc-guide/sphinx.rst
> @@ -34,8 +34,10 @@ format-specific subdirectories under ``Documentation/output``.
>  
>  To generate documentation, Sphinx (``sphinx-build``) must obviously be
>  installed. For prettier HTML output, the Read the Docs Sphinx theme
> -(``sphinx_rtd_theme``) is used if available. For PDF output, ``rst2pdf`` is also
> -needed. All of these are widely available and packaged in distributions.
> +(``sphinx_rtd_theme``) is used if available. For PDF output you'll also need
> +``XeLaTeX`` and CairoSVG (http://cairosvg.org) or alternatively ``convert(1)``
> +from ImageMagick (https://www.imagemagick.org). All of these are widely
> +available and packaged in distributions.
>  
>  To pass extra options to Sphinx, you can use the ``SPHINXOPTS`` make
>  variable. For example, use ``make SPHINXOPTS=-v htmldocs`` to get more verbose
> @@ -232,3 +234,87 @@ Rendered as:
>        * .. _`last row`:
>  
>          - column 3
> +
> +
> +Figures & Images
> +================
> +
> +If you want to add an image, you should use the ``kernel-figure`` and
> +``kernel-image`` directives. E.g. to insert a figure with a scalable
> +image format use SVG::
> +
> +    .. kernel-figure::  svg_image.svg
> +       :alt:    simple SVG image
> +
> +       SVG image example
> +
> +.. kernel-figure::  svg_image.svg
> +   :alt:    simple SVG image
> +
> +   SVG image example
> +
> +The kernel figure (and image) directive support **DOT** formated files, see
> +
> +* DOT: http://graphviz.org/pdf/dotguide.pdf
> +* Graphviz: http://www.graphviz.org/content/dot-language
> +
> +A simple example::
> +
> +  .. kernel-figure::  hello.dot
> +     :alt:    hello world
> +
> +     DOT's hello world example
> +
> +.. kernel-figure::  hello.dot
> +   :alt:    hello world
> +
> +   DOT's hello world example
> +
> +Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the
> +``kernel-render`` directives.::
> +
> +  .. kernel-render:: DOT
> +     :alt: foobar digraph
> +     :caption: Embedded **DOT** (Graphviz) code.
> +
> +     digraph foo {
> +      "bar" -> "baz";
> +     }
> +
> +How this will be rendered depends on the installed tools. If Graphviz is
> +installed, you will see an vector image. If not the raw markup is inserted as
> +*literal-block*.
> +
> +.. kernel-render:: DOT
> +   :alt: foobar digraph
> +   :caption: Embedded **DOT** (Graphviz) code.
> +
> +   digraph foo {
> +      "bar" -> "baz";
> +   }
> +
> +The *render* directive has all the options known from the *figure* directive,
> +plus option ``caption``.  If ``caption`` has a value, a *figure* node is
> +inserted. If not, a *image* node is inserted.
> +
> +Embedded **SVG**::
> +
> +  .. kernel-render:: SVG
> +     :caption: Embedded **SVG** markup.
> +     :alt: so-nw-arrow
> +
> +     <?xml version="1.0" encoding="UTF-8"?>
> +     <svg xmlns="http://www.w3.org/2000/svg" version="1.1" ...>
> +        ...
> +     </svg>
> +
> +.. kernel-render:: SVG
> +   :caption: Embedded **SVG** markup.
> +   :alt: so-nw-arrow
> +
> +   <?xml version="1.0" encoding="UTF-8"?>
> +   <svg xmlns="http://www.w3.org/2000/svg"
> +     version="1.1" baseProfile="full" width="70px" height="40px" viewBox="0 0 700 400">
> +   <line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/>
> +   <polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/>
> +   </svg>
> diff --git a/Documentation/doc-guide/svg_image.svg b/Documentation/doc-guide/svg_image.svg
> new file mode 100644
> index 000000000000..5405f85b8137
> --- /dev/null
> +++ b/Documentation/doc-guide/svg_image.svg
> @@ -0,0 +1,10 @@
> +<?xml version="1.0" encoding="UTF-8"?>
> +<!-- originate: https://commons.wikimedia.org/wiki/File:Variable_Resistor.svg -->
> +<svg xmlns="http://www.w3.org/2000/svg"
> +	version="1.1" baseProfile="full"
> +	width="70px" height="40px" viewBox="0 0 700 400">
> +	<line x1="0" y1="200" x2="700" y2="200" stroke="black" stroke-width="20px"/>
> +	<rect x="100" y="100" width="500" height="200" fill="white" stroke="black" stroke-width="20px"/>
> +	<line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/>
> +	<polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/>
> +</svg>
> diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
> index 56ce66114665..e4f25038ef65 100644
> --- a/Documentation/process/changes.rst
> +++ b/Documentation/process/changes.rst
> @@ -318,9 +318,10 @@ PDF outputs, it is recommended to use version 1.4.6.
>  .. note::
>  
>    Please notice that, for PDF and LaTeX output, you'll also need ``XeLaTeX``
> -  version 3.14159265. Depending on the distribution, you may also need
> -  to install a series of ``texlive`` packages that provide the minimal
> -  set of functionalities required for ``XeLaTex`` to work.
> +  version 3.14159265. Depending on the distribution, you may also need to
> +  install a series of ``texlive`` packages that provide the minimal set of
> +  functionalities required for ``XeLaTex`` to work. For PDF output you'll also
> +  need ``convert(1)`` from ImageMagick (https://www.imagemagick.org).
>  
>  Other tools
>  -----------
> diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py
> new file mode 100644
> index 000000000000..8f749d389dae
> --- /dev/null
> +++ b/Documentation/sphinx/kfigure.py
> @@ -0,0 +1,442 @@
> +# -*- coding: utf-8; mode: python -*-
> +# pylint: disable=C0103
> +u"""
> +    scalable figure and image handling
> +    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +    Sphinx extension which implements scalable image handling.
> +
> +    :copyright:  Copyright (C) 2016  Markus Heiser
> +    :license:    GPL Version 2, June 1991 see Linux/COPYING for details.
> +
> +    The build for image formats depence on image's source format and output's
> +    destination format. This extension implement methods to simplify image
> +    handling from the author's POV. Directives like ``kernel-figure`` implement
> +    methods *to* always get the best output-format even if some tools are not
> +    installed.For more details take a look at ``convert_image(...)`` which is
> +    the core of all conversions.
> +
> +    * ``.. kernel-image``: for image handling / ``.. image::`` replacement
> +
> +    * ``.. kernel-figure``: for figure handling / ``.. figure::`` replacement
> +
> +    * ``.. kernel-render``: for render markup / a concept to embed *render*
> +      markups (or languages). Supported markups (see ``RENDER_MARKUP_EXT``)
> +
> +      + ``DOT``: render embedded Graphviz's **DOC**
> +      + ``SVG``: render embedded Scalable Vector Graphics (**SVG**)
> +      + ... *developable*
> +
> +    Used tools:
> +
> +    * ``dot(1)``: Graphviz (http://www.graphviz.org). If Graphviz is not
> +      available, the DOT language is inserted as literal-block.
> +
> +    * SVG to PDF: To generate PDF, you need at least one of this tools:
> +
> +      - ``convert(1)``: ImageMagick (https://www.imagemagick.org)
> +
> +    List of customizations:
> +
> +    * generate PDF from SVG / used by PDF (LaTeX) builder
> +
> +    * generate SVG (html-builder) and PDF (latex-builder) from DOT files.
> +      DOT: see http://www.graphviz.org/content/dot-language
> +
> +    """
> +
> +import os
> +from os import path
> +import subprocess
> +from hashlib import sha1
> +import sys
> +
> +from docutils import nodes
> +from docutils.statemachine import ViewList
> +from docutils.parsers.rst import directives
> +from docutils.parsers.rst.directives import images
> +
> +from sphinx.directives import patches
> +
> +__version__  = '1.0'
> +
> +# simple helper
> +# -------------
> +
> +def mkdir(folder, mode=0o775):
> +    if not path.isdir(folder):
> +        os.makedirs(folder, mode)
> +
> +# def debug_handle(self, node):           # pylint: disable=W0613
> +#     from linuxdoc.kernel_doc import CONSOLE
> +#     CONSOLE()
> +
> +def pass_handle(self, node):           # pylint: disable=W0613
> +    pass
> +
> +# setup conversion tools and sphinx extension
> +# -------------------------------------------
> +
> +def setup(app):
> +    # image handling
> +    app.add_directive("kernel-image",  KernelImage)
> +    app.add_node(kernel_image,
> +                 html    = (visit_kernel_image, pass_handle),
> +                 latex   = (visit_kernel_image, pass_handle),
> +                 texinfo = (visit_kernel_image, pass_handle),
> +                 text    = (visit_kernel_image, pass_handle),
> +                 man     = (visit_kernel_image, pass_handle), )
> +
> +    # figure handling
> +    app.add_directive("kernel-figure", KernelFigure)
> +    app.add_node(kernel_figure,
> +                 html    = (visit_kernel_figure, pass_handle),
> +                 latex   = (visit_kernel_figure, pass_handle),
> +                 texinfo = (visit_kernel_figure, pass_handle),
> +                 text    = (visit_kernel_figure, pass_handle),
> +                 man     = (visit_kernel_figure, pass_handle), )
> +
> +    # render handling
> +    app.add_directive('kernel-render', KernelRender)
> +    app.add_node(kernel_render,
> +                 html    = (visit_kernel_render, pass_handle),
> +                 latex   = (visit_kernel_render, pass_handle),
> +                 texinfo = (visit_kernel_render, pass_handle),
> +                 text    = (visit_kernel_render, pass_handle),
> +                 man     = (visit_kernel_render, pass_handle), )
> +
> +    return dict(
> +        version = __version__,
> +        parallel_read_safe = True,
> +        parallel_write_safe = True
> +    )
> +
> +# integrate conversion tools
> +# --------------------------
> +
> +RENDER_MARKUP_EXT = {
> +    # The '.ext' must be handled by convert_image(..) function's *in_ext* input.
> +    # <name> : <.ext>
> +    'DOT'   : '.dot'
> +    , 'SVG' : '.svg'
> +}
> +
> +def convert_image(img_node, translator):  # pylint: disable=R0912
> +    """Convert an image node for the builder.
> +
> +    Different builder prefer different image formats, e.g. *latex* builder
> +    prefer PDF while *html* builder prefer SVG format for images.
> +
> +    This function handles outputs image formats in depence of source the format
> +    of the image and the translator's output format. This also means to
> +    manipulate/update the *image* dictionary of the builder (``builder.images``)
> +
> +    """
> +    fname, in_ext = path.splitext(path.basename(img_node['uri']))
> +    src_fname     = path.join(translator.builder.srcdir, img_node['uri'])
> +    src_folder    = path.dirname(img_node['uri'])
> +    out_dir       = translator.builder.outdir
> +    dst_fname     = None
> +
> +    # in kernel builds, use 'make SPHINXOPTS=-v' to see verbose messages
> +    verbose = translator.builder.app.verbose
> +    warn    = translator.builder.warn
> +
> +    verbose('assert best format for: ' + img_node['uri'])
> +
> +    if in_ext == '.dot':
> +
> +        # ----------
> +        # handle DOT
> +        # ----------
> +
> +        dst_fname = path.join(out_dir, fname + '.pdf')
> +
> +        if translator.builder.format == 'html':
> +            dst_fname = path.join(out_dir, src_folder, fname + '.svg')
> +        else:
> +            # all other builder formats will include DOT as raw
> +            with open(src_fname, "r") as dot:
> +                data = dot.read()
> +                node = nodes.literal_block(data, data)
> +                img_node.replace_self(node)
> +
> +
> +    elif in_ext == '.svg':
> +
> +        # ----------
> +        # handle SVG
> +        # ----------
> +
> +        if translator.builder.format == 'latex':
> +            dst_fname = path.join(out_dir, fname + '.pdf')
> +
> +    if dst_fname:
> +        name = dst_fname[len(out_dir) + 1:]
> +        # the builder needs not to copy one more time, so pop it if exists.
> +        translator.builder.images.pop(img_node['uri'], None)
> +        img_node['uri'] = dst_fname
> +        img_node['candidates'] = {'*': dst_fname}
> +
> +        mkdir(path.dirname(dst_fname))
> +
> +        if in_ext == '.dot':
> +            verbose('convert DOT to: {out}/' + name)
> +            dot2format(src_fname, dst_fname)
> +
> +        elif in_ext == '.svg':
> +            verbose('convert SVG to: {out}/' + name)
> +            svg2pdf(src_fname, dst_fname)
> +
> +def dot2format(dot_fname, out_fname):
> +    """Converts DOT file to ``out_fname`` using ``dot(1)``.
> +
> +    * ``dot_fname`` pathname of the input DOT file, including extension ``.dot``
> +    * ``out_fname`` pathname of the output file, including format extension
> +
> +    The *format extension* depends on the ``dot`` command (see ``man dot``
> +    option ``-Txxx``). Normally you will use one of the following extensions:
> +
> +    - ``.ps`` for PostScript,
> +    - ``.svg`` or ``svgz`` for Structured Vector Graphics,
> +    - ``.fig`` for XFIG graphics and
> +    - ``.png`` or ``gif`` for common bitmap graphics.
> +
> +    """
> +    out_format = path.splitext(out_fname)[1][1:]
> +    cmd = ['dot', '-T%s' % out_format, dot_fname]
> +    exit_code = 42
> +    with open(out_fname, "w") as out:
> +        p = subprocess.Popen(
> +            cmd, stdout = out, stderr = subprocess.PIPE )
> +        nil, err = p.communicate()
> +
> +        sys.stderr.write(err)
> +
> +        exit_code = p.returncode
> +        out.flush()
> +    return bool(exit_code == 0)
> +
> +def svg2pdf(svg_fname, pdf_fname):
> +    """Converts SVG to PDF with CairoSVG or ``convert(1)`` command.
> +
> +    Uses ``convert(1)`` from ImageMagick (https://www.imagemagick.org) for
> +    conversion.  Returns ``True`` on success and ``False`` if an error occurred
> +    (e.g. none of the conversion tool is available).
> +
> +    * ``svg_fname`` pathname of the input SVG file with extension (``.svg``)
> +    * ``pdf_name``  pathname of the output PDF file with extension (``.pdf``)
> +
> +    """
> +    cmd = [convert_cmd, svg_fname, pdf_fname]
> +    p = subprocess.Popen(
> +        cmd, stdout = out, stderr = subprocess.PIPE )
> +    nil, err = p.communicate()
> +
> +    sys.stderr.write(err)
> +
> +    exit_code = p.returncode
> +    return bool(exit_code == 0)
> +
> +
> +# image handling
> +# ---------------------
> +
> +def visit_kernel_image(self, node):    # pylint: disable=W0613
> +    """Visitor of the ``kernel_image`` Node.
> +
> +    Handles the ``image`` child-node with the ``convert_image(...)``.
> +    """
> +    img_node = node[0]
> +    convert_image(img_node, self)
> +
> +class kernel_image(nodes.General, nodes.Element):
> +    """Node for ``kernel-image`` directive."""
> +    pass
> +
> +class KernelImage(images.Image):
> +    u"""KernelImage directive
> +
> +    Earns everything from ``.. image::`` directive, except *remote URI* and
> +    *glob* pattern. The KernelImage wraps a image node into a
> +    kernel_image node. See ``visit_kernel_image``.
> +    """
> +
> +    def run(self):
> +        env = self.state.document.settings.env
> +
> +        uri = self.arguments[0]
> +        if uri.endswith('.*') or uri.find('://') != -1:
> +            raise self.severe(
> +                'Error in "%s: %s": glob pattern and remote images are not allowed'
> +                % (self.name, uri))
> +
> +        # Tell sphinx of the dependency
> +        env.note_dependency(os.path.abspath(uri))
> +
> +        result = images.Image.run(self)
> +        if len(result) == 2 or isinstance(result[0], nodes.system_message):
> +            return result
> +        (image_node,) = result
> +        # wrap image node into a kernel_image node / see visitors
> +        node = kernel_image('', image_node)
> +        return [node]
> +
> +# figure handling
> +# ---------------------
> +
> +def visit_kernel_figure(self, node):   # pylint: disable=W0613
> +    """Visitor of the ``kernel_figure`` Node.
> +
> +    Handles the ``image`` child-node with the ``convert_image(...)``.
> +    """
> +    img_node = node[0][0]
> +    convert_image(img_node, self)
> +
> +class kernel_figure(nodes.General, nodes.Element):
> +    """Node for ``kernel-figure`` directive."""
> +
> +class KernelFigure(patches.Figure):
> +    u"""KernelImage directive
> +
> +    Earns everything from ``.. figure::`` directive, except *remote URI* and
> +    *glob* pattern.  The KernelFigure wraps a figure node into a kernel_figure
> +    node. See ``visit_kernel_figure``.
> +    """
> +
> +    def run(self):
> +        env = self.state.document.settings.env
> +
> +        uri = self.arguments[0]
> +        if uri.endswith('.*') or uri.find('://') != -1:
> +            raise self.severe(
> +                'Error in "%s: %s":'
> +                ' glob pattern and remote images are not allowed'
> +                % (self.name, uri))
> +
> +        # Tell sphinx of the dependency
> +        env.note_dependency(os.path.abspath(uri))
> +
> +        result = patches.Figure.run(self)
> +        if len(result) == 2 or isinstance(result[0], nodes.system_message):
> +            return result
> +        (figure_node,) = result
> +        # wrap figure node into a kernel_figure node / see visitors
> +        node = kernel_figure('', figure_node)
> +        return [node]
> +
> +
> +# render handling
> +# ---------------------
> +
> +def visit_kernel_render(self, node):
> +    """Visitor of the ``kernel_render`` Node.
> +
> +    If rendering tools available, save the markup of the ``literal_block`` child
> +    node into a file and replace the ``literal_block`` node with a new created
> +    ``image`` node, pointing to the saved markup file. Afterwards, handle the
> +    image child-node with the ``convert_image(...)``.
> +    """
> +
> +    verbose = self.builder.app.verbose
> +    warn    = self.builder.warn
> +    srclang = node.get('srclang')
> +
> +    verbose('visit kernel-render node lang: "%s"' % (srclang))
> +
> +    tmp_ext = RENDER_MARKUP_EXT.get(srclang, None)
> +    if tmp_ext is None:
> +        warn('kernel-render: "%s" unknow / include raw.' % (srclang))
> +        return
> +
> +    literal_block = node[0]
> +    code = literal_block.astext()
> +
> +    if tmp_ext:
> +        hashobj   = code.encode('utf-8') #  str(node.attributes)
> +        fname     = '%s-%s' % (srclang, sha1(hashobj).hexdigest())
> +        tmp_fname = path.join(
> +            self.builder.outdir, self.builder.imagedir, fname + tmp_ext)
> +
> +        if not path.isfile(tmp_fname):
> +            mkdir(path.dirname(tmp_fname))
> +            with open(tmp_fname, "w") as out:
> +                out.write(code)
> +
> +        image_node = nodes.image(node.rawsource, **node.attributes)
> +        image_node['uri'] = tmp_fname
> +
> +        literal_block.replace_self(image_node)
> +        convert_image(image_node, self)
> +
> +
> +class kernel_render(nodes.General, nodes.Inline, nodes.Element):
> +    """Node for ``kernel-render`` directive."""
> +    pass
> +
> +class KernelRender(patches.Figure):
> +    u"""KernelRender directive
> +
> +    Render content by external tool.  Has all the options known from the
> +    *figure*  directive, plus option ``caption``.  If ``caption`` has a
> +    value, a figure node with the *caption* is inserted. If not, a image node is
> +    inserted.
> +
> +    The KernelRender directive wraps the text of the directive into a
> +    literal_block node and wraps it into a kernel_render node. See
> +    ``visit_kernel_render``.
> +    """
> +    has_content = True
> +    required_arguments = 1
> +    optional_arguments = 0
> +    final_argument_whitespace = False
> +
> +    # earn options from 'figure'
> +    option_spec = patches.Figure.option_spec.copy()
> +    option_spec['caption'] = directives.unchanged
> +
> +    def run(self):
> +        return [self.build_node()]
> +
> +    def build_node(self):
> +
> +        srclang = self.arguments[0].strip()
> +        if srclang not in RENDER_MARKUP_EXT.keys():
> +            return [self.state_machine.reporter.warning(
> +                'Unknow source language "%s", use one of: %s.' % (
> +                    srclang, ",".join(RENDER_MARKUP_EXT.keys())),
> +                line=self.lineno)]
> +
> +        code = '\n'.join(self.content)
> +        if not code.strip():
> +            return [self.state_machine.reporter.warning(
> +                'Ignoring "%s" directive without content.' % (
> +                    self.name),
> +                line=self.lineno)]
> +
> +        node = kernel_render()
> +        node['alt'] = self.options.get('alt','')
> +        node['srclang'] = srclang
> +        literal_node = nodes.literal_block(code, code)
> +        node += literal_node
> +
> +        caption = self.options.get('caption')
> +        if caption:
> +            # parse cation's content
> +            parsed = nodes.Element()
> +            self.state.nested_parse(
> +                ViewList([caption], source=''), self.content_offset, parsed)
> +            caption_node = nodes.caption(
> +                parsed[0].rawsource, '', *parsed[0].children)
> +            caption_node.source = parsed[0].source
> +            caption_node.line = parsed[0].line
> +
> +            figure_node = nodes.figure('', node)
> +            for k,v in self.options.items():
> +                figure_node[k] = v
> +            figure_node += caption_node
> +
> +            node = figure_node
> +
> +        return node
> +
> -- 
> 2.11.0
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Markus Heiser]

Am 06.03.2017 um 11:12 schrieb Daniel Vetter <daniel@ffwll.ch>:

> Just to avoid confusion: Markus&I chatted a bit in private, and he
> volunteered to take over, and fix the few issues that need to be fixed in
> his original patch:
> - drop the convertsvg python fallback, since we need convert anyway
> - relative paths
> - stderr fwd
> - and the small polish I noticed in docs
> 
> I think with my blind hacking I botched the job too much :-)
> 
> Anyway, still hoping we could land this for 4.12, I really want the
> drm-side doc improvements this provides ...
> 
> Thanks, Daniel

FYI: the patch is available:

  https://www.mail-archive.com/linux-doc@vger.kernel.org/msg09881.html

-- Markus --

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

[Daniel Vetter]

On Mon, Mar 06, 2017 at 02:13:00PM +0100, Markus Heiser wrote:
> 
> Am 06.03.2017 um 11:12 schrieb Daniel Vetter <daniel@ffwll.ch>:
> 
> > Just to avoid confusion: Markus&I chatted a bit in private, and he
> > volunteered to take over, and fix the few issues that need to be fixed in
> > his original patch:
> > - drop the convertsvg python fallback, since we need convert anyway
> > - relative paths
> > - stderr fwd
> > - and the small polish I noticed in docs
> > 
> > I think with my blind hacking I botched the job too much :-)
> > 
> > Anyway, still hoping we could land this for 4.12, I really want the
> > drm-side doc improvements this provides ...
> > 
> > Thanks, Daniel
> 
> FYI: the patch is available:
> 
>   https://www.mail-archive.com/linux-doc@vger.kernel.org/msg09881.html

Ah cool, I'll get right around to testing it.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 3/6] drm/doc: Add KMS overview graphs

[Daniel Vetter]

Oh, the shiny and pretties!

Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
---
 Documentation/gpu/drm-kms-helpers.rst |   4 ++
 Documentation/gpu/drm-kms.rst         | 132 ++++++++++++++++++++++++++++++++++
 2 files changed, 136 insertions(+)

diff --git a/Documentation/gpu/drm-kms-helpers.rst b/Documentation/gpu/drm-kms-helpers.rst
index 03040aa14fe8..012b6ff3ec3f 100644
--- a/Documentation/gpu/drm-kms-helpers.rst
+++ b/Documentation/gpu/drm-kms-helpers.rst
@@ -114,6 +114,8 @@ Framebuffer CMA Helper Functions Reference
 .. kernel-doc:: drivers/gpu/drm/drm_fb_cma_helper.c
    :export:
 
+.. _drm_bridges:
+
 Bridges
 =======
 
@@ -139,6 +141,8 @@ Bridge Helper Reference
 .. kernel-doc:: drivers/gpu/drm/drm_bridge.c
    :export:
 
+.. _drm_panel_helper:
+
 Panel Helper Reference
 ======================
 
diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
index 4d4068855ec4..87d8162c9a34 100644
--- a/Documentation/gpu/drm-kms.rst
+++ b/Documentation/gpu/drm-kms.rst
@@ -17,6 +17,138 @@ be setup by initializing the following fields.
 
 Mode Configuration
 
+Overview
+========
+
+.. kernel-render:: DOT
+   :alt: KMS Display Pipeline
+   :caption: KMS Display Pipeline Overview
+
+   digraph "KMS" {
+      node [shape=box]
+
+      subgraph cluster_static {
+          style=dashed
+          label="Static Objects"
+
+          node [bgcolor=grey style=filled]
+          "drm_plane A" -> "drm_crtc"
+          "drm_plane B" -> "drm_crtc"
+          "drm_crtc" -> "drm_encoder A"
+          "drm_crtc" -> "drm_encoder B"
+      }
+
+      subgraph cluster_user_created {
+          style=dashed
+          label="Userspace-Created"
+
+          node [shape=oval]
+          "drm_framebuffer 1" -> "drm_plane A"
+          "drm_framebuffer 2" -> "drm_plane B"
+      }
+
+      subgraph cluster_connector {
+          style=dashed
+          label="Hotpluggable"
+
+          "drm_encoder A" -> "drm_connector A"
+          "drm_encoder B" -> "drm_connector B"
+      }
+   }
+
+The basic object structure KMS presents to userspace is fairly simple.
+Framebuffers (represented by :c:type:`struct drm_framebuffer <drm_framebuffer>`,
+see `Frame Buffer Abstraction`_) feed into planes. Multiple (or just one, or
+even no) planes feed their pixel data into a CRTC (represented by
+:c:type:`struct drm_crtc <drm_crtc>`, see `CRTC Abstraction`_) for blending. The
+precise blending step is explained in more detail in `Plane Composition
+Properties`_ and related chapter.
+
+For the output routing the first step are Encoders (represented by
+:c:type:`struct drm_encoder <drm_encoder>`, see `Encoder Abstraction`_). Those
+are really just internal artifacts of the helper libraries used to implement KMS
+drivers. But unfortunately encoders have been exposed to userspace. Besides that
+they make it unecessarily more complicated for userspace to figure out which
+connections between a CRTC and a connector, and what kind of cloning is
+supported, they serve no purpose in the userspace API. Futhermore the exposed
+restrictions are often wrongly set by drivers, and in many cases not powerful
+enough to express the real restrictions. A CRTC can be connected to multiple
+Encoders, but for an active CRTC there must be at least one encoder.
+
+The final, and real, endpoint in the display chain is the connector (represented
+by :c:type:`struct drm_connector <drm_connector>`, see `Connector
+Abstraction`_). Connectors can have different possible encoders, but the kernel
+driver does this selection. The use case is DVI, which could switch between an
+analog and a digital encoder. There is only ever one active connector for any
+encoder.
+
+Internally the output pipeline is a bit more complex and matches todays hardware
+more closely:
+
+.. kernel-render:: DOT
+   :alt: KMS Output Pipeline
+   :caption: KMS Output Pipeline
+
+   digraph "Output Pipeline" {
+      node [shape=box]
+
+      subgraph {
+          "drm_crtc" [bgcolor=grey style=filled]
+      }
+
+      subgraph cluster_internal {
+          style=dashed
+          label="Internal Pipeline"
+          {
+              node [bgcolor=grey style=filled]
+              "drm_encoder A";
+              "drm_encoder B";
+              "drm_encoder C";
+          }
+
+          {
+              node [bgcolor=grey style=filled]
+              "drm_encoder B" -> "drm_bridge B"
+              "drm_encoder C" -> "drm_bridge C1"
+              "drm_bridge C1" -> "drm_bridge C2";
+          }
+      }
+
+      "drm_crtc" -> "drm_encoder A"
+      "drm_crtc" -> "drm_encoder B"
+      "drm_crtc" -> "drm_encoder C"
+
+
+      subgraph cluster_output {
+          style=dashed
+          label="Outputs"
+
+          "drm_encoder A" -> "drm_connector A";
+          "drm_bridge B" -> "drm_connector B";
+          "drm_bridge C2" -> "drm_connector C";
+
+          "drm_panel"
+      }
+   }
+
+Internally two additional helper objects come into play. First, to be able to
+share code for encoders (sometimes on the same SoC, sometimes off-chip) one or
+more :ref:`drm_bridges` (represented by :c:type:`struct drm_bridge
+<drm_bridge>`) can be linked to an encoder. This link is static and cannot be
+changed, which means the cross-bar (if there is any) needs to be mapped between
+the CRTC and any encoders. Often for drivers with bridges there's no code left
+at the encoder level. Atomic drivers can leave out all the encoder callbacks to
+essentially only leave a dummy routing object behind, which is needed for
+backwards compatibility since encoders are exposed to userspace.
+
+The second objects are panels, represented by :c:type:`struct drm_panel
+<drm_panel>`, see :ref:`drm_panel_helper`. Panels do not have a fixed binding
+point, but are generally linked to the driver private structure which embeds
+:c:type:`struct drm_connector <drm_connector>`.
+
+Note that currently the bridge chaining and interactions with connectors and
+panels are still in-flux and not really fully sorted out yet.
+
 KMS Core Structures and Functions
 =================================
 
-- 
2.11.0

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 3/6] drm/doc: Add KMS overview graphs

[Gabriel Krisman Bertazi]

Daniel Vetter <daniel.vetter@ffwll.ch> writes:

> Oh, the shiny and pretties!
>

Very nice!

Reviewed-by: Gabriel Krisman Bertazi <krisman@collabora.co.uk>


-- 
Gabriel Krisman Bertazi
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 3/6] drm/doc: Add KMS overview graphs

[Laurent Pinchart]

Hi Daniel,

Thank you for the patch.

On Tuesday 28 Feb 2017 18:13:16 Daniel Vetter wrote:
> Oh, the shiny and pretties!
> 
> Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>

Overall this looks good to me, please see below for a few minor issues.

> ---
>  Documentation/gpu/drm-kms-helpers.rst |   4 ++
>  Documentation/gpu/drm-kms.rst         | 132 +++++++++++++++++++++++++++++++
>  2 files changed, 136 insertions(+)
> 
> diff --git a/Documentation/gpu/drm-kms-helpers.rst
> b/Documentation/gpu/drm-kms-helpers.rst index 03040aa14fe8..012b6ff3ec3f
> 100644
> --- a/Documentation/gpu/drm-kms-helpers.rst
> +++ b/Documentation/gpu/drm-kms-helpers.rst
> @@ -114,6 +114,8 @@ Framebuffer CMA Helper Functions Reference
>  .. kernel-doc:: drivers/gpu/drm/drm_fb_cma_helper.c
> 
>     :export:
> +.. _drm_bridges:
> +
>  Bridges
>  =======
> 
> @@ -139,6 +141,8 @@ Bridge Helper Reference
>  .. kernel-doc:: drivers/gpu/drm/drm_bridge.c
> 
>     :export:
> +.. _drm_panel_helper:
> +
>  Panel Helper Reference
>  ======================
> 
> diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
> index 4d4068855ec4..87d8162c9a34 100644
> --- a/Documentation/gpu/drm-kms.rst
> +++ b/Documentation/gpu/drm-kms.rst
> @@ -17,6 +17,138 @@ be setup by initializing the following fields.
> 
>  Mode Configuration

Not part of this patch, but this line doesn't feel like it's where it shoul 
dbe.

> +Overview
> +========
> +
> +.. kernel-render:: DOT
> +   :alt: KMS Display Pipeline
> +   :caption: KMS Display Pipeline Overview
> +
> +   digraph "KMS" {
> +      node [shape=box]
> +
> +      subgraph cluster_static {
> +          style=dashed
> +          label="Static Objects"
> +
> +          node [bgcolor=grey style=filled]
> +          "drm_plane A" -> "drm_crtc"
> +          "drm_plane B" -> "drm_crtc"
> +          "drm_crtc" -> "drm_encoder A"
> +          "drm_crtc" -> "drm_encoder B"
> +      }
> +
> +      subgraph cluster_user_created {
> +          style=dashed
> +          label="Userspace-Created"
> +
> +          node [shape=oval]
> +          "drm_framebuffer 1" -> "drm_plane A"
> +          "drm_framebuffer 2" -> "drm_plane B"
> +      }
> +
> +      subgraph cluster_connector {
> +          style=dashed
> +          label="Hotpluggable"
> +
> +          "drm_encoder A" -> "drm_connector A"
> +          "drm_encoder B" -> "drm_connector B"
> +      }
> +   }
> +
> +The basic object structure KMS presents to userspace is fairly simple.
> +Framebuffers (represented by :c:type:`struct drm_framebuffer
> <drm_framebuffer>`, +see `Frame Buffer Abstraction`_) feed into planes.
> Multiple (or just one, or +even no) planes

I'd say "One or multiple (or even no) planes", but that's up to you.

> feed their pixel data into a
> CRTC (represented by +:c:type:`struct drm_crtc <drm_crtc>`, see `CRTC
> Abstraction`_) for blending. The +precise blending step is explained in
> more detail in `Plane Composition +Properties`_ and related chapter.

s/chapter/chapters/ ? Or /related chapter/the related chapter/ ?

> +
> +For the output routing the first step are Encoders (represented by

s/are/is/

> +:c:type:`struct drm_encoder <drm_encoder>`, see `Encoder Abstraction`_).
> Those +are really just internal artifacts of the helper libraries used to
> implement KMS +drivers. But unfortunately encoders have been exposed to

s/But u/U/

(http://blog.oxforddictionaries.com/2012/01/can-i-start-a-sentence-with-a-conjunction/)

I'd actually move the sentence towards the end of the paragraph and modify it 
to

"Unfortunately connectors have been exposed to userspace, so we can't remove 
them at this point."

or something similar.

> userspace. Besides that +they make it unecessarily more complicated for
> userspace to figure out which +connections between a CRTC and a connector,

I think you're missing a verb. s/which connections/which connections are 
possible/ ?

> and what kind of cloning is +supported, they serve no purpose in the
> userspace API. Futhermore the exposed +restrictions are often wrongly set
> by drivers, and in many cases not powerful +enough to express the real
> restrictions.

I'd move the "But" sentence here, and possible start a new paragraph.

> A CRTC can be connected to multiple +Encoders, but for an

s/but/and/

> active CRTC there must be at least one encoder.
> +
> +The final, and real, endpoint in the display chain is the connector
> (represented +by :c:type:`struct drm_connector <drm_connector>`, see
> `Connector +Abstraction`_). Connectors can have different possible
> encoders, but the kernel +driver does this selection.

s/but/and/
s/does this selection/selects which encoder to use for each connector/ ?

> The use case is DVI,
> which could switch between an +analog and a digital encoder. There is only
> ever one active connector for any +encoder.

Isn't it the other way around, a single encoder for any connector ?

> +Internally the output pipeline is a bit more complex and matches todays

s/todays/today's/

> hardware +more closely:
> +
> +.. kernel-render:: DOT
> +   :alt: KMS Output Pipeline
> +   :caption: KMS Output Pipeline
> +
> +   digraph "Output Pipeline" {
> +      node [shape=box]
> +
> +      subgraph {
> +          "drm_crtc" [bgcolor=grey style=filled]
> +      }
> +
> +      subgraph cluster_internal {
> +          style=dashed
> +          label="Internal Pipeline"
> +          {
> +              node [bgcolor=grey style=filled]
> +              "drm_encoder A";
> +              "drm_encoder B";
> +              "drm_encoder C";
> +          }
> +
> +          {
> +              node [bgcolor=grey style=filled]
> +              "drm_encoder B" -> "drm_bridge B"
> +              "drm_encoder C" -> "drm_bridge C1"
> +              "drm_bridge C1" -> "drm_bridge C2";
> +          }
> +      }
> +
> +      "drm_crtc" -> "drm_encoder A"
> +      "drm_crtc" -> "drm_encoder B"
> +      "drm_crtc" -> "drm_encoder C"
> +
> +
> +      subgraph cluster_output {
> +          style=dashed
> +          label="Outputs"
> +
> +          "drm_encoder A" -> "drm_connector A";
> +          "drm_bridge B" -> "drm_connector B";
> +          "drm_bridge C2" -> "drm_connector C";
> +
> +          "drm_panel"
> +      }
> +   }
> +
> +Internally two additional helper objects come into play. First, to be able
> to +share code for encoders (sometimes on the same SoC, sometimes off-chip)
> one or +more :ref:`drm_bridges` (represented by :c:type:`struct drm_bridge
> +<drm_bridge>`) can be linked to an encoder. This link is static and cannot
> be +changed, which means the cross-bar (if there is any) needs to be mapped
> between +the CRTC and any encoders. Often for drivers with bridges there's
> no code left +at the encoder level. Atomic drivers can leave out all the
> encoder callbacks to +essentially only leave a dummy routing object behind,
> which is needed for +backwards compatibility since encoders are exposed to

I would have written "backward compatibility" but after checking I'm not too 
sure anymore. Documentation/ contains the same number of occurrences of each 
(at least after this patch, so it's a win for "backward compatibility" with a 
very thin margin in current mainline :-)).

> userspace.
> +
> +The second objects are panels, represented by :c:type:`struct drm_panel

Using a plural here with "second" seems strange to me but I might be wrong.

> +<drm_panel>`, see :ref:`drm_panel_helper`. Panels do not have a fixed
> binding +point, but are generally linked to the driver private structure
> which embeds +:c:type:`struct drm_connector <drm_connector>`.

s/which/that/

> +
> +Note that currently the bridge chaining and interactions with connectors
> and +panels are still in-flux and not really fully sorted out yet.
> +
>  KMS Core Structures and Functions
>  =================================

-- 
Regards,

Laurent Pinchart

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 3/6] drm/doc: Add KMS overview graphs

[Daniel Vetter]

On Thu, Mar 02, 2017 at 04:34:18PM +0200, Laurent Pinchart wrote:
> Hi Daniel,
> 
> Thank you for the patch.
> 
> On Tuesday 28 Feb 2017 18:13:16 Daniel Vetter wrote:
> > Oh, the shiny and pretties!
> > 
> > Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
> 
> Overall this looks good to me, please see below for a few minor issues.
> 
> > ---
> >  Documentation/gpu/drm-kms-helpers.rst |   4 ++
> >  Documentation/gpu/drm-kms.rst         | 132 +++++++++++++++++++++++++++++++
> >  2 files changed, 136 insertions(+)
> > 
> > diff --git a/Documentation/gpu/drm-kms-helpers.rst
> > b/Documentation/gpu/drm-kms-helpers.rst index 03040aa14fe8..012b6ff3ec3f
> > 100644
> > --- a/Documentation/gpu/drm-kms-helpers.rst
> > +++ b/Documentation/gpu/drm-kms-helpers.rst
> > @@ -114,6 +114,8 @@ Framebuffer CMA Helper Functions Reference
> >  .. kernel-doc:: drivers/gpu/drm/drm_fb_cma_helper.c
> > 
> >     :export:
> > +.. _drm_bridges:
> > +
> >  Bridges
> >  =======
> > 
> > @@ -139,6 +141,8 @@ Bridge Helper Reference
> >  .. kernel-doc:: drivers/gpu/drm/drm_bridge.c
> > 
> >     :export:
> > +.. _drm_panel_helper:
> > +
> >  Panel Helper Reference
> >  ======================
> > 
> > diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
> > index 4d4068855ec4..87d8162c9a34 100644
> > --- a/Documentation/gpu/drm-kms.rst
> > +++ b/Documentation/gpu/drm-kms.rst
> > @@ -17,6 +17,138 @@ be setup by initializing the following fields.
> > 
> >  Mode Configuration
> 
> Not part of this patch, but this line doesn't feel like it's where it shoul 
> dbe.

Yeah that's an accident from a previous patch, I've removed it.
> 
> > +Overview
> > +========
> > +
> > +.. kernel-render:: DOT
> > +   :alt: KMS Display Pipeline
> > +   :caption: KMS Display Pipeline Overview
> > +
> > +   digraph "KMS" {
> > +      node [shape=box]
> > +
> > +      subgraph cluster_static {
> > +          style=dashed
> > +          label="Static Objects"
> > +
> > +          node [bgcolor=grey style=filled]
> > +          "drm_plane A" -> "drm_crtc"
> > +          "drm_plane B" -> "drm_crtc"
> > +          "drm_crtc" -> "drm_encoder A"
> > +          "drm_crtc" -> "drm_encoder B"
> > +      }
> > +
> > +      subgraph cluster_user_created {
> > +          style=dashed
> > +          label="Userspace-Created"
> > +
> > +          node [shape=oval]
> > +          "drm_framebuffer 1" -> "drm_plane A"
> > +          "drm_framebuffer 2" -> "drm_plane B"
> > +      }
> > +
> > +      subgraph cluster_connector {
> > +          style=dashed
> > +          label="Hotpluggable"
> > +
> > +          "drm_encoder A" -> "drm_connector A"
> > +          "drm_encoder B" -> "drm_connector B"
> > +      }
> > +   }
> > +
> > +The basic object structure KMS presents to userspace is fairly simple.
> > +Framebuffers (represented by :c:type:`struct drm_framebuffer
> > <drm_framebuffer>`, +see `Frame Buffer Abstraction`_) feed into planes.
> > Multiple (or just one, or +even no) planes
> 
> I'd say "One or multiple (or even no) planes", but that's up to you.
> 
> > feed their pixel data into a
> > CRTC (represented by +:c:type:`struct drm_crtc <drm_crtc>`, see `CRTC
> > Abstraction`_) for blending. The +precise blending step is explained in
> > more detail in `Plane Composition +Properties`_ and related chapter.
> 
> s/chapter/chapters/ ? Or /related chapter/the related chapter/ ?
> 
> > +
> > +For the output routing the first step are Encoders (represented by
> 
> s/are/is/
> 
> > +:c:type:`struct drm_encoder <drm_encoder>`, see `Encoder Abstraction`_).
> > Those +are really just internal artifacts of the helper libraries used to
> > implement KMS +drivers. But unfortunately encoders have been exposed to
> 
> s/But u/U/
> 
> (http://blog.oxforddictionaries.com/2012/01/can-i-start-a-sentence-with-a-conjunction/)
> 
> I'd actually move the sentence towards the end of the paragraph and modify it 
> to
> 
> "Unfortunately connectors have been exposed to userspace, so we can't remove 
> them at this point."
> 
> or something similar.
> 
> > userspace. Besides that +they make it unecessarily more complicated for
> > userspace to figure out which +connections between a CRTC and a connector,
> 
> I think you're missing a verb. s/which connections/which connections are 
> possible/ ?
> 
> > and what kind of cloning is +supported, they serve no purpose in the
> > userspace API. Futhermore the exposed +restrictions are often wrongly set
> > by drivers, and in many cases not powerful +enough to express the real
> > restrictions.
> 
> I'd move the "But" sentence here, and possible start a new paragraph.
> 
> > A CRTC can be connected to multiple +Encoders, but for an
> 
> s/but/and/
> 
> > active CRTC there must be at least one encoder.
> > +
> > +The final, and real, endpoint in the display chain is the connector
> > (represented +by :c:type:`struct drm_connector <drm_connector>`, see
> > `Connector +Abstraction`_). Connectors can have different possible
> > encoders, but the kernel +driver does this selection.
> 
> s/but/and/
> s/does this selection/selects which encoder to use for each connector/ ?
> 
> > The use case is DVI,
> > which could switch between an +analog and a digital encoder. There is only
> > ever one active connector for any +encoder.
> 
> Isn't it the other way around, a single encoder for any connector ?

For each active connector theres exactly one active encoder. The possible
linking is n:m. I've clarified this.

All other suggestions applied, thanks.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 4/6] drm/doc: Consistent kerneldoc include order

[Daniel Vetter]

First overview text (if there is any), then headers (since generally
you want to start out with the data structures), then all the other
stuff with functions.

Most of this is pre-shpinx, since with the old docbook only the
overview stuff was pulled in directly. Everything else was put in a
per-section index, so include order didn't really matter.

Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
---
 Documentation/gpu/drm-internals.rst   |  6 +++---
 Documentation/gpu/drm-kms-helpers.rst | 22 +++++++++++-----------
 Documentation/gpu/drm-kms.rst         | 24 ++++++++++++------------
 Documentation/gpu/drm-mm.rst          | 24 ++++++++++++------------
 4 files changed, 38 insertions(+), 38 deletions(-)

diff --git a/Documentation/gpu/drm-internals.rst b/Documentation/gpu/drm-internals.rst
index e35920db1f4c..29d6bf7bb1ac 100644
--- a/Documentation/gpu/drm-internals.rst
+++ b/Documentation/gpu/drm-internals.rst
@@ -140,12 +140,12 @@ Device Instance and Driver Handling
 .. kernel-doc:: drivers/gpu/drm/drm_drv.c
    :doc: driver instance overview
 
-.. kernel-doc:: drivers/gpu/drm/drm_drv.c
-   :export:
-
 .. kernel-doc:: include/drm/drm_drv.h
    :internal:
 
+.. kernel-doc:: drivers/gpu/drm/drm_drv.c
+   :export:
+
 Driver Load
 -----------
 
diff --git a/Documentation/gpu/drm-kms-helpers.rst b/Documentation/gpu/drm-kms-helpers.rst
index 012b6ff3ec3f..050ebe81d256 100644
--- a/Documentation/gpu/drm-kms-helpers.rst
+++ b/Documentation/gpu/drm-kms-helpers.rst
@@ -37,10 +37,10 @@ Modeset Helper Reference for Common Vtables
 ===========================================
 
 .. kernel-doc:: include/drm/drm_modeset_helper_vtables.h
-   :internal:
+   :doc: overview
 
 .. kernel-doc:: include/drm/drm_modeset_helper_vtables.h
-   :doc: overview
+   :internal:
 
 Atomic Modeset Helper Functions Reference
 =========================================
@@ -84,27 +84,27 @@ Legacy CRTC/Modeset Helper Functions Reference
 Simple KMS Helper Reference
 ===========================
 
+.. kernel-doc:: drivers/gpu/drm/drm_simple_kms_helper.c
+   :doc: overview
+
 .. kernel-doc:: include/drm/drm_simple_kms_helper.h
    :internal:
 
 .. kernel-doc:: drivers/gpu/drm/drm_simple_kms_helper.c
    :export:
 
-.. kernel-doc:: drivers/gpu/drm/drm_simple_kms_helper.c
-   :doc: overview
-
 fbdev Helper Functions Reference
 ================================
 
 .. kernel-doc:: drivers/gpu/drm/drm_fb_helper.c
    :doc: fbdev helpers
 
-.. kernel-doc:: drivers/gpu/drm/drm_fb_helper.c
-   :export:
-
 .. kernel-doc:: include/drm/drm_fb_helper.h
    :internal:
 
+.. kernel-doc:: drivers/gpu/drm/drm_fb_helper.c
+   :export:
+
 Framebuffer CMA Helper Functions Reference
 ==========================================
 
@@ -146,15 +146,15 @@ Bridge Helper Reference
 Panel Helper Reference
 ======================
 
+.. kernel-doc:: drivers/gpu/drm/drm_panel.c
+   :doc: drm panel
+
 .. kernel-doc:: include/drm/drm_panel.h
    :internal:
 
 .. kernel-doc:: drivers/gpu/drm/drm_panel.c
    :export:
 
-.. kernel-doc:: drivers/gpu/drm/drm_panel.c
-   :doc: drm panel
-
 Display Port Helper Functions Reference
 =======================================
 
diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
index 87d8162c9a34..17a4cd5b14fd 100644
--- a/Documentation/gpu/drm-kms.rst
+++ b/Documentation/gpu/drm-kms.rst
@@ -152,12 +152,12 @@ panels are still in-flux and not really fully sorted out yet.
 KMS Core Structures and Functions
 =================================
 
-.. kernel-doc:: drivers/gpu/drm/drm_mode_config.c
-   :export:
-
 .. kernel-doc:: include/drm/drm_mode_config.h
    :internal:
 
+.. kernel-doc:: drivers/gpu/drm/drm_mode_config.c
+   :export:
+
 Modeset Base Object Abstraction
 ===============================
 
@@ -170,12 +170,12 @@ Modeset Base Object Abstraction
 Atomic Mode Setting Function Reference
 ======================================
 
-.. kernel-doc:: drivers/gpu/drm/drm_atomic.c
-   :export:
-
 .. kernel-doc:: include/drm/drm_atomic.h
    :internal:
 
+.. kernel-doc:: drivers/gpu/drm/drm_atomic.c
+   :export:
+
 CRTC Abstraction
 ================
 
@@ -200,12 +200,12 @@ Frame Buffer Abstraction
 Frame Buffer Functions Reference
 --------------------------------
 
-.. kernel-doc:: drivers/gpu/drm/drm_framebuffer.c
-   :export:
-
 .. kernel-doc:: include/drm/drm_framebuffer.h
    :internal:
 
+.. kernel-doc:: drivers/gpu/drm/drm_framebuffer.c
+   :export:
+
 DRM Format Handling
 ===================
 
@@ -508,8 +508,8 @@ operation handler.
 Vertical Blanking and Interrupt Handling Functions Reference
 ------------------------------------------------------------
 
-.. kernel-doc:: drivers/gpu/drm/drm_irq.c
-   :export:
-
 .. kernel-doc:: include/drm/drm_irq.h
    :internal:
+
+.. kernel-doc:: drivers/gpu/drm/drm_irq.c
+   :export:
diff --git a/Documentation/gpu/drm-mm.rst b/Documentation/gpu/drm-mm.rst
index f5760b140f13..257b0197bc2e 100644
--- a/Documentation/gpu/drm-mm.rst
+++ b/Documentation/gpu/drm-mm.rst
@@ -367,36 +367,36 @@ from the client in libdrm.
 GEM Function Reference
 ----------------------
 
-.. kernel-doc:: drivers/gpu/drm/drm_gem.c
-   :export:
-
 .. kernel-doc:: include/drm/drm_gem.h
    :internal:
 
+.. kernel-doc:: drivers/gpu/drm/drm_gem.c
+   :export:
+
 GEM CMA Helper Functions Reference
 ----------------------------------
 
 .. kernel-doc:: drivers/gpu/drm/drm_gem_cma_helper.c
    :doc: cma helpers
 
-.. kernel-doc:: drivers/gpu/drm/drm_gem_cma_helper.c
-   :export:
-
 .. kernel-doc:: include/drm/drm_gem_cma_helper.h
    :internal:
 
+.. kernel-doc:: drivers/gpu/drm/drm_gem_cma_helper.c
+   :export:
+
 VMA Offset Manager
 ==================
 
 .. kernel-doc:: drivers/gpu/drm/drm_vma_manager.c
    :doc: vma offset manager
 
-.. kernel-doc:: drivers/gpu/drm/drm_vma_manager.c
-   :export:
-
 .. kernel-doc:: include/drm/drm_vma_manager.h
    :internal:
 
+.. kernel-doc:: drivers/gpu/drm/drm_vma_manager.c
+   :export:
+
 PRIME Buffer Sharing
 ====================
 
@@ -472,12 +472,12 @@ LRU Scan/Eviction Support
 DRM MM Range Allocator Function References
 ------------------------------------------
 
-.. kernel-doc:: drivers/gpu/drm/drm_mm.c
-   :export:
-
 .. kernel-doc:: include/drm/drm_mm.h
    :internal:
 
+.. kernel-doc:: drivers/gpu/drm/drm_mm.c
+   :export:
+
 DRM Cache Handling
 ==================
 
-- 
2.11.0

Re: [PATCH 4/6] drm/doc: Consistent kerneldoc include order

[Eric Anholt]

[-- Attachment #1.1: Type: text/plain, Size: 291 bytes --]

Daniel Vetter <daniel.vetter@ffwll.ch> writes:

> First overview text (if there is any), then headers (since generally
> you want to start out with the data structures), then all the other
> stuff with functions.
>
> Most of this is pre-shpinx, since with the old docbook only the

"sphinx"

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 5/6] drm/doc: diagram for mode objects and properties

[Daniel Vetter]

Resulted in confusion a few times in the past.

Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Manasi Navare <manasi.d.navare@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 Documentation/gpu/drm-kms.rst | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
index 17a4cd5b14fd..20378881445f 100644
--- a/Documentation/gpu/drm-kms.rst
+++ b/Documentation/gpu/drm-kms.rst
@@ -161,6 +161,28 @@ KMS Core Structures and Functions
 Modeset Base Object Abstraction
 ===============================
 
+.. kernel-render:: DOT
+   :alt: Mode Objects and Properties
+   :caption: Mode Objects and Properties
+
+   digraph {
+      node [shape=box]
+
+      "drm_property A" -> "drm_mode_object A"
+      "drm_property A" -> "drm_mode_object B"
+      "drm_property B" -> "drm_mode_object A"
+   }
+
+The base structure for all KMS objects is :c:type:`struct drm_mode_object
+<drm_mode_object>`. One of the base services is provides is tracking properties,
+which are especially important for the atomic IOCTL (see `Atomic Mode
+Setting`_). The somewhat surprising part here is that properties are not
+directly instantiated on each object, but free-standing mode objects themselves,
+represented by :c:type:`struct drm_property <drm_property>`, which only specify
+the type and value range of a property. Any given property can be attached
+multiple times to different objects using :c:func:`drm_object_attach_property()
+<drm_object_attach_property>`.
+
 .. kernel-doc:: include/drm/drm_mode_object.h
    :internal:
 
-- 
2.11.0

Re: [PATCH 5/6] drm/doc: diagram for mode objects and properties

[Eric Anholt]

[-- Attachment #1.1: Type: text/plain, Size: 1257 bytes --]

Daniel Vetter <daniel.vetter@ffwll.ch> writes:

> Resulted in confusion a few times in the past.
>
> Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Cc: Manasi Navare <manasi.d.navare@intel.com>
> Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
> ---
>  Documentation/gpu/drm-kms.rst | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
>
> diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
> index 17a4cd5b14fd..20378881445f 100644
> --- a/Documentation/gpu/drm-kms.rst
> +++ b/Documentation/gpu/drm-kms.rst

> +The base structure for all KMS objects is :c:type:`struct drm_mode_object
> +<drm_mode_object>`. One of the base services is provides is tracking properties,

"it provides"?

> +which are especially important for the atomic IOCTL (see `Atomic Mode
> +Setting`_). The somewhat surprising part here is that properties are not
> +directly instantiated on each object, but free-standing mode objects themselves,
> +represented by :c:type:`struct drm_property <drm_property>`, which only specify
> +the type and value range of a property. Any given property can be attached
> +multiple times to different objects using :c:func:`drm_object_attach_property()
> +<drm_object_attach_property>`.
> +


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH] drm/doc: diagram for mode objects and properties

[Daniel Vetter]

Resulted in confusion a few times in the past.

v2: Spelling fix (Eric).

Cc: Eric Anholt <eric@anholt.net>
Acked-by: Eric Anholt <eric@anholt.net>
Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Manasi Navare <manasi.d.navare@intel.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
---
 Documentation/gpu/drm-kms.rst | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
index 17a4cd5b14fd..a504d9ee4d94 100644
--- a/Documentation/gpu/drm-kms.rst
+++ b/Documentation/gpu/drm-kms.rst
@@ -161,6 +161,28 @@ KMS Core Structures and Functions
 Modeset Base Object Abstraction
 ===============================
 
+.. kernel-render:: DOT
+   :alt: Mode Objects and Properties
+   :caption: Mode Objects and Properties
+
+   digraph {
+      node [shape=box]
+
+      "drm_property A" -> "drm_mode_object A"
+      "drm_property A" -> "drm_mode_object B"
+      "drm_property B" -> "drm_mode_object A"
+   }
+
+The base structure for all KMS objects is :c:type:`struct drm_mode_object
+<drm_mode_object>`. One of the base services it provides is tracking properties,
+which are especially important for the atomic IOCTL (see `Atomic Mode
+Setting`_). The somewhat surprising part here is that properties are not
+directly instantiated on each object, but free-standing mode objects themselves,
+represented by :c:type:`struct drm_property <drm_property>`, which only specify
+the type and value range of a property. Any given property can be attached
+multiple times to different objects using :c:func:`drm_object_attach_property()
+<drm_object_attach_property>`.
+
 .. kernel-doc:: include/drm/drm_mode_object.h
    :internal:
 
-- 
2.11.0

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH] drm/doc: diagram for mode objects and properties

[Gabriel Krisman Bertazi]

Daniel Vetter <daniel.vetter@ffwll.ch> writes:

> Resulted in confusion a few times in the past.
>

Reviewed-by: Gabriel Krisman Bertazi <krisman@collabora.co.uk>

-- 
Gabriel Krisman Bertazi
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH] drm/doc: diagram for mode objects and properties

[Laurent Pinchart]

Hi Daniel,

Thank you for the patch.

On Wednesday 01 Mar 2017 09:27:14 Daniel Vetter wrote:
> Resulted in confusion a few times in the past.
> 
> v2: Spelling fix (Eric).
> 
> Cc: Eric Anholt <eric@anholt.net>
> Acked-by: Eric Anholt <eric@anholt.net>
> Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Cc: Manasi Navare <manasi.d.navare@intel.com>
> Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  Documentation/gpu/drm-kms.rst | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
> 
> diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
> index 17a4cd5b14fd..a504d9ee4d94 100644
> --- a/Documentation/gpu/drm-kms.rst
> +++ b/Documentation/gpu/drm-kms.rst
> @@ -161,6 +161,28 @@ KMS Core Structures and Functions
>  Modeset Base Object Abstraction
>  ===============================
> 
> +.. kernel-render:: DOT
> +   :alt: Mode Objects and Properties
> +   :caption: Mode Objects and Properties
> +
> +   digraph {
> +      node [shape=box]
> +
> +      "drm_property A" -> "drm_mode_object A"
> +      "drm_property A" -> "drm_mode_object B"
> +      "drm_property B" -> "drm_mode_object A"
> +   }
> +
> +The base structure for all KMS objects is :c:type:`struct drm_mode_object
> +<drm_mode_object>`. One of the base services it provides is tracking
> properties, +which are especially important for the atomic IOCTL (see
> `Atomic Mode +Setting`_). The somewhat surprising part here is that
> properties are not +directly instantiated on each object, but free-standing
> mode objects themselves, +represented by :c:type:`struct drm_property
> <drm_property>`, which only specify +the type and value range of a
> property. Any given property can be attached +multiple times to different
> objects using :c:func:`drm_object_attach_property()
> +<drm_object_attach_property>`.
> +
>  .. kernel-doc:: include/drm/drm_mode_object.h
> 
>     :internal:

-- 
Regards,

Laurent Pinchart

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH 6/6] drm/doc: atomic overview, with graph

[Daniel Vetter]

I want to split up a few more things and document some details better
(like how exactly to subclass drm_atomic_state). And maybe also split
up the helpers a bit per-topic, but this should be a ok-ish start for
better atomic overview.

One thing I failed at is getting DOT to layout the overview graph how
I want it. The highlevel structure I want is:

	Free-standing State

	Current State

i.e. one over the other. Currently it lays it out side-by-side, but
not even that really - "Current State" is somewhat offset below. Makes
the graph look like garbage, and also way too wide for proper
rendering. Ideas appreciated.

Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
---
 Documentation/gpu/drm-kms-helpers.rst |  2 +
 Documentation/gpu/drm-kms.rst         | 85 ++++++++++++++++++++++++++++++++++-
 2 files changed, 86 insertions(+), 1 deletion(-)

diff --git a/Documentation/gpu/drm-kms-helpers.rst b/Documentation/gpu/drm-kms-helpers.rst
index 050ebe81d256..ac53c0b893f6 100644
--- a/Documentation/gpu/drm-kms-helpers.rst
+++ b/Documentation/gpu/drm-kms-helpers.rst
@@ -42,6 +42,8 @@ Modeset Helper Reference for Common Vtables
 .. kernel-doc:: include/drm/drm_modeset_helper_vtables.h
    :internal:
 
+.. _drm_atomic_helper:
+
 Atomic Modeset Helper Functions Reference
 =========================================
 
diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
index 20378881445f..979cee853bb1 100644
--- a/Documentation/gpu/drm-kms.rst
+++ b/Documentation/gpu/drm-kms.rst
@@ -189,8 +189,91 @@ multiple times to different objects using :c:func:`drm_object_attach_property()
 .. kernel-doc:: drivers/gpu/drm/drm_mode_object.c
    :export:
 
+Atomic Mode Setting
+===================
+
+
+.. FIXME: The I want the below graph to be laid out so that the 2 subgraph
+   clusters are below each another. But I failed.
+
+.. kernel-render:: DOT
+   :alt: Mode Objects and Properties
+   :caption: Mode Objects and Properties
+
+   digraph {
+      node [shape=box]
+
+      subgraph cluster_state {
+          style=dashed
+          label="Free-standing state"
+
+          "drm_atomic_state" -> "duplicated drm_plane_state A"
+          "drm_atomic_state" -> "duplicated drm_plane_state B"
+          "drm_atomic_state" -> "duplicated drm_crtc_state"
+          "drm_atomic_state" -> "duplicated drm_connector_state"
+          "drm_atomic_state" -> "duplicated driver private state"
+      }
+
+      subgraph cluster_current {
+          style=dashed
+          label="Current state"
+
+          "drm_device" -> "drm_plane A"
+          "drm_device" -> "drm_plane B"
+          "drm_device" -> "drm_crtc"
+          "drm_device" -> "drm_connector"
+          "drm_device" -> "driver private object"
+
+          "drm_plane A" -> "drm_plane_state A"
+          "drm_plane B" -> "drm_plane_state B"
+          "drm_crtc" -> "drm_crtc_state"
+          "drm_connector" -> "drm_connector_state"
+          "driver private object" -> "driver private state"
+      }
+
+      "drm_atomic_state" -> "drm_device" [label="atomic_commit"]
+   }
+
+Essentially atomic is transactional modeset (including planes) updates, but
+compared to the usual transactional approach of try-commit and rollback on
+failure atomic modesetting is a bit different:
+
+- Firstly, no hardware changes are allowed when the commit would fail. This
+  allows us to implement the DRM_MODE_ATOMIC_TEST_ONLY mode, which allows
+  userspace to explore whether certain configurations would work or not.
+
+- This would still allows setting and rollback of just the software state,
+  simplifying conversion of existing drivers. But auditing drivers for
+  correctness of the atomic_check code because really hard with that.
+
+- Lastly, for backwards compatibility and to support all use-cases, atomic
+  updates need to be incremental and be able to execute in parallel. Hardware
+  doesn't always allow it, but where possible plane updates on different CRTCs
+  should not interfere, and not get stalled due to output routing changing on
+  different CRTCs.
+
+Taken all together there's two consequence for the atomic design:
+
+- The overall state is split up into per-object state structures:
+  :c:type:`struct drm_plane_state <drm_plane_state>` for planes, :c:type:`struct
+  drm_crtc_state <drm_crtc_state>` for CRTCs and :c:type:`struct
+  drm_connector_state <drm_connector_state` for connectors. These are the only
+  objects with userspace-visible and settable state. For internal state drivers
+  can subclass these structures through embeddeding, or add entirely new state
+  structures for their globally shared hardware functions.
+
+- An atomic update is assembled and validated as an enterily free-standing pile
+  of structures within the :c:type:`drm_atomic_state <drm_atomic_state>`
+  container. Again drivers can subclass that container for their own state
+  structure tracking needs. Only when a state is commit is it applied to the
+  driver and modeset objects. This way rolling back an update boils down to
+  releasing memory and unreference objects like framebuffers.
+
+Read on in this chapter, and also in :ref:`drm_atomic_helper` for more detailed
+coverage of specific topics.
+
 Atomic Mode Setting Function Reference
-======================================
+--------------------------------------
 
 .. kernel-doc:: include/drm/drm_atomic.h
    :internal:
-- 
2.11.0

Re: [PATCH 6/6] drm/doc: atomic overview, with graph

[Eric Anholt]

[-- Attachment #1.1: Type: text/plain, Size: 5855 bytes --]

Daniel Vetter <daniel.vetter@ffwll.ch> writes:

> I want to split up a few more things and document some details better
> (like how exactly to subclass drm_atomic_state). And maybe also split
> up the helpers a bit per-topic, but this should be a ok-ish start for
> better atomic overview.
>
> One thing I failed at is getting DOT to layout the overview graph how
> I want it. The highlevel structure I want is:
>
> 	Free-standing State
>
> 	Current State
>
> i.e. one over the other. Currently it lays it out side-by-side, but
> not even that really - "Current State" is somewhat offset below. Makes
> the graph look like garbage, and also way too wide for proper
> rendering. Ideas appreciated.
>
> Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Cc: Harry Wentland <harry.wentland@amd.com>
> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>

Thanks for writing these docs.  I wish I had them back when I was
starting vc4's atomic code!  With the two little spelling nits fixed,
3-5 are:

Acked-by: Eric Anholt <eric@anholt.net>

A few copyedits on this one below, but it sounds like you don't want to
push quite yet while you sort out the rendering.

> ---
>  Documentation/gpu/drm-kms-helpers.rst |  2 +
>  Documentation/gpu/drm-kms.rst         | 85 ++++++++++++++++++++++++++++++++++-
>  2 files changed, 86 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/gpu/drm-kms-helpers.rst b/Documentation/gpu/drm-kms-helpers.rst
> index 050ebe81d256..ac53c0b893f6 100644
> --- a/Documentation/gpu/drm-kms-helpers.rst
> +++ b/Documentation/gpu/drm-kms-helpers.rst
> @@ -42,6 +42,8 @@ Modeset Helper Reference for Common Vtables
>  .. kernel-doc:: include/drm/drm_modeset_helper_vtables.h
>     :internal:
>  
> +.. _drm_atomic_helper:
> +
>  Atomic Modeset Helper Functions Reference
>  =========================================
>  
> diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
> index 20378881445f..979cee853bb1 100644
> --- a/Documentation/gpu/drm-kms.rst
> +++ b/Documentation/gpu/drm-kms.rst
> @@ -189,8 +189,91 @@ multiple times to different objects using :c:func:`drm_object_attach_property()
>  .. kernel-doc:: drivers/gpu/drm/drm_mode_object.c
>     :export:
>  
> +Atomic Mode Setting
> +===================
> +
> +
> +.. FIXME: The I want the below graph to be laid out so that the 2 subgraph
> +   clusters are below each another. But I failed.
> +
> +.. kernel-render:: DOT
> +   :alt: Mode Objects and Properties
> +   :caption: Mode Objects and Properties
> +
> +   digraph {
> +      node [shape=box]
> +
> +      subgraph cluster_state {
> +          style=dashed
> +          label="Free-standing state"
> +
> +          "drm_atomic_state" -> "duplicated drm_plane_state A"
> +          "drm_atomic_state" -> "duplicated drm_plane_state B"
> +          "drm_atomic_state" -> "duplicated drm_crtc_state"
> +          "drm_atomic_state" -> "duplicated drm_connector_state"
> +          "drm_atomic_state" -> "duplicated driver private state"
> +      }
> +
> +      subgraph cluster_current {
> +          style=dashed
> +          label="Current state"
> +
> +          "drm_device" -> "drm_plane A"
> +          "drm_device" -> "drm_plane B"
> +          "drm_device" -> "drm_crtc"
> +          "drm_device" -> "drm_connector"
> +          "drm_device" -> "driver private object"
> +
> +          "drm_plane A" -> "drm_plane_state A"
> +          "drm_plane B" -> "drm_plane_state B"
> +          "drm_crtc" -> "drm_crtc_state"
> +          "drm_connector" -> "drm_connector_state"
> +          "driver private object" -> "driver private state"
> +      }
> +
> +      "drm_atomic_state" -> "drm_device" [label="atomic_commit"]
> +   }
> +
> +Essentially atomic is transactional modeset (including planes) updates, but
> +compared to the usual transactional approach of try-commit and rollback on
> +failure atomic modesetting is a bit different:

Maybe reword:

"Atomic provides transactional modeset (including planes) updates, but a
bit differently from the usual transactional approach of try-commit and
rollback:"

> +- Firstly, no hardware changes are allowed when the commit would fail. This
> +  allows us to implement the DRM_MODE_ATOMIC_TEST_ONLY mode, which allows
> +  userspace to explore whether certain configurations would work or not.
> +
> +- This would still allows setting and rollback of just the software state,

"allow"

> +  simplifying conversion of existing drivers. But auditing drivers for
> +  correctness of the atomic_check code because really hard with that.

s/because/becomes/?

> +Taken all together there's two consequence for the atomic design:

"consequences"

> +
> +- The overall state is split up into per-object state structures:
> +  :c:type:`struct drm_plane_state <drm_plane_state>` for planes, :c:type:`struct
> +  drm_crtc_state <drm_crtc_state>` for CRTCs and :c:type:`struct
> +  drm_connector_state <drm_connector_state` for connectors. These are the only
> +  objects with userspace-visible and settable state. For internal state drivers
> +  can subclass these structures through embeddeding, or add entirely new state
> +  structures for their globally shared hardware functions.
> +
> +- An atomic update is assembled and validated as an enterily free-standing pile
> +  of structures within the :c:type:`drm_atomic_state <drm_atomic_state>`
> +  container. Again drivers can subclass that container for their own state
> +  structure tracking needs. Only when a state is commit is it applied to the

"is committed"

> +  driver and modeset objects. This way rolling back an update boils down to
> +  releasing memory and unreference objects like framebuffers.

"unreferencing"

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 6/6] drm/doc: atomic overview, with graph

[Daniel Vetter]

On Tue, Feb 28, 2017 at 03:48:47PM -0800, Eric Anholt wrote:
> Daniel Vetter <daniel.vetter@ffwll.ch> writes:
> 
> > I want to split up a few more things and document some details better
> > (like how exactly to subclass drm_atomic_state). And maybe also split
> > up the helpers a bit per-topic, but this should be a ok-ish start for
> > better atomic overview.
> >
> > One thing I failed at is getting DOT to layout the overview graph how
> > I want it. The highlevel structure I want is:
> >
> > 	Free-standing State
> >
> > 	Current State
> >
> > i.e. one over the other. Currently it lays it out side-by-side, but
> > not even that really - "Current State" is somewhat offset below. Makes
> > the graph look like garbage, and also way too wide for proper
> > rendering. Ideas appreciated.
> >
> > Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Cc: Harry Wentland <harry.wentland@amd.com>
> > Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
> 
> Thanks for writing these docs.  I wish I had them back when I was
> starting vc4's atomic code!  With the two little spelling nits fixed,
> 3-5 are:
> 
> Acked-by: Eric Anholt <eric@anholt.net>
> 
> A few copyedits on this one below, but it sounds like you don't want to
> push quite yet while you sort out the rendering.

I've spent quite some time trying to beat DOT into submission, this is the
best I can do. The FIXME really is just a hint for someone with more clue
to maybe make it better, or if not possible at all, what would look better
when doing a proper diagram with .svg or something like that.

Assuming no one knows how to fix this, I'd still like to push it - it's
still better than nothing imo, you just need to look at the picture
full-screen.
-Daniel

> 
> > ---
> >  Documentation/gpu/drm-kms-helpers.rst |  2 +
> >  Documentation/gpu/drm-kms.rst         | 85 ++++++++++++++++++++++++++++++++++-
> >  2 files changed, 86 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/gpu/drm-kms-helpers.rst b/Documentation/gpu/drm-kms-helpers.rst
> > index 050ebe81d256..ac53c0b893f6 100644
> > --- a/Documentation/gpu/drm-kms-helpers.rst
> > +++ b/Documentation/gpu/drm-kms-helpers.rst
> > @@ -42,6 +42,8 @@ Modeset Helper Reference for Common Vtables
> >  .. kernel-doc:: include/drm/drm_modeset_helper_vtables.h
> >     :internal:
> >  
> > +.. _drm_atomic_helper:
> > +
> >  Atomic Modeset Helper Functions Reference
> >  =========================================
> >  
> > diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
> > index 20378881445f..979cee853bb1 100644
> > --- a/Documentation/gpu/drm-kms.rst
> > +++ b/Documentation/gpu/drm-kms.rst
> > @@ -189,8 +189,91 @@ multiple times to different objects using :c:func:`drm_object_attach_property()
> >  .. kernel-doc:: drivers/gpu/drm/drm_mode_object.c
> >     :export:
> >  
> > +Atomic Mode Setting
> > +===================
> > +
> > +
> > +.. FIXME: The I want the below graph to be laid out so that the 2 subgraph
> > +   clusters are below each another. But I failed.
> > +
> > +.. kernel-render:: DOT
> > +   :alt: Mode Objects and Properties
> > +   :caption: Mode Objects and Properties
> > +
> > +   digraph {
> > +      node [shape=box]
> > +
> > +      subgraph cluster_state {
> > +          style=dashed
> > +          label="Free-standing state"
> > +
> > +          "drm_atomic_state" -> "duplicated drm_plane_state A"
> > +          "drm_atomic_state" -> "duplicated drm_plane_state B"
> > +          "drm_atomic_state" -> "duplicated drm_crtc_state"
> > +          "drm_atomic_state" -> "duplicated drm_connector_state"
> > +          "drm_atomic_state" -> "duplicated driver private state"
> > +      }
> > +
> > +      subgraph cluster_current {
> > +          style=dashed
> > +          label="Current state"
> > +
> > +          "drm_device" -> "drm_plane A"
> > +          "drm_device" -> "drm_plane B"
> > +          "drm_device" -> "drm_crtc"
> > +          "drm_device" -> "drm_connector"
> > +          "drm_device" -> "driver private object"
> > +
> > +          "drm_plane A" -> "drm_plane_state A"
> > +          "drm_plane B" -> "drm_plane_state B"
> > +          "drm_crtc" -> "drm_crtc_state"
> > +          "drm_connector" -> "drm_connector_state"
> > +          "driver private object" -> "driver private state"
> > +      }
> > +
> > +      "drm_atomic_state" -> "drm_device" [label="atomic_commit"]
> > +   }
> > +
> > +Essentially atomic is transactional modeset (including planes) updates, but
> > +compared to the usual transactional approach of try-commit and rollback on
> > +failure atomic modesetting is a bit different:
> 
> Maybe reword:
> 
> "Atomic provides transactional modeset (including planes) updates, but a
> bit differently from the usual transactional approach of try-commit and
> rollback:"
> 
> > +- Firstly, no hardware changes are allowed when the commit would fail. This
> > +  allows us to implement the DRM_MODE_ATOMIC_TEST_ONLY mode, which allows
> > +  userspace to explore whether certain configurations would work or not.
> > +
> > +- This would still allows setting and rollback of just the software state,
> 
> "allow"
> 
> > +  simplifying conversion of existing drivers. But auditing drivers for
> > +  correctness of the atomic_check code because really hard with that.
> 
> s/because/becomes/?
> 
> > +Taken all together there's two consequence for the atomic design:
> 
> "consequences"
> 
> > +
> > +- The overall state is split up into per-object state structures:
> > +  :c:type:`struct drm_plane_state <drm_plane_state>` for planes, :c:type:`struct
> > +  drm_crtc_state <drm_crtc_state>` for CRTCs and :c:type:`struct
> > +  drm_connector_state <drm_connector_state` for connectors. These are the only
> > +  objects with userspace-visible and settable state. For internal state drivers
> > +  can subclass these structures through embeddeding, or add entirely new state
> > +  structures for their globally shared hardware functions.
> > +
> > +- An atomic update is assembled and validated as an enterily free-standing pile
> > +  of structures within the :c:type:`drm_atomic_state <drm_atomic_state>`
> > +  container. Again drivers can subclass that container for their own state
> > +  structure tracking needs. Only when a state is commit is it applied to the
> 
> "is committed"
> 
> > +  driver and modeset objects. This way rolling back an update boils down to
> > +  releasing memory and unreference objects like framebuffers.
> 
> "unreferencing"



-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 6/6] drm/doc: atomic overview, with graph

[Gabriel Krisman Bertazi]

Daniel Vetter <daniel@ffwll.ch> writes:

> I've spent quite some time trying to beat DOT into submission, this is the
> best I can do. The FIXME really is just a hint for someone with more clue
> to maybe make it better, or if not possible at all, what would look better
> when doing a proper diagram with .svg or something like that.
>
> Assuming no one knows how to fix this, I'd still like to push it - it's
> still better than nothing imo, you just need to look at the picture
> full-screen.

If you add a hidden edge from any of the "duplicated" nodes to the
drm_device node, the second cluster will be pushed to the third rank
and do what I think you want.

Something like:

 "duplicated drm_plane_state A" -> "drm_device"[style=invis]

This is the result for me:

https://people.collabora.com/~krisman/atomic_modesetting.svg

I think it got a little better.

-- 
Gabriel Krisman Bertazi
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH 6/6] drm/doc: atomic overview, with graph

[Daniel Vetter]

On Wed, Mar 01, 2017 at 02:42:02PM -0300, Gabriel Krisman Bertazi wrote:
> Daniel Vetter <daniel@ffwll.ch> writes:
> 
> > I've spent quite some time trying to beat DOT into submission, this is the
> > best I can do. The FIXME really is just a hint for someone with more clue
> > to maybe make it better, or if not possible at all, what would look better
> > when doing a proper diagram with .svg or something like that.
> >
> > Assuming no one knows how to fix this, I'd still like to push it - it's
> > still better than nothing imo, you just need to look at the picture
> > full-screen.
> 
> If you add a hidden edge from any of the "duplicated" nodes to the
> drm_device node, the second cluster will be pushed to the third rank
> and do what I think you want.
> 
> Something like:
> 
>  "duplicated drm_plane_state A" -> "drm_device"[style=invis]
> 
> This is the result for me:
> 
> https://people.collabora.com/~krisman/atomic_modesetting.svg
> 
> I think it got a little better.

Yeah, much better, thanks for the suggestion. Still rather small-ish, but
it's a busy graph, so that can't be helped really.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH] drm/doc: atomic overview, with graph

[Daniel Vetter]

I want to split up a few more things and document some details better
(like how exactly to subclass drm_atomic_state). And maybe also split
up the helpers a bit per-topic, but this should be a ok-ish start for
better atomic overview.

One thing I failed at is getting DOT to layout the overview graph how
I want it. The highlevel structure I want is:

	Free-standing State

	Current State

i.e. one over the other. Currently it lays it out side-by-side, but
not even that really - "Current State" is somewhat offset below. Makes
the graph look like garbage, and also way too wide for proper
rendering. Ideas appreciated.

v2: Spelling and clarifications (Eric).

Cc: Eric Anholt <eric@anholt.net>
Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
---
 Documentation/gpu/drm-kms-helpers.rst |  2 +
 Documentation/gpu/drm-kms.rst         | 86 ++++++++++++++++++++++++++++++++++-
 2 files changed, 87 insertions(+), 1 deletion(-)

diff --git a/Documentation/gpu/drm-kms-helpers.rst b/Documentation/gpu/drm-kms-helpers.rst
index 050ebe81d256..ac53c0b893f6 100644
--- a/Documentation/gpu/drm-kms-helpers.rst
+++ b/Documentation/gpu/drm-kms-helpers.rst
@@ -42,6 +42,8 @@ Modeset Helper Reference for Common Vtables
 .. kernel-doc:: include/drm/drm_modeset_helper_vtables.h
    :internal:
 
+.. _drm_atomic_helper:
+
 Atomic Modeset Helper Functions Reference
 =========================================
 
diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
index a504d9ee4d94..4823df03c773 100644
--- a/Documentation/gpu/drm-kms.rst
+++ b/Documentation/gpu/drm-kms.rst
@@ -189,8 +189,92 @@ multiple times to different objects using :c:func:`drm_object_attach_property()
 .. kernel-doc:: drivers/gpu/drm/drm_mode_object.c
    :export:
 
+Atomic Mode Setting
+===================
+
+
+.. FIXME: The I want the below graph to be laid out so that the 2 subgraph
+   clusters are below each another. But I failed.
+
+.. kernel-render:: DOT
+   :alt: Mode Objects and Properties
+   :caption: Mode Objects and Properties
+
+   digraph {
+      node [shape=box]
+
+      subgraph cluster_state {
+          style=dashed
+          label="Free-standing state"
+
+          "drm_atomic_state" -> "duplicated drm_plane_state A"
+          "drm_atomic_state" -> "duplicated drm_plane_state B"
+          "drm_atomic_state" -> "duplicated drm_crtc_state"
+          "drm_atomic_state" -> "duplicated drm_connector_state"
+          "drm_atomic_state" -> "duplicated driver private state"
+      }
+
+      subgraph cluster_current {
+          style=dashed
+          label="Current state"
+
+          "drm_device" -> "drm_plane A"
+          "drm_device" -> "drm_plane B"
+          "drm_device" -> "drm_crtc"
+          "drm_device" -> "drm_connector"
+          "drm_device" -> "driver private object"
+
+          "drm_plane A" -> "drm_plane_state A"
+          "drm_plane B" -> "drm_plane_state B"
+          "drm_crtc" -> "drm_crtc_state"
+          "drm_connector" -> "drm_connector_state"
+          "driver private object" -> "driver private state"
+      }
+
+      "drm_atomic_state" -> "drm_device" [label="atomic_commit"]
+   }
+
+Atomic provides transactional modeset (including planes) updates, but a
+bit differently from the usual transactional approach of try-commit and
+rollback:
+
+- Firstly, no hardware changes are allowed when the commit would fail. This
+  allows us to implement the DRM_MODE_ATOMIC_TEST_ONLY mode, which allows
+  userspace to explore whether certain configurations would work or not.
+
+- This would still allow setting and rollback of just the software state,
+  simplifying conversion of existing drivers. But auditing drivers for
+  correctness of the atomic_check code becomes really hard with that: Rolling
+  back changes in data structures all over the place is hard to get right.
+
+- Lastly, for backwards compatibility and to support all use-cases, atomic
+  updates need to be incremental and be able to execute in parallel. Hardware
+  doesn't always allow it, but where possible plane updates on different CRTCs
+  should not interfere, and not get stalled due to output routing changing on
+  different CRTCs.
+
+Taken all together there's two consequences for the atomic design:
+
+- The overall state is split up into per-object state structures:
+  :c:type:`struct drm_plane_state <drm_plane_state>` for planes, :c:type:`struct
+  drm_crtc_state <drm_crtc_state>` for CRTCs and :c:type:`struct
+  drm_connector_state <drm_connector_state` for connectors. These are the only
+  objects with userspace-visible and settable state. For internal state drivers
+  can subclass these structures through embeddeding, or add entirely new state
+  structures for their globally shared hardware functions.
+
+- An atomic update is assembled and validated as an enterily free-standing pile
+  of structures within the :c:type:`drm_atomic_state <drm_atomic_state>`
+  container. Again drivers can subclass that container for their own state
+  structure tracking needs. Only when a state is committed is it applied to the
+  driver and modeset objects. This way rolling back an update boils down to
+  releasing memory and unreferencing objects like framebuffers.
+
+Read on in this chapter, and also in :ref:`drm_atomic_helper` for more detailed
+coverage of specific topics.
+
 Atomic Mode Setting Function Reference
-======================================
+--------------------------------------
 
 .. kernel-doc:: include/drm/drm_atomic.h
    :internal:
-- 
2.11.0

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH] drm/doc: atomic overview, with graph

[Eric Anholt]

[-- Attachment #1: Type: text/plain, Size: 1607 bytes --]

Daniel Vetter <daniel.vetter@ffwll.ch> writes:

> I want to split up a few more things and document some details better
> (like how exactly to subclass drm_atomic_state). And maybe also split
> up the helpers a bit per-topic, but this should be a ok-ish start for
> better atomic overview.
>
> One thing I failed at is getting DOT to layout the overview graph how
> I want it. The highlevel structure I want is:
>
> 	Free-standing State
>
> 	Current State
>
> i.e. one over the other. Currently it lays it out side-by-side, but
> not even that really - "Current State" is somewhat offset below. Makes
> the graph look like garbage, and also way too wide for proper
> rendering. Ideas appreciated.
>
> v2: Spelling and clarifications (Eric).
>
> Cc: Eric Anholt <eric@anholt.net>
> Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Cc: Harry Wentland <harry.wentland@amd.com>
> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
> ---
>  Documentation/gpu/drm-kms-helpers.rst |  2 +
>  Documentation/gpu/drm-kms.rst         | 86 ++++++++++++++++++++++++++++++++++-
>  2 files changed, 87 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/gpu/drm-kms-helpers.rst b/Documentation/gpu/drm-kms-helpers.rst
> index 050ebe81d256..ac53c0b893f6 100644
> --- a/Documentation/gpu/drm-kms-helpers.rst
> +++ b/Documentation/gpu/drm-kms-helpers.rst

> +Atomic Mode Setting
> +===================
> +
> +
> +.. FIXME: The I want the below graph to be laid out so that the 2 subgraph
> +   clusters are below each another. But I failed.

s/The //

With that,

Acked-by: Eric Anholt <eric@anholt.net>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[PATCH] drm/doc: atomic overview, with graph

[Daniel Vetter]

I want to split up a few more things and document some details better
(like how exactly to subclass drm_atomic_state). And maybe also split
up the helpers a bit per-topic, but this should be a ok-ish start for
better atomic overview.

v2: Spelling and clarifications (Eric).

v3: Implement suggestion from Gabriel to fix the graph.

Cc: Gabriel Krisman Bertazi <krisman@collabora.co.uk>
Acked-by: Eric Anholt <eric@anholt.net>
Cc: Eric Anholt <eric@anholt.net>
Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
---
 Documentation/gpu/drm-kms-helpers.rst |  2 +
 Documentation/gpu/drm-kms.rst         | 84 ++++++++++++++++++++++++++++++++++-
 2 files changed, 85 insertions(+), 1 deletion(-)

diff --git a/Documentation/gpu/drm-kms-helpers.rst b/Documentation/gpu/drm-kms-helpers.rst
index 050ebe81d256..ac53c0b893f6 100644
--- a/Documentation/gpu/drm-kms-helpers.rst
+++ b/Documentation/gpu/drm-kms-helpers.rst
@@ -42,6 +42,8 @@ Modeset Helper Reference for Common Vtables
 .. kernel-doc:: include/drm/drm_modeset_helper_vtables.h
    :internal:
 
+.. _drm_atomic_helper:
+
 Atomic Modeset Helper Functions Reference
 =========================================
 
diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
index a504d9ee4d94..eb9d29865c41 100644
--- a/Documentation/gpu/drm-kms.rst
+++ b/Documentation/gpu/drm-kms.rst
@@ -189,8 +189,90 @@ multiple times to different objects using :c:func:`drm_object_attach_property()
 .. kernel-doc:: drivers/gpu/drm/drm_mode_object.c
    :export:
 
+Atomic Mode Setting
+===================
+
+
+.. kernel-render:: DOT
+   :alt: Mode Objects and Properties
+   :caption: Mode Objects and Properties
+
+   digraph {
+      node [shape=box]
+
+      subgraph cluster_state {
+          style=dashed
+          label="Free-standing state"
+
+          "drm_atomic_state" -> "duplicated drm_plane_state A"
+          "drm_atomic_state" -> "duplicated drm_plane_state B"
+          "drm_atomic_state" -> "duplicated drm_crtc_state"
+          "drm_atomic_state" -> "duplicated drm_connector_state"
+          "drm_atomic_state" -> "duplicated driver private state"
+      }
+
+      subgraph cluster_current {
+          style=dashed
+          label="Current state"
+
+          "drm_device" -> "drm_plane A"
+          "drm_device" -> "drm_plane B"
+          "drm_device" -> "drm_crtc"
+          "drm_device" -> "drm_connector"
+          "drm_device" -> "driver private object"
+
+          "drm_plane A" -> "drm_plane_state A"
+          "drm_plane B" -> "drm_plane_state B"
+          "drm_crtc" -> "drm_crtc_state"
+          "drm_connector" -> "drm_connector_state"
+          "driver private object" -> "driver private state"
+      }
+
+      "drm_atomic_state" -> "drm_device" [label="atomic_commit"]
+      "duplicated drm_plane_state A" -> "drm_device"[style=invis]
+   }
+
+Atomic provides transactional modeset (including planes) updates, but a
+bit differently from the usual transactional approach of try-commit and
+rollback:
+
+- Firstly, no hardware changes are allowed when the commit would fail. This
+  allows us to implement the DRM_MODE_ATOMIC_TEST_ONLY mode, which allows
+  userspace to explore whether certain configurations would work or not.
+
+- This would still allow setting and rollback of just the software state,
+  simplifying conversion of existing drivers. But auditing drivers for
+  correctness of the atomic_check code becomes really hard with that: Rolling
+  back changes in data structures all over the place is hard to get right.
+
+- Lastly, for backwards compatibility and to support all use-cases, atomic
+  updates need to be incremental and be able to execute in parallel. Hardware
+  doesn't always allow it, but where possible plane updates on different CRTCs
+  should not interfere, and not get stalled due to output routing changing on
+  different CRTCs.
+
+Taken all together there's two consequences for the atomic design:
+
+- The overall state is split up into per-object state structures:
+  :c:type:`struct drm_plane_state <drm_plane_state>` for planes, :c:type:`struct
+  drm_crtc_state <drm_crtc_state>` for CRTCs and :c:type:`struct
+  drm_connector_state <drm_connector_state` for connectors. These are the only
+  objects with userspace-visible and settable state. For internal state drivers
+  can subclass these structures through embeddeding, or add entirely new state
+  structures for their globally shared hardware functions.
+
+- An atomic update is assembled and validated as an enterily free-standing pile
+  of structures within the :c:type:`drm_atomic_state <drm_atomic_state>`
+  container. Again drivers can subclass that container for their own state
+  structure tracking needs. Only when a state is committed is it applied to the
+  driver and modeset objects. This way rolling back an update boils down to
+  releasing memory and unreferencing objects like framebuffers.
+
+Read on in this chapter, and also in :ref:`drm_atomic_helper` for more detailed
+coverage of specific topics.
+
 Atomic Mode Setting Function Reference
-======================================
+--------------------------------------
 
 .. kernel-doc:: include/drm/drm_atomic.h
    :internal:
-- 
2.11.0

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH] drm/doc: atomic overview, with graph

[Laurent Pinchart]

Hi Daniel,

Thank you for the patch.

On Thursday 02 Mar 2017 08:19:58 Daniel Vetter wrote:
> I want to split up a few more things and document some details better
> (like how exactly to subclass drm_atomic_state). And maybe also split
> up the helpers a bit per-topic, but this should be a ok-ish start for
> better atomic overview.
> 
> v2: Spelling and clarifications (Eric).
> 
> v3: Implement suggestion from Gabriel to fix the graph.
> 
> Cc: Gabriel Krisman Bertazi <krisman@collabora.co.uk>
> Acked-by: Eric Anholt <eric@anholt.net>
> Cc: Eric Anholt <eric@anholt.net>
> Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Cc: Harry Wentland <harry.wentland@amd.com>
> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
> ---
>  Documentation/gpu/drm-kms-helpers.rst |  2 +
>  Documentation/gpu/drm-kms.rst         | 84 +++++++++++++++++++++++++++++++-
>  2 files changed, 85 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/gpu/drm-kms-helpers.rst
> b/Documentation/gpu/drm-kms-helpers.rst index 050ebe81d256..ac53c0b893f6
> 100644
> --- a/Documentation/gpu/drm-kms-helpers.rst
> +++ b/Documentation/gpu/drm-kms-helpers.rst
> @@ -42,6 +42,8 @@ Modeset Helper Reference for Common Vtables
>  .. kernel-doc:: include/drm/drm_modeset_helper_vtables.h
> 
>     :internal:
> +.. _drm_atomic_helper:
> +
>  Atomic Modeset Helper Functions Reference
>  =========================================
> 
> diff --git a/Documentation/gpu/drm-kms.rst b/Documentation/gpu/drm-kms.rst
> index a504d9ee4d94..eb9d29865c41 100644
> --- a/Documentation/gpu/drm-kms.rst
> +++ b/Documentation/gpu/drm-kms.rst
> @@ -189,8 +189,90 @@ multiple times to different objects using
> :c:func:`drm_object_attach_property() .. kernel-doc::
> drivers/gpu/drm/drm_mode_object.c
> 
>     :export:
> +Atomic Mode Setting
> +===================
> +
> +
> +.. kernel-render:: DOT
> +   :alt: Mode Objects and Properties
> +   :caption: Mode Objects and Properties
> +
> +   digraph {
> +      node [shape=box]
> +
> +      subgraph cluster_state {
> +          style=dashed
> +          label="Free-standing state"
> +
> +          "drm_atomic_state" -> "duplicated drm_plane_state A"
> +          "drm_atomic_state" -> "duplicated drm_plane_state B"
> +          "drm_atomic_state" -> "duplicated drm_crtc_state"
> +          "drm_atomic_state" -> "duplicated drm_connector_state"
> +          "drm_atomic_state" -> "duplicated driver private state"
> +      }
> +
> +      subgraph cluster_current {
> +          style=dashed
> +          label="Current state"
> +
> +          "drm_device" -> "drm_plane A"
> +          "drm_device" -> "drm_plane B"
> +          "drm_device" -> "drm_crtc"
> +          "drm_device" -> "drm_connector"
> +          "drm_device" -> "driver private object"
> +
> +          "drm_plane A" -> "drm_plane_state A"
> +          "drm_plane B" -> "drm_plane_state B"
> +          "drm_crtc" -> "drm_crtc_state"
> +          "drm_connector" -> "drm_connector_state"
> +          "driver private object" -> "driver private state"
> +      }
> +
> +      "drm_atomic_state" -> "drm_device" [label="atomic_commit"]
> +      "duplicated drm_plane_state A" -> "drm_device"[style=invis]
> +   }
> +
> +Atomic provides transactional modeset (including planes) updates, but a
> +bit differently from the usual transactional approach of try-commit and
> +rollback:
> +
> +- Firstly, no hardware changes are allowed when the commit would fail. This
> +  allows us to implement the DRM_MODE_ATOMIC_TEST_ONLY mode, which allows
> +  userspace to explore whether certain configurations would work or not. +
> +- This would still allow setting and rollback of just the software state,
> +  simplifying conversion of existing drivers. But auditing drivers for
> +  correctness of the atomic_check code becomes really hard with that:
> Rolling
> +  back changes in data structures all over the place is hard to get right.
> +
> +- Lastly, for backwards compatibility and to support all use-cases, atomic

s/backwards/backward/ (see my comment to patch 3/6, and not that the margin is 
now slightly larger :-))

> +  updates need to be incremental and be able to execute in parallel.
> Hardware
> +  doesn't always allow it, but where possible plane updates on
> different CRTCs
> +  should not interfere, and not get stalled due to output routing changing
> on
> +  different CRTCs.
> +
> +Taken all together there's two consequences for the atomic design:
> +
> +- The overall state is split up into per-object state structures:
> +  :c:type:`struct drm_plane_state <drm_plane_state>` for planes,
> :c:type:`struct +  drm_crtc_state <drm_crtc_state>` for CRTCs and
> :c:type:`struct
> +  drm_connector_state <drm_connector_state`

Missing >

> for connectors. These are the
> only +  objects with userspace-visible and settable state. For internal
> state drivers +  can subclass these structures through embeddeding, or add
> entirely new state +  structures for their globally shared hardware
> functions.
> +
> +- An atomic update is assembled and validated as an enterily free-standing

s/enterily/entirely/

With this fixed,

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

(this applies to patch 3/6 too, I forgot to mention it there)

> pile +  of structures within the :c:type:`drm_atomic_state
> <drm_atomic_state>` +  container. Again drivers can subclass that container
> for their own state +  structure tracking needs. Only when a state is
> committed is it applied to the +  driver and modeset objects. This way
> rolling back an update boils down to +  releasing memory and unreferencing
> objects like framebuffers.
> +
> +Read on in this chapter, and also in :ref:`drm_atomic_helper` for more
> detailed +coverage of specific topics.
> +
>  Atomic Mode Setting Function Reference
> -======================================
> +--------------------------------------
> 
>  .. kernel-doc:: include/drm/drm_atomic.h
> 
>     :internal:

-- 
Regards,

Laurent Pinchart

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel