[PATCH v2 0/4] docs: sphinx/kfigure.py: Improve conversion to PDF

[PATCH v2 0/4] docs: sphinx/kfigure.py: Improve conversion to PDF

[Akira Yokosawa]

This patch set improves conversions of DOT -> PDF and SVG -> PDF
for PDF docs.

* DOT -> PDF conversion

Current scheme uses "dot -Tpdf" (of graphviz).

Cons:
  - openSUSE's dot(1) does not support -Tpdf.
  - Other distro's dot(1) generates PDFs with unnecessarily wide
    margins for inclusion into LaTeX docs.

Patch 1/4 changes the route to the following two steps:

  1. DOT -> SVG by "dot -Tsvg"
  2. SVG -> PDF by "rsvg-convert -f pdf" with fallback to convert(1)

Pros:
  - Improved portability across distros
  - Less space around graphs in final PDF documents

Con:
  - On systems without rsvg-convert, generated PDF will be of raster
    image.

Patch 2/4 avoids raster-image PDF by using "dot -Tpdf" on systems where
the option is available.

* SVG -> PDF conversion

Current scheme uses convert(1) (of ImageMagick)

Cons:
  - Generated PDFs are of raster image.  Some of them look blurry.
  - Raster images tend to be large in size.
  - convert(1) delegates SVG decoding to rsvg-convert(1).
    It doesn't cover full range of Inkscape-specific SVG features
    and fails to convert some of SVG figures properly.

Improper conversions are observed with SVGs listed below (incomplete,
conversion quality depends on the version of rsvg-convert):
  - Documentation/userspace-api/media/v4l/selection.svg
  - Documentation/userspace-api/media/v4l/vbi_525.svg
  - Documentation/userspace-api/media/v4l/vbi_625.svg
  - Documentation/userspace-api/media/v4l/vbi_hsync.svg
  - Documentation/admin-guide/blockdev/drbd/DRBD-8.3-data-packets.svg
  - Documentation/admin-guide/blockdev/drbd/DRBD-data-packages.svg

If you have Inkscape installed as well, convert(1) delegates SVG
decoding to inkscape(1) rather than to rsvg-convert(1) and SVGs listed
above can be rendered properly.

So if Inkscape is required for converting those SVGs properly, why not
use it directly in the first place?

Patches 3/4 and 4/4 add code to utilize inkscape(1) for SVG -> PDF
conversion when it is available.  They don't modify any existing
requirements for kernel-doc.

Patch 3/4 adds the alternative route of SVG -> PDF conversion by
inkscape(1).
Patch 4/4 delegates warning messages from inkscape(1) to kernellog.verbose
as they are likely harmless in command-line uses.

Pros:
  - Generated PDFs are of vector graphics.
  - Vector graphics tends to be smaller in size and looks nicer when
    zoomed in.
  - SVGs drawn by Inkscape are fully supported.

On systems without Inkscape, no regression is expected by these two
patches.

Changes since v1 (as of Patch 5/3) [1]:

- Reorder and merge patches to reduce/eliminate regression windows of
  raster-image PDF and stderr redirection.
    v1        v2
    1/3       1/4
    4/3       2/4
    2/3       3/4
    3/3+5/3   4/4

- Massage kernellog.verbose/warn messages. They now show command(s)
  used in DOT -> PDF conversion.

- Pass actual exit code of inkscape(1) to kernellog.warn.

FWIW, diff of v1 vs. v2 follows:

--------------------------------------------------------------
diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py
index a275ee0fec02..24d2b2addcce 100644
--- a/Documentation/sphinx/kfigure.py
+++ b/Documentation/sphinx/kfigure.py
@@ -220,10 +220,16 @@ def setupTools(app):
 
         if rsvg_convert_cmd:
             kernellog.verbose(app, "use rsvg-convert(1) from: " + rsvg_convert_cmd)
+            kernellog.verbose(app, "use 'dot -Tsvg' and rsvg-convert(1) for DOT -> PDF conversion")
             dot_Tpdf = False
         else:
-            kernellog.verbose(app, "rsvg-convert(1) not found, "
-                              "falling back to raster image conversion")
+            kernellog.verbose(app,
+                "rsvg-convert(1) not found.\n"
+                "  SVG rendering of convert(1) is done by ImageMagick-native renderer.")
+            if dot_Tpdf:
+                kernellog.verbose(app, "use 'dot -Tpdf' for DOT -> PDF conversion")
+            else:
+                kernellog.verbose(app, "use 'dot -Tsvg' and convert(1) for DOT -> PDF conversion")
 
 
 # integrate conversion tools
@@ -368,8 +374,10 @@ def svg2pdf(app, svg_fname, pdf_fname):
 
     """
     cmd = [convert_cmd, svg_fname, pdf_fname]
+    cmd_name = 'convert(1)'
 
     if inkscape_cmd:
+        cmd_name = 'inkscape(1)'
         if inkscape_ver_one:
             cmd = [inkscape_cmd, '-o', pdf_fname, svg_fname]
         else:
@@ -380,15 +388,17 @@ def svg2pdf(app, svg_fname, pdf_fname):
         exit_code = 0
     except subprocess.CalledProcessError as err:
         warning_msg = err.output
-        exit_code = 1
+        exit_code = err.returncode
         pass
 
     if exit_code != 0:
         kernellog.warn(app, "Error #%d when calling: %s" % (exit_code, " ".join(cmd)))
-        kernellog.warn(app, "Warning msg from inkscape: %s" % str(warning_msg, 'utf-8'))
-    if warning_msg:
-        kernellog.verbose(app, "Warning msg from inkscape (likely harmless):\n%s"
-                          % str(warning_msg, 'utf-8'))
+        if warning_msg:
+            kernellog.warn(app, "Warning msg from %s: %s"
+                           % (cmd_name, str(warning_msg, 'utf-8')))
+    elif warning_msg:
+        kernellog.verbose(app, "Warning msg from %s (likely harmless):\n%s"
+                          % (cmd_name, str(warning_msg, 'utf-8')))
 
     return bool(exit_code == 0)
 
--------------------------------------------------------------

[1] v1: https://lkml.kernel.org/r/de8def13-efbc-1d98-acb5-5cc1f6902e4b@gmail.com

        Thanks, Akira
--
Akira Yokosawa (4):
  docs: sphinx/kfigure.py: Use rsvg-convert(1) for DOT -> PDF conversion
  docs: sphinx/kfigure.py: Add check of 'dot -Tpdf'
  docs: sphinx/kfigure.py: Use inkscape(1) for SVG -> PDF conversion
  docs: sphinx/kfigure.py: Delegate inkscape msg to kernellog.verbose

 Documentation/sphinx/kfigure.py | 134 ++++++++++++++++++++++++++++----
 1 file changed, 121 insertions(+), 13 deletions(-)


base-commit: b36064425a18e29a3bad9c007b4dd1223f8aadc5
-- 
2.17.1

[PATCH v2 1/4] docs: sphinx/kfigure.py: Use rsvg-convert(1) for DOT -> PDF conversion

[Akira Yokosawa]

On openSUSE, dot(1) command does not support direct PDF output.
On other distros, generated PDF images have unnecessarily wide margins,
especially for small graphs.

By using dot(1) for DOT -> SVG, then rsvg-convert(1) for SVG -> PDF,
more optimal PDF images can be obtained, with the bonus of improved
portability across various distros.

Add rules in kfigure.py so that the above mentioned route is taken
when rsvg-convert(1) is available.

Note that rsvg-convert(1) is recommended by sphinx_pre_install.
So it is most likely that existing systems for building pdfdocs have
rsvg-convert(1) installed.

Note:
    SVG features supported by rsvg-convert(1) vary depending on its
    version and distro config.
    For example, the one found on Ubuntu Bionic (version 2.40.20) does
    poor job in rendering some of SVG files drawn by Inkscape.
    SVG files generated by dot(1) are converted nicely even with such
    old versions of rsvg-convert.

    So this change does not affect the quality of such figures in any
    way.

Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 Documentation/sphinx/kfigure.py | 46 +++++++++++++++++++++++++++++++--
 1 file changed, 44 insertions(+), 2 deletions(-)

diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py
index 3c78828330be..955e3ec5de5a 100644
--- a/Documentation/sphinx/kfigure.py
+++ b/Documentation/sphinx/kfigure.py
@@ -31,6 +31,8 @@ u"""
 
     * ``dot(1)``: Graphviz (https://www.graphviz.org). If Graphviz is not
       available, the DOT language is inserted as literal-block.
+      For conversion to PDF, ``rsvg-convert(1)`` of librsvg
+      (https://gitlab.gnome.org/GNOME/librsvg) is used when available.
 
     * SVG to PDF: To generate PDF, you need at least one of this tools:
 
@@ -113,6 +115,9 @@ dot_cmd = None
 # ImageMagick' convert(1) support
 convert_cmd = None
 
+# librsvg's rsvg-convert(1) support
+rsvg_convert_cmd = None
+
 
 def setup(app):
     # check toolchain first
@@ -160,11 +165,12 @@ def setupTools(app):
 
     This function is called once, when the builder is initiated.
     """
-    global dot_cmd, convert_cmd   # pylint: disable=W0603
+    global dot_cmd, convert_cmd, rsvg_convert_cmd   # pylint: disable=W0603
     kernellog.verbose(app, "kfigure: check installed tools ...")
 
     dot_cmd = which('dot')
     convert_cmd = which('convert')
+    rsvg_convert_cmd = which('rsvg-convert')
 
     if dot_cmd:
         kernellog.verbose(app, "use dot(1) from: " + dot_cmd)
@@ -177,6 +183,11 @@ def setupTools(app):
         kernellog.warn(app,
             "convert(1) not found, for SVG to PDF conversion install "
             "ImageMagick (https://www.imagemagick.org)")
+    if rsvg_convert_cmd:
+        kernellog.verbose(app, "use rsvg-convert(1) from: " + rsvg_convert_cmd)
+    else:
+        kernellog.verbose(app, "rsvg-convert(1) not found, "
+                          "falling back to raster image conversion")
 
 
 # integrate conversion tools
@@ -266,7 +277,13 @@ def convert_image(img_node, translator, src_fname=None):
 
             if in_ext == '.dot':
                 kernellog.verbose(app, 'convert DOT to: {out}/' + _name)
-                ok = dot2format(app, src_fname, dst_fname)
+                if translator.builder.format == 'latex':
+                    svg_fname = path.join(translator.builder.outdir, fname + '.svg')
+                    ok1 = dot2format(app, src_fname, svg_fname)
+                    ok2 = svg2pdf_by_rsvg(app, svg_fname, dst_fname)
+                    ok = ok1 and ok2
+                else:
+                    ok = dot2format(app, src_fname, dst_fname)
 
             elif in_ext == '.svg':
                 kernellog.verbose(app, 'convert SVG to: {out}/' + _name)
@@ -319,6 +336,31 @@ def svg2pdf(app, svg_fname, pdf_fname):
         kernellog.warn(app, "Error #%d when calling: %s" % (exit_code, " ".join(cmd)))
     return bool(exit_code == 0)
 
+def svg2pdf_by_rsvg(app, svg_fname, pdf_fname):
+    """Convert SVG to PDF with ``rsvg-convert(1)`` command.
+
+    * ``svg_fname`` pathname of input SVG file, including extension ``.svg``
+    * ``pdf_fname`` pathname of output PDF file, including extension ``.pdf``
+
+    Input SVG file should be the one generated by ``dot2format()``.
+    SVG -> PDF conversion is done by ``rsvg-convert(1)``.
+
+    If ``rsvg-convert(1)`` is unavailable, fall back to ``svg2pdf()``.
+
+    """
+
+    if rsvg_convert_cmd is None:
+        ok = svg2pdf(app, svg_fname, pdf_fname)
+    else:
+        cmd = [rsvg_convert_cmd, '--format=pdf', '-o', pdf_fname, svg_fname]
+        # use stdout and stderr from parent
+        exit_code = subprocess.call(cmd)
+        if exit_code != 0:
+            kernellog.warn(app, "Error #%d when calling: %s" % (exit_code, " ".join(cmd)))
+        ok = bool(exit_code == 0)
+
+    return ok
+
 
 # image handling
 # ---------------------
-- 
2.17.1

[PATCH v2 2/4] docs: sphinx/kfigure.py: Add check of 'dot -Tpdf'

[Akira Yokosawa]

To prevent any regression on existing build systems, limit the
fallback of converting DOT -> raster PDF only when both of the
following conditions are met.

 o dot(1) doesn't support -Tpdf
 o rsvg-convert(1) is not found

While we are here, add kernellog.verbose messages related to
rsvg-convert, 'dot -Tpdf', and 'dot -Tsvg' commands.

Suggested-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
Cc: Jonathan Corbet <corbet@lwn.net>
---
 Documentation/sphinx/kfigure.py | 30 ++++++++++++++++++++++++++----
 1 file changed, 26 insertions(+), 4 deletions(-)

diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py
index 955e3ec5de5a..77b0d15dba31 100644
--- a/Documentation/sphinx/kfigure.py
+++ b/Documentation/sphinx/kfigure.py
@@ -51,6 +51,7 @@ import os
 from os import path
 import subprocess
 from hashlib import sha1
+import re
 from docutils import nodes
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives
@@ -111,6 +112,8 @@ def pass_handle(self, node):           # pylint: disable=W0613
 
 # Graphviz's dot(1) support
 dot_cmd = None
+# dot(1) -Tpdf should be used
+dot_Tpdf = False
 
 # ImageMagick' convert(1) support
 convert_cmd = None
@@ -165,7 +168,7 @@ def setupTools(app):
 
     This function is called once, when the builder is initiated.
     """
-    global dot_cmd, convert_cmd, rsvg_convert_cmd   # pylint: disable=W0603
+    global dot_cmd, dot_Tpdf, convert_cmd, rsvg_convert_cmd   # pylint: disable=W0603
     kernellog.verbose(app, "kfigure: check installed tools ...")
 
     dot_cmd = which('dot')
@@ -174,6 +177,16 @@ def setupTools(app):
 
     if dot_cmd:
         kernellog.verbose(app, "use dot(1) from: " + dot_cmd)
+
+        try:
+            dot_Thelp_list = subprocess.check_output([dot_cmd, '-Thelp'],
+                                    stderr=subprocess.STDOUT)
+        except subprocess.CalledProcessError as err:
+            dot_Thelp_list = err.output
+            pass
+
+        dot_Tpdf_ptn = b'pdf'
+        dot_Tpdf = re.search(dot_Tpdf_ptn, dot_Thelp_list)
     else:
         kernellog.warn(app, "dot(1) not found, for better output quality install "
                        "graphviz from https://www.graphviz.org")
@@ -185,9 +198,17 @@ def setupTools(app):
             "ImageMagick (https://www.imagemagick.org)")
     if rsvg_convert_cmd:
         kernellog.verbose(app, "use rsvg-convert(1) from: " + rsvg_convert_cmd)
+        kernellog.verbose(app, "use 'dot -Tsvg' and rsvg-convert(1) for DOT -> PDF conversion")
+        dot_Tpdf = False
     else:
-        kernellog.verbose(app, "rsvg-convert(1) not found, "
-                          "falling back to raster image conversion")
+        kernellog.verbose(app,
+            "rsvg-convert(1) not found.\n"
+            "  SVG -> PDF conversion by convert() can be poor quality.\n"
+            "  Install librsvg (https://gitlab.gnome.org/GNOME/librsvg)")
+        if dot_Tpdf:
+            kernellog.verbose(app, "use 'dot -Tpdf' for DOT -> PDF conversion")
+        else:
+            kernellog.verbose(app, "use 'dot -Tsvg' and convert(1) for DOT -> PDF conversion")
 
 
 # integrate conversion tools
@@ -277,11 +298,12 @@ def convert_image(img_node, translator, src_fname=None):
 
             if in_ext == '.dot':
                 kernellog.verbose(app, 'convert DOT to: {out}/' + _name)
-                if translator.builder.format == 'latex':
+                if translator.builder.format == 'latex' and not dot_Tpdf:
                     svg_fname = path.join(translator.builder.outdir, fname + '.svg')
                     ok1 = dot2format(app, src_fname, svg_fname)
                     ok2 = svg2pdf_by_rsvg(app, svg_fname, dst_fname)
                     ok = ok1 and ok2
+
                 else:
                     ok = dot2format(app, src_fname, dst_fname)
 
-- 
2.17.1

[PATCH v2 3/4] docs: sphinx/kfigure.py: Use inkscape(1) for SVG -> PDF conversion

[Akira Yokosawa]

Using convert(1) of ImageMagick for SVG -> PDF conversion results in
PDFs containing raster (bitmap) images which sometimes look blurry.

Ideally speaking, SVG to PDF conversion should retain vector graphics
in SVG.

rsvg-convert(1) can do such conversions with regard to SVG files
generated by dot(1).

Unfortunately, rsvg-convert(1) does not cover some of SVG features
specific to Inkscape.
inkscape(1) of Inkscape naturally covers such SVG features.

So add a route in svg2pdf() so that inkscape(1) is used when it is
available.

Note:
    After this change, if you have Inkscape installed, ImageMagick nor
    librsvg are not required.

Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 Documentation/sphinx/kfigure.py | 68 +++++++++++++++++++++++----------
 1 file changed, 48 insertions(+), 20 deletions(-)

diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py
index 77b0d15dba31..e616e49669eb 100644
--- a/Documentation/sphinx/kfigure.py
+++ b/Documentation/sphinx/kfigure.py
@@ -37,6 +37,7 @@ u"""
     * SVG to PDF: To generate PDF, you need at least one of this tools:
 
       - ``convert(1)``: ImageMagick (https://www.imagemagick.org)
+      - ``inkscape(1)``: Inkscape (https://inkscape.org/)
 
     List of customizations:
 
@@ -121,6 +122,11 @@ convert_cmd = None
 # librsvg's rsvg-convert(1) support
 rsvg_convert_cmd = None
 
+# Inkscape's inkscape(1) support
+inkscape_cmd = None
+# Inkscape prior to 1.0 uses different command options
+inkscape_ver_one = False
+
 
 def setup(app):
     # check toolchain first
@@ -169,11 +175,13 @@ def setupTools(app):
     This function is called once, when the builder is initiated.
     """
     global dot_cmd, dot_Tpdf, convert_cmd, rsvg_convert_cmd   # pylint: disable=W0603
+    global inkscape_cmd, inkscape_ver_one  # pylint: disable=W0603
     kernellog.verbose(app, "kfigure: check installed tools ...")
 
     dot_cmd = which('dot')
     convert_cmd = which('convert')
     rsvg_convert_cmd = which('rsvg-convert')
+    inkscape_cmd = which('inkscape')
 
     if dot_cmd:
         kernellog.verbose(app, "use dot(1) from: " + dot_cmd)
@@ -190,25 +198,37 @@ def setupTools(app):
     else:
         kernellog.warn(app, "dot(1) not found, for better output quality install "
                        "graphviz from https://www.graphviz.org")
-    if convert_cmd:
-        kernellog.verbose(app, "use convert(1) from: " + convert_cmd)
-    else:
-        kernellog.warn(app,
-            "convert(1) not found, for SVG to PDF conversion install "
-            "ImageMagick (https://www.imagemagick.org)")
-    if rsvg_convert_cmd:
-        kernellog.verbose(app, "use rsvg-convert(1) from: " + rsvg_convert_cmd)
-        kernellog.verbose(app, "use 'dot -Tsvg' and rsvg-convert(1) for DOT -> PDF conversion")
+    if inkscape_cmd:
+        kernellog.verbose(app, "use inkscape(1) from: " + inkscape_cmd)
+        inkscape_ver = subprocess.check_output([inkscape_cmd, '--version'])
+        ver_one_ptn = b'Inkscape 1'
+        inkscape_ver_one = re.search(ver_one_ptn, inkscape_ver)
+        convert_cmd = None
+        rsvg_convert_cmd = None
         dot_Tpdf = False
+
     else:
-        kernellog.verbose(app,
-            "rsvg-convert(1) not found.\n"
-            "  SVG -> PDF conversion by convert() can be poor quality.\n"
-            "  Install librsvg (https://gitlab.gnome.org/GNOME/librsvg)")
-        if dot_Tpdf:
-            kernellog.verbose(app, "use 'dot -Tpdf' for DOT -> PDF conversion")
+        if convert_cmd:
+            kernellog.verbose(app, "use convert(1) from: " + convert_cmd)
+        else:
+            kernellog.warn(app,
+                "Neither inkscape(1) nor convert(1) found.\n"
+                "For SVG to PDF conversion, "
+                "install either Inkscape (https://inkscape.org/) (preferred) or\n"
+                "ImageMagick (https://www.imagemagick.org)")
+
+        if rsvg_convert_cmd:
+            kernellog.verbose(app, "use rsvg-convert(1) from: " + rsvg_convert_cmd)
+            kernellog.verbose(app, "use 'dot -Tsvg' and rsvg-convert(1) for DOT -> PDF conversion")
+            dot_Tpdf = False
         else:
-            kernellog.verbose(app, "use 'dot -Tsvg' and convert(1) for DOT -> PDF conversion")
+            kernellog.verbose(app,
+                "rsvg-convert(1) not found.\n"
+                "  SVG rendering of convert(1) is done by ImageMagick-native renderer.")
+            if dot_Tpdf:
+                kernellog.verbose(app, "use 'dot -Tpdf' for DOT -> PDF conversion")
+            else:
+                kernellog.verbose(app, "use 'dot -Tsvg' and convert(1) for DOT -> PDF conversion")
 
 
 # integrate conversion tools
@@ -274,7 +294,7 @@ def convert_image(img_node, translator, src_fname=None):
     elif in_ext == '.svg':
 
         if translator.builder.format == 'latex':
-            if convert_cmd is None:
+            if not inkscape_cmd and convert_cmd is None:
                 kernellog.verbose(app,
                                   "no SVG to PDF conversion available / include SVG raw.")
                 img_node.replace_self(file2literal(src_fname))
@@ -342,16 +362,24 @@ def dot2format(app, dot_fname, out_fname):
     return bool(exit_code == 0)
 
 def svg2pdf(app, svg_fname, pdf_fname):
-    """Converts SVG to PDF with ``convert(1)`` command.
+    """Converts SVG to PDF with ``inkscape(1)`` or ``convert(1)`` command.
 
-    Uses ``convert(1)`` from ImageMagick (https://www.imagemagick.org) for
-    conversion.  Returns ``True`` on success and ``False`` if an error occurred.
+    Uses ``inkscape(1)`` from Inkscape (https://inkscape.org/) or ``convert(1)``
+    from ImageMagick (https://www.imagemagick.org) for conversion.
+    Returns ``True`` on success and ``False`` if an error occurred.
 
     * ``svg_fname`` pathname of the input SVG file with extension (``.svg``)
     * ``pdf_name``  pathname of the output PDF file with extension (``.pdf``)
 
     """
     cmd = [convert_cmd, svg_fname, pdf_fname]
+
+    if inkscape_cmd:
+        if inkscape_ver_one:
+            cmd = [inkscape_cmd, '-o', pdf_fname, svg_fname]
+        else:
+            cmd = [inkscape_cmd, '-z', '--export-pdf=%s' % pdf_fname, svg_fname]
+
     # use stdout and stderr from parent
     exit_code = subprocess.call(cmd)
     if exit_code != 0:
-- 
2.17.1

[PATCH v2 4/4] docs: sphinx/kfigure.py: Delegate inkscape msg to kernellog.verbose

[Akira Yokosawa]

Subject: [PATCH v2 4/4] docs: sphinx/kfigure.py: Delegate inkscape msg to kernellog.verbose

Depending on its version, distro config, and system-setup type,
inkscape(1) emits various warning messages which are harmless in
command-line uses.

List of such warning messages (incomplete, long ones wrapped):

  - Gtk-Message: hh:mm:ss.nnn: Failed to load module "canberra-gtk-module"
  - Unable to init server: Could not connect: Connection refused
  - Failed to get connection
  - ** (inkscape:xxx): CRITICAL **: hh:mm:ss.nnn: dbus_g_proxy_new_for_name:
    assertion 'connection != NULL' failed
  - ** (inkscape:xxx): CRITICAL **: hh:mm:ss.nnn: dbus_g_proxy_call:
    assertion 'DBUS_IS_G_PROXY (proxy)' failed
  - ** (inkscape:xxx): CRITICAL **: hh:mm:ss.nnn: dbus_g_connection_register_g_object:
    assertion 'connection != NULL' failed
  - ** (inkscape:xxx): WARNING **: hh:mm:ss.nnn:
    Fonts dir '/usr/share/inkscape/fonts' does not exist and will be ignored.

To avoid unnecessary anxiety, capture the message and output it via
kernellog.verbose or kernellog.warn depending on the exit code.

Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
Cc: Randy Dunlap <rdunlap@infradead.org>
---
 Documentation/sphinx/kfigure.py | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py
index e616e49669eb..24d2b2addcce 100644
--- a/Documentation/sphinx/kfigure.py
+++ b/Documentation/sphinx/kfigure.py
@@ -200,7 +200,8 @@ def setupTools(app):
                        "graphviz from https://www.graphviz.org")
     if inkscape_cmd:
         kernellog.verbose(app, "use inkscape(1) from: " + inkscape_cmd)
-        inkscape_ver = subprocess.check_output([inkscape_cmd, '--version'])
+        inkscape_ver = subprocess.check_output([inkscape_cmd, '--version'],
+                                               stderr=subprocess.DEVNULL)
         ver_one_ptn = b'Inkscape 1'
         inkscape_ver_one = re.search(ver_one_ptn, inkscape_ver)
         convert_cmd = None
@@ -373,17 +374,32 @@ def svg2pdf(app, svg_fname, pdf_fname):
 
     """
     cmd = [convert_cmd, svg_fname, pdf_fname]
+    cmd_name = 'convert(1)'
 
     if inkscape_cmd:
+        cmd_name = 'inkscape(1)'
         if inkscape_ver_one:
             cmd = [inkscape_cmd, '-o', pdf_fname, svg_fname]
         else:
             cmd = [inkscape_cmd, '-z', '--export-pdf=%s' % pdf_fname, svg_fname]
 
-    # use stdout and stderr from parent
-    exit_code = subprocess.call(cmd)
+    try:
+        warning_msg = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+        exit_code = 0
+    except subprocess.CalledProcessError as err:
+        warning_msg = err.output
+        exit_code = err.returncode
+        pass
+
     if exit_code != 0:
         kernellog.warn(app, "Error #%d when calling: %s" % (exit_code, " ".join(cmd)))
+        if warning_msg:
+            kernellog.warn(app, "Warning msg from %s: %s"
+                           % (cmd_name, str(warning_msg, 'utf-8')))
+    elif warning_msg:
+        kernellog.verbose(app, "Warning msg from %s (likely harmless):\n%s"
+                          % (cmd_name, str(warning_msg, 'utf-8')))
+
     return bool(exit_code == 0)
 
 def svg2pdf_by_rsvg(app, svg_fname, pdf_fname):
-- 
2.17.1

Re: [PATCH v2 0/4] docs: sphinx/kfigure.py: Improve conversion to PDF

[Akira Yokosawa]

On Wed, 29 Dec 2021 20:42:00 +0900, Akira Yokosawa wrote:
> This patch set improves conversions of DOT -> PDF and SVG -> PDF
> for PDF docs.

Gentle ping.

Mauro, any comments?

> 
> * DOT -> PDF conversion
> 
> Current scheme uses "dot -Tpdf" (of graphviz).
> 
> Cons:
>   - openSUSE's dot(1) does not support -Tpdf.
>   - Other distro's dot(1) generates PDFs with unnecessarily wide
>     margins for inclusion into LaTeX docs.
> 
> Patch 1/4 changes the route to the following two steps:
> 
>   1. DOT -> SVG by "dot -Tsvg"
>   2. SVG -> PDF by "rsvg-convert -f pdf" with fallback to convert(1)
> 
> Pros:
>   - Improved portability across distros
>   - Less space around graphs in final PDF documents
> 
> Con:
>   - On systems without rsvg-convert, generated PDF will be of raster
>     image.
> 
> Patch 2/4 avoids raster-image PDF by using "dot -Tpdf" on systems where
> the option is available.
> 
> * SVG -> PDF conversion
> 
> Current scheme uses convert(1) (of ImageMagick)

I was not aware of security concerns regarding ImageMagick until
Christoph brought them up in another thread [1].

[1]: https://lore.kernel.org/linux-doc/20220104131952.GA21933@lst.de/

Now I can add another Con as bellow.

> 
> Cons:
    - ImageMagick is not allowed to read/write PDF by default under
      Debian/Ubuntu and Gentoo systems.  The policy is a band-aide
      fix to its security issues.
>   - Generated PDFs are of raster image.  Some of them look blurry.
>   - Raster images tend to be large in size.
>   - convert(1) delegates SVG decoding to rsvg-convert(1).
>     It doesn't cover full range of Inkscape-specific SVG features
>     and fails to convert some of SVG figures properly.

        Thanks, Akira

> 
> Improper conversions are observed with SVGs listed below (incomplete,
> conversion quality depends on the version of rsvg-convert):
>   - Documentation/userspace-api/media/v4l/selection.svg
>   - Documentation/userspace-api/media/v4l/vbi_525.svg
>   - Documentation/userspace-api/media/v4l/vbi_625.svg
>   - Documentation/userspace-api/media/v4l/vbi_hsync.svg
>   - Documentation/admin-guide/blockdev/drbd/DRBD-8.3-data-packets.svg
>   - Documentation/admin-guide/blockdev/drbd/DRBD-data-packages.svg
> 
> If you have Inkscape installed as well, convert(1) delegates SVG
> decoding to inkscape(1) rather than to rsvg-convert(1) and SVGs listed
> above can be rendered properly.
> 
> So if Inkscape is required for converting those SVGs properly, why not
> use it directly in the first place?
> 
> Patches 3/4 and 4/4 add code to utilize inkscape(1) for SVG -> PDF
> conversion when it is available.  They don't modify any existing
> requirements for kernel-doc.
> 
> Patch 3/4 adds the alternative route of SVG -> PDF conversion by
> inkscape(1).
> Patch 4/4 delegates warning messages from inkscape(1) to kernellog.verbose
> as they are likely harmless in command-line uses.
> 
> Pros:
>   - Generated PDFs are of vector graphics.
>   - Vector graphics tends to be smaller in size and looks nicer when
>     zoomed in.
>   - SVGs drawn by Inkscape are fully supported.
> 
> On systems without Inkscape, no regression is expected by these two
> patches.
> 
> Changes since v1 (as of Patch 5/3) [1]:
> 
> - Reorder and merge patches to reduce/eliminate regression windows of
>   raster-image PDF and stderr redirection.
>     v1        v2
>     1/3       1/4
>     4/3       2/4
>     2/3       3/4
>     3/3+5/3   4/4
> 
> - Massage kernellog.verbose/warn messages. They now show command(s)
>   used in DOT -> PDF conversion.
> 
> - Pass actual exit code of inkscape(1) to kernellog.warn.
> 
> FWIW, diff of v1 vs. v2 follows:
> 
> --------------------------------------------------------------
[...]

Re: [PATCH v2 0/4] docs: sphinx/kfigure.py: Improve conversion to PDF

[Mauro Carvalho Chehab]

Em Fri, 7 Jan 2022 22:45:47 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> On Wed, 29 Dec 2021 20:42:00 +0900, Akira Yokosawa wrote:
> > This patch set improves conversions of DOT -> PDF and SVG -> PDF
> > for PDF docs.  
> 
> Gentle ping.
> 
> Mauro, any comments?

Sorry, have been busy those days with not much time to test it,
and I'm not expecting any time to test it on the next couple of
weeks.

The main concern from my last review is that inkscape is too noisy 
(well, frankly, textlive is also too noisy). If this was solved
on a nice way, and provided that the output files on both html and
pdf are working fine with those conversions, I don't have any 
objections to this series.

Regards,
Mauro

> 
> > 
> > * DOT -> PDF conversion
> > 
> > Current scheme uses "dot -Tpdf" (of graphviz).
> > 
> > Cons:
> >   - openSUSE's dot(1) does not support -Tpdf.
> >   - Other distro's dot(1) generates PDFs with unnecessarily wide
> >     margins for inclusion into LaTeX docs.
> > 
> > Patch 1/4 changes the route to the following two steps:
> > 
> >   1. DOT -> SVG by "dot -Tsvg"
> >   2. SVG -> PDF by "rsvg-convert -f pdf" with fallback to convert(1)
> > 
> > Pros:
> >   - Improved portability across distros
> >   - Less space around graphs in final PDF documents
> > 
> > Con:
> >   - On systems without rsvg-convert, generated PDF will be of raster
> >     image.
> > 
> > Patch 2/4 avoids raster-image PDF by using "dot -Tpdf" on systems where
> > the option is available.
> > 
> > * SVG -> PDF conversion
> > 
> > Current scheme uses convert(1) (of ImageMagick)  
> 
> I was not aware of security concerns regarding ImageMagick until
> Christoph brought them up in another thread [1].
> 
> [1]: https://lore.kernel.org/linux-doc/20220104131952.GA21933@lst.de/
> 
> Now I can add another Con as bellow.
> 
> > 
> > Cons:  
>     - ImageMagick is not allowed to read/write PDF by default under
>       Debian/Ubuntu and Gentoo systems.  The policy is a band-aide
>       fix to its security issues.
> >   - Generated PDFs are of raster image.  Some of them look blurry.
> >   - Raster images tend to be large in size.
> >   - convert(1) delegates SVG decoding to rsvg-convert(1).
> >     It doesn't cover full range of Inkscape-specific SVG features
> >     and fails to convert some of SVG figures properly.  
> 
>         Thanks, Akira
> 
> > 
> > Improper conversions are observed with SVGs listed below (incomplete,
> > conversion quality depends on the version of rsvg-convert):
> >   - Documentation/userspace-api/media/v4l/selection.svg
> >   - Documentation/userspace-api/media/v4l/vbi_525.svg
> >   - Documentation/userspace-api/media/v4l/vbi_625.svg
> >   - Documentation/userspace-api/media/v4l/vbi_hsync.svg
> >   - Documentation/admin-guide/blockdev/drbd/DRBD-8.3-data-packets.svg
> >   - Documentation/admin-guide/blockdev/drbd/DRBD-data-packages.svg
> > 
> > If you have Inkscape installed as well, convert(1) delegates SVG
> > decoding to inkscape(1) rather than to rsvg-convert(1) and SVGs listed
> > above can be rendered properly.
> > 
> > So if Inkscape is required for converting those SVGs properly, why not
> > use it directly in the first place?
> > 
> > Patches 3/4 and 4/4 add code to utilize inkscape(1) for SVG -> PDF
> > conversion when it is available.  They don't modify any existing
> > requirements for kernel-doc.
> > 
> > Patch 3/4 adds the alternative route of SVG -> PDF conversion by
> > inkscape(1).
> > Patch 4/4 delegates warning messages from inkscape(1) to kernellog.verbose
> > as they are likely harmless in command-line uses.
> > 
> > Pros:
> >   - Generated PDFs are of vector graphics.
> >   - Vector graphics tends to be smaller in size and looks nicer when
> >     zoomed in.
> >   - SVGs drawn by Inkscape are fully supported.
> > 
> > On systems without Inkscape, no regression is expected by these two
> > patches.
> > 
> > Changes since v1 (as of Patch 5/3) [1]:
> > 
> > - Reorder and merge patches to reduce/eliminate regression windows of
> >   raster-image PDF and stderr redirection.
> >     v1        v2
> >     1/3       1/4
> >     4/3       2/4
> >     2/3       3/4
> >     3/3+5/3   4/4
> > 
> > - Massage kernellog.verbose/warn messages. They now show command(s)
> >   used in DOT -> PDF conversion.
> > 
> > - Pass actual exit code of inkscape(1) to kernellog.warn.
> > 
> > FWIW, diff of v1 vs. v2 follows:
> > 
> > --------------------------------------------------------------  
> [...]
> 



Thanks,
Mauro

Re: [PATCH v2 0/4] docs: sphinx/kfigure.py: Improve conversion to PDF

[Akira Yokosawa]

On Fri, 14 Jan 2022 09:45:35 +0100,
Mauro Carvalho Chehab wrote:
> Em Fri, 7 Jan 2022 22:45:47 +0900
> Akira Yokosawa <akiyks@gmail.com> escreveu:
> 
>> On Wed, 29 Dec 2021 20:42:00 +0900, Akira Yokosawa wrote:
>>> This patch set improves conversions of DOT -> PDF and SVG -> PDF
>>> for PDF docs.  
>>
>> Gentle ping.
>>
>> Mauro, any comments?
> 
> Sorry, have been busy those days with not much time to test it,
> and I'm not expecting any time to test it on the next couple of
> weeks.

Mauro, no need of apologies.

We are in the middle of the v5.17 merge window, and I think of this
series as a v5.18 material.
Which means it won't be merged into doc-next until v5.17-rc5 or -rc6
(mid March or so), unless Jon thinks otherwise.

I'd like to have your Tested-by: and/or Reviewed-by: tags if you
could manage to spare time for testing.

> 
> The main concern from my last review is that inkscape is too noisy 
> (well, frankly, textlive is also too noisy).

You mean the harmless warning msgs delegated to kernellog.verbose()?
Or the direct redirection to /dev/null as of v1's 3/3?

>                                               If this was solved
> on a nice way,

An excerpt of messages from Inkscape is as follows
(on Debian bullseye, with "make SPHINXOPTS=-v SPHINXDIRS=doc-guide pdfdocs"):

----------
convert SVG to: {out}/svg_image.pdf
Warning msg from inkscape(1) (likely harmless):
Unable to init server: Could not connect: Connection refused
Failed to get connection
** (inkscape:119): CRITICAL **: 01:58:03.988: dbus_g_proxy_new_for_name: assertion 'connection != NULL' failed

** (inkscape:119): CRITICAL **: 01:58:03.988: dbus_g_proxy_call: assertion 'DBUS_IS_G_PROXY (proxy)' failed

** (inkscape:119): CRITICAL **: 01:58:03.988: dbus_g_connection_register_g_object: assertion 'connection != NULL' failed

** (inkscape:119): WARNING **: 01:58:04.300: Fonts dir '/usr/share/inkscape/fonts' does not exist and will be ignored.

assert best format for: hello.dot
convert DOT to: {out}/hello.pdf
Warning msg from inkscape(1) (likely harmless):
Unable to init server: Could not connect: Connection refused
Failed to get connection
** (inkscape:129): CRITICAL **: 01:58:04.454: dbus_g_proxy_new_for_name: assertion 'connection != NULL' failed

** (inkscape:129): CRITICAL **: 01:58:04.454: dbus_g_proxy_call: assertion 'DBUS_IS_G_PROXY (proxy)' failed

** (inkscape:129): CRITICAL **: 01:58:04.454: dbus_g_connection_register_g_object: assertion 'connection != NULL' failed

** (inkscape:129): WARNING **: 01:58:04.628: Fonts dir '/usr/share/inkscape/fonts' does not exist and will be ignored.
[...]
----------

On Fedora 35, I don't see any message from Inkscape.

Is this acceptable to you?

>                 and provided that the output files on both html and
> pdf are working fine with those conversions, I don't have any 
> objections to this series.

This series affects only PDF.
My test coverage is not perfect, but I don't expect any regression
in "make pdfdocs" or "make htmldocs".

        Thanks, Akira

> 
> Regards,
> Mauro
> 
[...]

Re: [PATCH v2 0/4] docs: sphinx/kfigure.py: Improve conversion to PDF

[Jonathan Corbet]

Akira Yokosawa <akiyks@gmail.com> writes:

> We are in the middle of the v5.17 merge window, and I think of this
> series as a v5.18 material.
> Which means it won't be merged into doc-next until v5.17-rc5 or -rc6
> (mid March or so), unless Jon thinks otherwise.

I'd rather merge it rather sooner than that if possible; 5.18 stuff will
start going into docs-next shortly.

Thanks,

jon

Re: [PATCH v2 0/4] docs: sphinx/kfigure.py: Improve conversion to PDF

[Akira Yokosawa]

On Sat, 15 Jan 2022 14:17:30 -0700,
Jonathan Corbet wrote:
> Akira Yokosawa <akiyks@gmail.com> writes:
> 
>> We are in the middle of the v5.17 merge window, and I think of this
>> series as a v5.18 material.
>> Which means it won't be merged into doc-next until v5.17-rc5 or -rc6
>> (mid March or so), unless Jon thinks otherwise.
> 
> I'd rather merge it rather sooner than that if possible; 5.18 stuff will
> start going into docs-next shortly.

Jon,

Hearing no objection since mid January, I think it is time for you to
merge this series into docs-next.

        Thanks, Akira

> 
> Thanks,
> 
> jon

Re: [PATCH v2 0/4] docs: sphinx/kfigure.py: Improve conversion to PDF

[Jonathan Corbet]

Akira Yokosawa <akiyks@gmail.com> writes:

> On Sat, 15 Jan 2022 14:17:30 -0700,
> Jonathan Corbet wrote:
>> Akira Yokosawa <akiyks@gmail.com> writes:
>> 
>>> We are in the middle of the v5.17 merge window, and I think of this
>>> series as a v5.18 material.
>>> Which means it won't be merged into doc-next until v5.17-rc5 or -rc6
>>> (mid March or so), unless Jon thinks otherwise.
>> 
>> I'd rather merge it rather sooner than that if possible; 5.18 stuff will
>> start going into docs-next shortly.
>
> Jon,
>
> Hearing no objection since mid January, I think it is time for you to
> merge this series into docs-next.

That has been done - thanks for your work on this!

Since you seem to have a reasonably good understanding of the
PDF-generation side of things...I noted as I was testing this work that
we are still getting the "restricted \write18 enabled" messages.  I
asked about this years ago, and nobody seemed to have a clue of what was
enabling it.  I'd *really* like to turn that off if possible; do you
have any idea what's going on there?

Thanks,

jon

Re: [PATCH v2 0/4] docs: sphinx/kfigure.py: Improve conversion to PDF

[Akira Yokosawa]

On Wed, 09 Feb 2022 17:30:38 -0700,
Jonathan Corbet wrote:
> Akira Yokosawa <akiyks@gmail.com> writes:
> 
>> On Sat, 15 Jan 2022 14:17:30 -0700,
>> Jonathan Corbet wrote:
>>> Akira Yokosawa <akiyks@gmail.com> writes:
>>>
>>>> We are in the middle of the v5.17 merge window, and I think of this
>>>> series as a v5.18 material.
>>>> Which means it won't be merged into doc-next until v5.17-rc5 or -rc6
>>>> (mid March or so), unless Jon thinks otherwise.
>>>
>>> I'd rather merge it rather sooner than that if possible; 5.18 stuff will
>>> start going into docs-next shortly.
>>
>> Jon,
>>
>> Hearing no objection since mid January, I think it is time for you to
>> merge this series into docs-next.
> 
> That has been done - thanks for your work on this!

You are welcome!

> 
> Since you seem to have a reasonably good understanding of the
> PDF-generation side of things...I noted as I was testing this work that
> we are still getting the "restricted \write18 enabled" messages.  I
> asked about this years ago, and nobody seemed to have a clue of what was
> enabling it.  I'd *really* like to turn that off if possible; do you
> have any idea what's going on there?

I think I know the answer.
The message can be silenced by adding an option to xelatex.

Will send a patch soon.

        Thanks, Akira

> 
> Thanks,
> 
> jon