[PATCH] doc: flat-table directive

[PATCH] doc: flat-table directive

[Markus Heiser]

Hi Jonathan,

this is my flat-table patch on top of your docs-next branch / we discussed on
the ML[1]

[1] http://mid.gmane.org/573D454C-2F55-4DD7-9C16-00B3897AEFB0@darmarit.de

Markus Heiser (1):
  doc-rst: flat-table directive - initial implementation

 Documentation/conf.py                  |   2 +-
 Documentation/kernel-documentation.rst |  85 ++++++++
 Documentation/sphinx/rstFlatTable.py   | 365 +++++++++++++++++++++++++++++++++
 3 files changed, 451 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/sphinx/rstFlatTable.py

-- 
sphinx-4.8

[PATCH] doc-rst: flat-table directive - initial implementation

[Markus Heiser]

Implements the reST flat-table directive.

The ``flat-table`` is a double-stage list similar to the ``list-table`` with
some additional features:

* column-span: with the role ``cspan`` a cell can be extended through
  additional columns

* row-span: with the role ``rspan`` a cell can be extended through
  additional rows

* auto span rightmost cell of a table row over the missing cells on the right
  side of that table-row.  With Option ``:fill-cells:`` this behavior can
  changed from *auto span* to *auto fill*, which automaticly inserts (empty)

list tables

  The *list tables* formats are double stage lists. Compared to the
  ASCII-art they migth be less comfortable for readers of the
  text-files. Their advantage is, that they are easy to create/modify
  and that the diff of a modification is much more meaningfull, because
  it is limited to the modified content.

The initial implementation was taken from the sphkerneldoc project [1]

[1] https://github.com/return42/sphkerneldoc/commits/master/scripts/site-python/linuxdoc/rstFlatTable.py

Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
---
 Documentation/conf.py                  |   2 +-
 Documentation/kernel-documentation.rst |  85 ++++++++
 Documentation/sphinx/rstFlatTable.py   | 365 +++++++++++++++++++++++++++++++++
 3 files changed, 451 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/sphinx/rstFlatTable.py

diff --git a/Documentation/conf.py b/Documentation/conf.py
index 6cc41a0..792b633 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -28,7 +28,7 @@ sys.path.insert(0, os.path.abspath('sphinx'))
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['kernel-doc']
+extensions = ['kernel-doc', 'rstFlatTable']
 
 # Gracefully handle missing rst2pdf.
 try:
diff --git a/Documentation/kernel-documentation.rst b/Documentation/kernel-documentation.rst
index ad3506c..6066c74 100644
--- a/Documentation/kernel-documentation.rst
+++ b/Documentation/kernel-documentation.rst
@@ -107,6 +107,91 @@ Here are some specific guidelines for the kernel documentation:
   the order as encountered."), having the higher levels the same overall makes
   it easier to follow the documents.
 
+list tables
+-----------
+
+We recommend to use *list table* formats. The *list table* formats are double
+stage lists. Compared to the ASCII-art they migth not be as comfortable for
+readers of the text-files. Their advantage is, that they are easy to
+create/modify and that the diff of a modification is much more meaningfull,
+because it is limited to the modified content.
+
+The ``flat-table`` is a double-stage list similar to the ``list-table`` with
+some additional features:
+
+* column-span: with the role ``cspan`` a cell can be extended through
+  additional columns
+
+* row-span: with the role ``rspan`` a cell can be extended through
+  additional rows
+
+* auto span rightmost cell of a table row over the missing cells on the right
+  side of that table-row.  With Option ``:fill-cells:`` this behavior can
+  changed from *auto span* to *auto fill*, which automaticly inserts (empty)
+  cells instead of spanning the last cell.
+
+options:
+
+* ``:header-rows:``   [int] count of header rows
+* ``:stub-columns:``  [int] count of stub columns
+* ``:widths:``        [[int] [int] ... ] widths of columns
+* ``:fill-cells:``    instead of autospann missing cells, insert missing cells
+
+roles:
+
+* ``:cspan:`` [int] additionale columns (*morecols*)
+* ``:rspan:`` [int] additionale rows (*morerows*)
+
+The example below shows how to use this markup.  The first level of the staged
+list is the *table-row*. In the *table-row* there is only one markup allowed,
+the list of the cells in this *table-row*. Exception are *comments* ( ``..`` )
+and *targets* (e.g. a ref to ``:ref:`last row <last row>``` / :ref:`last row
+<last row>`).
+
+.. code-block:: rst
+
+   .. flat-table:: table title
+      :widths: 2 1 1 3
+
+      * - head col 1
+        - head col 2
+        - head col 3
+        - head col 4
+
+      * - column 1
+        - field 1.1
+        - field 1.2 with autospan
+
+      * - column 2
+        - field 2.1
+        - :rspan:`1` :cspan:`1` field 2.2 - 3.3
+
+      * .. _`last row`:
+
+        - column 3
+
+Rendered as:
+
+   .. flat-table:: table title
+      :widths: 2 1 1 3
+
+      * - head col 1
+        - head col 2
+        - head col 3
+        - head col 4
+
+      * - column 1
+        - field 1.1
+        - field 1.2 with autospan
+
+      * - column 2
+        - field 2.1
+        - :rspan:`1` :cspan:`1` field 2.2 - 3.3
+
+      * .. _`last row`:
+
+        - column 3
+
 
 Including kernel-doc comments
 =============================
diff --git a/Documentation/sphinx/rstFlatTable.py b/Documentation/sphinx/rstFlatTable.py
new file mode 100644
index 0000000..26db852
--- /dev/null
+++ b/Documentation/sphinx/rstFlatTable.py
@@ -0,0 +1,365 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8; mode: python -*-
+# pylint: disable=C0330, R0903, R0912
+
+u"""
+    flat-table
+    ~~~~~~~~~~
+
+    Implementation of the ``flat-table`` reST-directive.
+
+    :copyright:  Copyright (C) 2016  Markus Heiser
+    :license:    GPL Version 2, June 1991 see linux/COPYING for details.
+
+    The ``flat-table`` (:py:class:`FlatTable`) is a double-stage list similar to
+    the ``list-table`` with some additional features:
+
+    * *column-span*: with the role ``cspan`` a cell can be extended through
+      additional columns
+
+    * *row-span*: with the role ``rspan`` a cell can be extended through
+      additional rows
+
+    * *auto span* rightmost cell of a table row over the missing cells on the
+      right side of that table-row.  With Option ``:fill-cells:`` this behavior
+      can changed from *auto span* to *auto fill*, which automaticly inserts
+      (empty) cells instead of spanning the last cell.
+
+    Options:
+
+    * header-rows:   [int] count of header rows
+    * stub-columns:  [int] count of stub columns
+    * widths:        [[int] [int] ... ] widths of columns
+    * fill-cells:    instead of autospann missing cells, insert missing cells
+
+    roles:
+
+    * cspan: [int] additionale columns (*morecols*)
+    * rspan: [int] additionale rows (*morerows*)
+"""
+
+# ==============================================================================
+# imports
+# ==============================================================================
+
+import sys
+
+from docutils import nodes
+from docutils.parsers.rst import directives, roles
+from docutils.parsers.rst.directives.tables import Table
+from docutils.utils import SystemMessagePropagation
+
+# ==============================================================================
+# common globals
+# ==============================================================================
+
+# The version numbering follows numbering of the specification
+# (Documentation/books/kernel-doc-HOWTO).
+__version__  = '1.0'
+
+PY3 = sys.version_info[0] == 3
+PY2 = sys.version_info[0] == 2
+
+if PY3:
+    # pylint: disable=C0103, W0622
+    unicode     = str
+    basestring  = str
+
+# ==============================================================================
+def setup(app):
+# ==============================================================================
+
+    app.add_directive("flat-table", FlatTable)
+    roles.register_local_role('cspan', c_span)
+    roles.register_local_role('rspan', r_span)
+
+# ==============================================================================
+def c_span(name, rawtext, text, lineno, inliner, options=None, content=None):
+# ==============================================================================
+    # pylint: disable=W0613
+
+    options  = options if options is not None else {}
+    content  = content if content is not None else []
+    nodelist = [colSpan(span=int(text))]
+    msglist  = []
+    return nodelist, msglist
+
+# ==============================================================================
+def r_span(name, rawtext, text, lineno, inliner, options=None, content=None):
+# ==============================================================================
+    # pylint: disable=W0613
+
+    options  = options if options is not None else {}
+    content  = content if content is not None else []
+    nodelist = [rowSpan(span=int(text))]
+    msglist  = []
+    return nodelist, msglist
+
+
+# ==============================================================================
+class rowSpan(nodes.General, nodes.Element): pass # pylint: disable=C0103,C0321
+class colSpan(nodes.General, nodes.Element): pass # pylint: disable=C0103,C0321
+# ==============================================================================
+
+# ==============================================================================
+class FlatTable(Table):
+# ==============================================================================
+
+    u"""FlatTable (``flat-table``) directive"""
+
+    option_spec = {
+        'name': directives.unchanged
+        , 'class': directives.class_option
+        , 'header-rows': directives.nonnegative_int
+        , 'stub-columns': directives.nonnegative_int
+        , 'widths': directives.positive_int_list
+        , 'fill-cells' : directives.flag }
+
+    def run(self):
+
+        if not self.content:
+            error = self.state_machine.reporter.error(
+                'The "%s" directive is empty; content required.' % self.name,
+                nodes.literal_block(self.block_text, self.block_text),
+                line=self.lineno)
+            return [error]
+
+        title, messages = self.make_title()
+        node = nodes.Element()          # anonymous container for parsing
+        self.state.nested_parse(self.content, self.content_offset, node)
+
+        tableBuilder = ListTableBuilder(self)
+        tableBuilder.parseFlatTableNode(node)
+        tableNode = tableBuilder.buildTableNode()
+        # SDK.CONSOLE()  # print --> tableNode.asdom().toprettyxml()
+        if title:
+            tableNode.insert(0, title)
+        return [tableNode] + messages
+
+
+# ==============================================================================
+class ListTableBuilder(object):
+# ==============================================================================
+
+    u"""Builds a table from a double-stage list"""
+
+    def __init__(self, directive):
+        self.directive = directive
+        self.rows      = []
+        self.max_cols  = 0
+
+    def buildTableNode(self):
+
+        colwidths    = self.directive.get_column_widths(self.max_cols)
+        stub_columns = self.directive.options.get('stub-columns', 0)
+        header_rows  = self.directive.options.get('header-rows', 0)
+
+        table = nodes.table()
+        tgroup = nodes.tgroup(cols=len(colwidths))
+        table += tgroup
+
+
+        for colwidth in colwidths:
+            colspec = nodes.colspec(colwidth=colwidth)
+            # FIXME: It seems, that the stub method only works well in the
+            # absence of rowspan (observed by the html buidler, the docutils-xml
+            # build seems OK).  This is not extraordinary, because there exists
+            # no table directive (except *this* flat-table) which allows to
+            # define coexistent of rowspan and stubs (there was no use-case
+            # before flat-table). This should be reviewed (later).
+            if stub_columns:
+                colspec.attributes['stub'] = 1
+                stub_columns -= 1
+            tgroup += colspec
+        stub_columns = self.directive.options.get('stub-columns', 0)
+
+        if header_rows:
+            thead = nodes.thead()
+            tgroup += thead
+            for row in self.rows[:header_rows]:
+                thead += self.buildTableRowNode(row)
+
+        tbody = nodes.tbody()
+        tgroup += tbody
+
+        for row in self.rows[header_rows:]:
+            tbody += self.buildTableRowNode(row)
+        return table
+
+    def buildTableRowNode(self, row_data, classes=None):
+        classes = [] if classes is None else classes
+        row = nodes.row()
+        for cell in row_data:
+            if cell is None:
+                continue
+            cspan, rspan, cellElements = cell
+
+            attributes = {"classes" : classes}
+            if rspan:
+                attributes['morerows'] = rspan
+            if cspan:
+                attributes['morecols'] = cspan
+            entry = nodes.entry(**attributes)
+            entry.extend(cellElements)
+            row += entry
+        return row
+
+    def raiseError(self, msg):
+        error =  self.directive.state_machine.reporter.error(
+            msg
+            , nodes.literal_block(self.directive.block_text
+                                  , self.directive.block_text)
+            , line = self.directive.lineno )
+        raise SystemMessagePropagation(error)
+
+    def parseFlatTableNode(self, node):
+        u"""parses the node from a :py:class:`FlatTable` directive's body"""
+
+        if len(node) != 1 or not isinstance(node[0], nodes.bullet_list):
+            self.raiseError(
+                'Error parsing content block for the "%s" directive: '
+                'exactly one bullet list expected.' % self.directive.name )
+
+        for rowNum, rowItem in enumerate(node[0]):
+            row = self.parseRowItem(rowItem, rowNum)
+            self.rows.append(row)
+        self.roundOffTableDefinition()
+
+    def roundOffTableDefinition(self):
+        u"""Round off the table definition.
+
+        This method rounds off the table definition in :py:member:`rows`.
+
+        * This method inserts the needed ``None`` values for the missing cells
+        arising from spanning cells over rows and/or columns.
+
+        * recount the :py:member:`max_cols`
+
+        * Autospan or fill (option ``fill-cells``) missing cells on the right
+          side of the table-row
+        """
+
+        y = 0
+        while y < len(self.rows):
+            x = 0
+
+            while x < len(self.rows[y]):
+                cell = self.rows[y][x]
+                if cell is None:
+                    x += 1
+                    continue
+                cspan, rspan = cell[:2]
+                # handle colspan in current row
+                for c in range(cspan):
+                    try:
+                        self.rows[y].insert(x+c+1, None)
+                    except: # pylint: disable=W0702
+                        # the user sets ambiguous rowspans
+                        pass # SDK.CONSOLE()
+                # handle colspan in spanned rows
+                for r in range(rspan):
+                    for c in range(cspan + 1):
+                        try:
+                            self.rows[y+r+1].insert(x+c, None)
+                        except: # pylint: disable=W0702
+                            # the user sets ambiguous rowspans
+                            pass # SDK.CONSOLE()
+                x += 1
+            y += 1
+
+        # Insert the missing cells on the right side. For this, first
+        # re-calculate the max columns.
+
+        for row in self.rows:
+            if self.max_cols < len(row):
+                self.max_cols = len(row)
+
+        # fill with empty cells or cellspan?
+
+        fill_cells = False
+        if 'fill-cells' in self.directive.options:
+            fill_cells = True
+
+        for row in self.rows:
+            x =  self.max_cols - len(row)
+            if x and not fill_cells:
+                if row[-1] is None:
+                    row.append( ( x - 1, 0, []) )
+                else:
+                    cspan, rspan, content = row[-1]
+                    row[-1] = (cspan + x, rspan, content)
+            elif x and fill_cells:
+                for i in range(x):
+                    row.append( (0, 0, nodes.comment()) )
+
+    def pprint(self):
+        # for debugging
+        retVal = "[   "
+        for row in self.rows:
+            retVal += "[ "
+            for col in row:
+                if col is None:
+                    retVal += ('%r' % col)
+                    retVal += "\n    , "
+                else:
+                    content = col[2][0].astext()
+                    if len (content) > 30:
+                        content = content[:30] + "..."
+                    retVal += ('(cspan=%s, rspan=%s, %r)'
+                               % (col[0], col[1], content))
+                    retVal += "]\n    , "
+            retVal = retVal[:-2]
+            retVal += "]\n  , "
+        retVal = retVal[:-2]
+        return retVal + "]"
+
+    def parseRowItem(self, rowItem, rowNum):
+        row = []
+        childNo = 0
+        error   = False
+        cell    = None
+        target  = None
+
+        for child in rowItem:
+            if (isinstance(child , nodes.comment)
+                or isinstance(child, nodes.system_message)):
+                pass
+            elif isinstance(child , nodes.target):
+                target = child
+            elif isinstance(child, nodes.bullet_list):
+                childNo += 1
+                cell = child
+            else:
+                error = True
+                break
+
+        if childNo != 1 or error:
+            self.raiseError(
+                'Error parsing content block for the "%s" directive: '
+                'two-level bullet list expected, but row %s does not '
+                'contain a second-level bullet list.'
+                % (self.directive.name, rowNum + 1))
+
+        for cellItem in cell:
+            cspan, rspan, cellElements = self.parseCellItem(cellItem)
+            if target is not None:
+                cellElements.insert(0, target)
+            row.append( (cspan, rspan, cellElements) )
+        return row
+
+    def parseCellItem(self, cellItem):
+        # search and remove cspan, rspan colspec from the first element in
+        # this listItem (field).
+        cspan = rspan = 0
+        if not len(cellItem):
+            return cspan, rspan, []
+        for elem in cellItem[0]:
+            if isinstance(elem, colSpan):
+                cspan = elem.get("span")
+                elem.parent.remove(elem)
+                continue
+            if isinstance(elem, rowSpan):
+                rspan = elem.get("span")
+                elem.parent.remove(elem)
+                continue
+        return cspan, rspan, cellItem[:]
-- 
sphinx-4.8

Re: [PATCH] doc: flat-table directive

[Jonathan Corbet]

On Thu, 30 Jun 2016 14:00:21 +0200
Markus Heiser <markus.heiser@darmarIT.de> wrote:

> this is my flat-table patch on top of your docs-next branch / we discussed on
> the ML

Hmm...  we don't have an official kernel coding style for Python, but if
we did, I'd sure like it to be a lot more like the usual Python
conventions.

The added documentation describes the "widths" parameter, but does not
tell the reader which units are used.

There were also a lot of typos and misspellings in the documentation
addition.

Anyway, I don't want to delay this work, so I have gone ahead and applied
it; I fixed the spelling errors on the way.  When you get a chance, can I
get a doc tweak for :widths:?

Thanks for separating this work out,

jon

Re: [PATCH] doc: flat-table directive

[Mauro Carvalho Chehab]

Hi Markus/Jon,

Em Thu, 30 Jun 2016 13:05:11 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> On Thu, 30 Jun 2016 14:00:21 +0200
> Markus Heiser <markus.heiser@darmarIT.de> wrote:
> 
> > this is my flat-table patch on top of your docs-next branch / we discussed on
> > the ML  
> 
> Hmm...  we don't have an official kernel coding style for Python, but if
> we did, I'd sure like it to be a lot more like the usual Python
> conventions.
> 
> The added documentation describes the "widths" parameter, but does not
> tell the reader which units are used.
> 
> There were also a lot of typos and misspellings in the documentation
> addition.
> 
> Anyway, I don't want to delay this work, so I have gone ahead and applied
> it; I fixed the spelling errors on the way. 

Thank you!

I rebased my tree to apply on the top of the master branch of
the media tree and to use the latest changeset from Jon's tree.

I kept it at the same place:

https://git.linuxtv.org/mchehab/experimental.git/log/?h=docs-next

I'm starting to manually edit and fix it right away. I'll later merge
it as one additional topic branch on my main tree, once I'm confident
that it is ok.

Markus,

Thank you very much for your hard work!

> When you get a chance, can I
> get a doc tweak for :widths:?
> 
> Thanks for separating this work out,
> 
> jon


-- 
Thanks,
Mauro

Re: [PATCH] doc: flat-table directive

[Mauro Carvalho Chehab]

Em Thu, 30 Jun 2016 13:05:11 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Anyway, I don't want to delay this work, so I have gone ahead and applied
> it; 

Got already one issue... Maybe on Jeni's changes to the makefiles...

I want to be able to compile just the book I'm working. Using the usual
syntax to build just one book worked: the build from the *.rst files
succeeded. Yet, it tried to compile it also as DocBook, causing an
error at the end.

See the logs below:

$ make DOCBOOKS=linux_tv htmldocs
  SPHINX  htmldocs
Running Sphinx v1.3.1
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] linux_tv/media/dvb/dvbapi                             
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] linux_tv/media/dvb/dvbapi                              
generating indices... genindex
writing additional pages... search
copying images... [100%] linux_tv/media_api_files/typical_media_device.svg      
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded.
make[1]: *** No rule to make target 'Documentation/DocBook/linux_tv', needed by 'htmldocs'.  Pare.
Makefile:1420: recipe for target 'htmldocs' failed
make: *** [htmldocs] Error 2


-- 
Thanks,
Mauro

Re: [PATCH] doc: flat-table directive

[Markus Heiser]

Am 30.06.2016 um 21:05 schrieb Jonathan Corbet <corbet@lwn.net>:

> On Thu, 30 Jun 2016 14:00:21 +0200
> Markus Heiser <markus.heiser@darmarIT.de> wrote:
> 
>> this is my flat-table patch on top of your docs-next branch / we discussed on
>> the ML
> 
> Hmm...  we don't have an official kernel coding style for Python, but if
> we did, I'd sure like it to be a lot more like the usual Python
> conventions.

Please help me, what in the patch is not usual Python? 

There is a "Style Guide for Python Code" PEP 8 [1] from which
I (hopefully) differ only in marginal details. I use the 
pylint [3] which checks PEP8 style, may I switched of to much?
E.g. the "one-liner classes" of mine is not PEP8 style, is this what
you mean? Or is it about the (comma) operators in front of
a new line [2]? Could you give me more detail hints / thanks.

[1] https://www.python.org/dev/peps/pep-0008/
[2] https://www.python.org/dev/peps/pep-0008/#should-a-line-break-before-or-after-a-binary-operator
[3] https://www.pylint.org/

> The added documentation describes the "widths" parameter, but does not
> tell the reader which units are used.

It has no units, it's relativ. Sorry, I implied that it is a common practice
on table layouts [4]:

  """A comma- or space-separated list of relative column widths. 
     The default is equal-width columns (100%/#columns)."""

[4] http://docutils.sourceforge.net/docs/ref/rst/directives.html#list-table 

> There were also a lot of typos and misspellings in the documentation
> addition.

Sorry, I did it in a hurry because Mauro needs it promptly. I will improve 
myself in the future.

> Anyway, I don't want to delay this work, so I have gone ahead and applied
> it; I fixed the spelling errors on the way.

Thanks a lot!

>  When you get a chance, can I
> get a doc tweak for :widths:?

Yes, I will send you a patch later.

-- Markus --

> Thanks for separating this work out,
> 
> jon

Re: [PATCH] doc: flat-table directive

[Markus Heiser]

Am 30.06.2016 um 21:44 schrieb Mauro Carvalho Chehab <mchehab@osg.samsung.com>:

> Em Thu, 30 Jun 2016 13:05:11 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
> 
>> Anyway, I don't want to delay this work, so I have gone ahead and applied
>> it; 
> 
> Got already one issue... Maybe on Jeni's changes to the makefiles...
> 
> I want to be able to compile just the book I'm working.

This feature does not exists (yet) for the reST content. Jani and I
discussed it on the ML, but for the start he didn't want to have
books (aka sphinx sub-projects). Anyway, I don't want to restart
the discussion again, just the fact for information: no books
in reST, there is only one HTML site which includes all reST content.

> Using the usual
> syntax to build just one book worked: the build from the *.rst files
> succeeded. Yet, it tried to compile it also as DocBook, causing an
> error at the end.
> 
> See the logs below:
> 
> $ make DOCBOOKS=linux_tv htmldocs
>  SPHINX  htmldocs

As far as I know, there is only the one html-target "htmldocs" building
all HTML from reST & DocBook content. Selecting DocBook-XML books with the
environment "DOCBOOKS=..." is new for me. Did this feature exists in the 
past? I tried it with v4.6 but no success.

> Running Sphinx v1.3.1
> loading pickled environment... done
> building [mo]: targets for 0 po files that are out of date
> building [html]: targets for 1 source files that are out of date
> updating environment: 0 added, 1 changed, 0 removed
> reading sources... [100%] linux_tv/media/dvb/dvbapi                             
> looking for now-outdated files... none found
> pickling environment... done
> checking consistency... done
> preparing documents... done
> writing output... [100%] linux_tv/media/dvb/dvbapi                              
> generating indices... genindex
> writing additional pages... search
> copying images... [100%] linux_tv/media_api_files/typical_media_device.svg      
> copying static files... done
> copying extra files... done
> dumping search index in English (code: en) ... done
> dumping object inventory... done
> build succeeded.
> make[1]: *** No rule to make target 'Documentation/DocBook/linux_tv', needed by 'htmldocs'.  Pare.
> Makefile:1420: recipe for target 'htmldocs' failed
> make: *** [htmldocs] Error 2
> 
> -- 
> Thanks,
> Mauro


-- Markus --

Re: [PATCH] doc: flat-table directive

[Mauro Carvalho Chehab]

Em Fri, 01 Jul 2016 10:44:27 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 30.06.2016 um 21:44 schrieb Mauro Carvalho Chehab <mchehab@osg.samsung.com>:
> 
> > Em Thu, 30 Jun 2016 13:05:11 -0600
> > Jonathan Corbet <corbet@lwn.net> escreveu:
> >   
> >> Anyway, I don't want to delay this work, so I have gone ahead and applied
> >> it;   
> > 
> > Got already one issue... Maybe on Jeni's changes to the makefiles...
> > 
> > I want to be able to compile just the book I'm working.  
> 
> This feature does not exists (yet) for the reST content. Jani and I
> discussed it on the ML, but for the start he didn't want to have
> books (aka sphinx sub-projects). Anyway, I don't want to restart
> the discussion again, just the fact for information: no books
> in reST, there is only one HTML site which includes all reST content.
> 
> > Using the usual
> > syntax to build just one book worked: the build from the *.rst files
> > succeeded. Yet, it tried to compile it also as DocBook, causing an
> > error at the end.
> > 
> > See the logs below:
> > 
> > $ make DOCBOOKS=linux_tv htmldocs
> >  SPHINX  htmldocs  
> 
> As far as I know, there is only the one html-target "htmldocs" building
> all HTML from reST & DocBook content. Selecting DocBook-XML books with the
> environment "DOCBOOKS=..." is new for me. Did this feature exists in the 
> past? I tried it with v4.6 but no success.

It exists, and I rely on such feature when merging patches.
Basically, for each patch I merge on my tree, I compile it.
if make returns an error, I play a buzz audio and I reject the
patch. I do the same for documentation: if the patch touches a file
listed at device_drivers.xml, I run this script:

	rm Documentation/DocBook/device-drivers.aux.xml  Documentation/DocBook/device-drivers.xml
	echo "Building device-drivers.xml"
	make DOCBOOKS=device-drivers.xml htmldocs 2>&1
	if [ "$?" != "0" ]; then
		echo -en '\e[0;31m'
		echo -e "***********\n***ERROR***\n***********\n\n"
		echo -en '\e[0;37m'
		play ~/sounds/pipe.wav 2>/dev/null >/dev/null
		exit -1 
	fi

And, if it touches on any file under Documentation/DocBook/media,
I run this script:

	make cleanmediadocs
	make DOCBOOKS=media_api.xml htmldocs 2>&1 | grep -v "element.*: validity error : ID .* already defined"
	echo
	echo "Do some pedantic checks and generate DocBook/media/media_api.html without chunks"
	echo
	xmllint --noent --postvalid "$PWD/Documentation/DocBook/media_api.xml" >/tmp/x.xml 2>/dev/null
	xmllint --noent --postvalid --noout /tmp/x.xml
	xmlto html-nochunks -m ./Documentation/DocBook/stylesheet.xsl -o Documentation/DocBook/media Documentation/DocBook/media_api.xml >/dev/null 2>&1
	if [ "$?" != "0" ]; then
		echo -en '\e[0;31m'
		echo -e "***********\n***ERROR***\n***********\n\n"
		echo -en '\e[0;37m'
		play ~/sounds/pipe.wav 2>/dev/null >/dev/null
		exit -1 
	fi

In the case of the DocBook, as we have all header files using references
to the code (and such references are dynamically created when make htmldocs
run), if someone adds a new API and forgets to send documentation, or if
it broke the DocBook, I'll also hear an annoying beep, and will reject
the patch.

Not being able to compile just one docbook is a regression and breaks
my process. This needs to be fixed.

Btw, yesterday, I tried to add references to a C code, at video.rst,
just like we did with DocBook:

.. code-block:: c
    :caption: Example 2: Switching to the first video input

    int index;

    index = 0;

    if (-1 == ioctl(fd, :ref:`VIDIOC_S_INPUT <vidioc-s-input>`, &index)) {
        perror("VIDIOC_S_INPUT");
        exit(EXIT_FAILURE);
    }

But Sphinx didn't allow doing it. I was unable to find any syntax
on it that would make Sphinx use a monospaced font but still parse
the references at the code. While for those small examples this would
be ok, This is something that we do want for the header files that
we put at V4L and DVB annexes. For DocBook, we do a lot of things like
this at the DocBook/media/Makefile:
	sed -e "s/\(enum *\)v4l2_mpeg_cx2341x_video_\([a-z]*_spatial_filter_type\)/\1<link linkend=\"\2\">v4l2_mpeg_cx2341x_video_\2<\/link>/g" videodev2.h

The actual code is a way more complex, but basically the idea is that
it escapes anything that DocBook might interpret as a command, and adds
<link> tags for every enum, typedef, ioctl, struct, syscall and define
it founds at the header files. the xmllint will produce errors when
links are not solved, and we'll be able to detect that the API is not
fully documented.

We need to do a similar process with Sphinx. A side effect is that, if
someone looks at the header files in the anexes, it can click on any
symbol at the API and see the full documentation.

--

Btw, in the case of the above example, I had to manually number it as
"Example 2", because I was unable to find a way with Sphinx to auto
numerate code-block captions. This is also something we want to fix,
as it is very hard to manually number things on a 600+ page document.

Regards,
Mauro

Re: [PATCH] doc: flat-table directive

[Jani Nikula]

On Fri, 01 Jul 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> Not being able to compile just one docbook is a regression and breaks
> my process. This needs to be fixed.

Do you have a regression with *DocBook XML* on docs-next now? If yes,
clearly that must be fixed ASAP.

As to Sphinx, I fear your DocBook workflows are so elaborate that we are
bound to break some of them when switching over. Personally, I don't
think that's all bad. It's an opportunity to make things better. But we
can't make all of that happen overnight for media. This is why we have
Sphinx and DocBook side by side for now.

We specifically wanted to make the switch for e.g. GPU documentation
fast, act as guinea pigs, hit the issues, and make it easier for others
in the end. We'd also love to get some of those validation things you
have, for everyone, but using Sphinx instead of a bunch of scripts and
sed on the input files.

This is also one of the reasons why I so much want to keep everything
behind one configuration file, and build everything in the Sphinx
toolchain. To keep it all more uniform, to not duplicate stuff, and not
deviate to some silos like we've done in the past. I think when we have
things working, we can add dedicated config files for the select few
things that have additional special needs. Media is probably one of
them. But that said, I think we should be able to keep including that to
the main documentation build too.

Also, many thanks for digging into this and reporting issues that you do
face.


BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH] doc: flat-table directive

[Markus Heiser]

Am 01.07.2016 um 12:44 schrieb Jani Nikula <jani.nikula@intel.com>:

> On Fri, 01 Jul 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
>> Not being able to compile just one docbook is a regression and breaks
>> my process. This needs to be fixed.
> 
> Do you have a regression with *DocBook XML* on docs-next now? If yes,
> clearly that must be fixed ASAP.

Now I see the problem ...

It seems that we didn't neglected that the DocBook Makefile 
builds single books, e.g.

  make DOCBOOKS=device-drivers.xml htmldocs

Since the Makefile.sphinx does not care the environment DOCBOOKS,
the XML is build and the reST also, where only the XML content
should be build.

Jani, may could you apply a small fix on the Makefile.sphinx which
takes into account, not to run the sphinx-build if the DOCBOOK 
environment is set .. is it a solution for a start?

-- Markus --

> As to Sphinx, I fear your DocBook workflows are so elaborate that we are
> bound to break some of them when switching over. Personally, I don't
> think that's all bad. It's an opportunity to make things better. But we
> can't make all of that happen overnight for media. This is why we have
> Sphinx and DocBook side by side for now.
> 
> We specifically wanted to make the switch for e.g. GPU documentation
> fast, act as guinea pigs, hit the issues, and make it easier for others
> in the end. We'd also love to get some of those validation things you
> have, for everyone, but using Sphinx instead of a bunch of scripts and
> sed on the input files.
> 
> This is also one of the reasons why I so much want to keep everything
> behind one configuration file, and build everything in the Sphinx
> toolchain. To keep it all more uniform, to not duplicate stuff, and not
> deviate to some silos like we've done in the past. I think when we have
> things working, we can add dedicated config files for the select few
> things that have additional special needs. Media is probably one of
> them. But that said, I think we should be able to keep including that to
> the main documentation build too.
> 
> Also, many thanks for digging into this and reporting issues that you do
> face.
> 
> 
> BR,
> Jani.
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center

Re: [PATCH] doc: flat-table directive

[Jani Nikula]

On Fri, 01 Jul 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Am 01.07.2016 um 12:44 schrieb Jani Nikula <jani.nikula@intel.com>:
>
>> On Fri, 01 Jul 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
>>> Not being able to compile just one docbook is a regression and breaks
>>> my process. This needs to be fixed.
>> 
>> Do you have a regression with *DocBook XML* on docs-next now? If yes,
>> clearly that must be fixed ASAP.
>
> Now I see the problem ...
>
> It seems that we didn't neglected that the DocBook Makefile 
> builds single books, e.g.
>
>   make DOCBOOKS=device-drivers.xml htmldocs
>
> Since the Makefile.sphinx does not care the environment DOCBOOKS,
> the XML is build and the reST also, where only the XML content
> should be build.
>
> Jani, may could you apply a small fix on the Makefile.sphinx which
> takes into account, not to run the sphinx-build if the DOCBOOK 
> environment is set .. is it a solution for a start?

That is not the problem. Makefile.sphinx happily ignores DOCBOOKS. For
example 'make DOCBOOKS=kernel-api.xml htmldocs' on docs-next works just
fine for me. Sphinx build ignores DOCBOOKS, the DocBook build works just
like it always did.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[docs-next PATCH] Documentation: add cleanmediadocs to the documentation targets

[Jani Nikula]

This was broken when updating the documentation targets for the Sphinx
build, and moving from %docs target pattern to explicitly listed
targets.

Cc: Markus Heiser <markus.heiser@darmarit.de>
Cc: Mauro Carvalho Chehab <mchehab@osg.samsung.com>
Fixes: 22cba31bae9d ("Documentation/sphinx: add basic working Sphinx configuration and build")
Signed-off-by: Jani Nikula <jani.nikula@intel.com>

---

This fixes 'make cleanmediadocs' on docs-next, but I don't think this is
the problem Mauro is seeing.
---
 Documentation/Makefile.sphinx | 1 +
 Makefile                      | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index 810c11f0d37f..6c203745b6c6 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -56,6 +56,7 @@ sgmldocs:
 psdocs:
 mandocs:
 installmandocs:
+cleanmediadocs:
 
 cleandocs:
 	$(Q)rm -rf $(BUILDDIR)
diff --git a/Makefile b/Makefile
index f7f393c40886..34bc4e22ec0b 100644
--- a/Makefile
+++ b/Makefile
@@ -1414,7 +1414,7 @@ $(help-board-dirs): help-%:
 
 # Documentation targets
 # ---------------------------------------------------------------------------
-DOC_TARGETS := xmldocs sgmldocs psdocs pdfdocs htmldocs mandocs installmandocs epubdocs cleandocs
+DOC_TARGETS := xmldocs sgmldocs psdocs pdfdocs htmldocs mandocs installmandocs epubdocs cleandocs cleanmediadocs
 PHONY += $(DOC_TARGETS)
 $(DOC_TARGETS): scripts_basic FORCE
 	$(Q)$(MAKE) $(build)=scripts build_docproc build_check-lc_ctype
-- 
2.1.4

Re: [PATCH] doc: flat-table directive

[Markus Heiser]

Am 01.07.2016 um 13:56 schrieb Jani Nikula <jani.nikula@intel.com>:

> On Fri, 01 Jul 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>> Am 01.07.2016 um 12:44 schrieb Jani Nikula <jani.nikula@intel.com>:
>> 
>>> On Fri, 01 Jul 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
>>>> Not being able to compile just one docbook is a regression and breaks
>>>> my process. This needs to be fixed.
>>> 
>>> Do you have a regression with *DocBook XML* on docs-next now? If yes,
>>> clearly that must be fixed ASAP.
>> 
>> Now I see the problem ...
>> 
>> It seems that we didn't neglected that the DocBook Makefile 
>> builds single books, e.g.
>> 
>>  make DOCBOOKS=device-drivers.xml htmldocs
>> 
>> Since the Makefile.sphinx does not care the environment DOCBOOKS,
>> the XML is build and the reST also, where only the XML content
>> should be build.
>> 
>> Jani, may could you apply a small fix on the Makefile.sphinx which
>> takes into account, not to run the sphinx-build if the DOCBOOK 
>> environment is set .. is it a solution for a start?
> 
> That is not the problem. Makefile.sphinx happily ignores DOCBOOKS. For
> example 'make DOCBOOKS=kernel-api.xml htmldocs' on docs-next works just
> fine for me. Sphinx build ignores DOCBOOKS, the DocBook build works just
> like it always did.

Are you sure .. may I do something wrong::

$ make DOCBOOKS=kernel-api.xml htmldocs
  SPHINX  htmldocs
Running Sphinx v1.4.4
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
no targets are out of date.
build succeeded.
  DOCPROC Documentation/DocBook/kernel-api.xml
.//lib/crc32.c:148: warning: No description found for parameter 'tab)[256]'
.//lib/crc32.c:148: warning: Excess function parameter 'tab' description in 'crc32_le_generic'
.//lib/crc32.c:293: warning: No description found for parameter 'tab)[256]'
.//lib/crc32.c:293: warning: Excess function parameter 'tab' description in 'crc32_be_generic'
.//lib/crc32.c:1: warning: no structured comments found
.//mm/memory.c:2881: warning: No description found for parameter 'old'
  XMLREF  Documentation/DocBook/kernel-api.aux.xml
  HTML    Documentation/DocBook/kernel-api.html
  ....

In this example, the sphinx build is started as a side effect
even if it is not rebuild in this example, because nothing
in reST content has changed: "looking for now-outdated files... none found".

As far as I can see, this is done **because** Makefile.sphinx
ignores DOCBOOK.

Thanks for your feedback

 -- Markus --



> BR,
> Jani.
> 
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center

Re: [PATCH] doc: flat-table directive

[Jani Nikula]

On Thu, 30 Jun 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> Em Thu, 30 Jun 2016 13:05:11 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
>
>> Anyway, I don't want to delay this work, so I have gone ahead and applied
>> it; 
>
> Got already one issue... Maybe on Jeni's changes to the makefiles...
>
> I want to be able to compile just the book I'm working. Using the usual
> syntax to build just one book worked: the build from the *.rst files
> succeeded. Yet, it tried to compile it also as DocBook, causing an
> error at the end.
>
> See the logs below:
>
> $ make DOCBOOKS=linux_tv htmldocs
>   SPHINX  htmldocs
> Running Sphinx v1.3.1
> loading pickled environment... done
> building [mo]: targets for 0 po files that are out of date
> building [html]: targets for 1 source files that are out of date
> updating environment: 0 added, 1 changed, 0 removed
> reading sources... [100%] linux_tv/media/dvb/dvbapi                             
> looking for now-outdated files... none found
> pickling environment... done
> checking consistency... done
> preparing documents... done
> writing output... [100%] linux_tv/media/dvb/dvbapi                              
> generating indices... genindex
> writing additional pages... search
> copying images... [100%] linux_tv/media_api_files/typical_media_device.svg      
> copying static files... done
> copying extra files... done
> dumping search index in English (code: en) ... done
> dumping object inventory... done
> build succeeded.
> make[1]: *** No rule to make target 'Documentation/DocBook/linux_tv', needed by 'htmldocs'.  Pare.
> Makefile:1420: recipe for target 'htmldocs' failed
> make: *** [htmldocs] Error 2

This is just because there isn't a DocBook called linux_tv. DOCBOOKS
means nothing to Sphinx, it's only relevant for DocBook build. Top level
'make help' lists the valid values for DOCBOOKS.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH] doc: flat-table directive

[Markus Heiser]

Am 01.07.2016 um 11:38 schrieb Mauro Carvalho Chehab <mchehab@osg.samsung.com>:


> Btw, yesterday, I tried to add references to a C code, at video.rst,
> just like we did with DocBook:
> 
> .. code-block:: c
>   :caption: Example 2: Switching to the first video input
> 
>   int index;
> 
>   index = 0;
> 
>   if (-1 == ioctl(fd, :ref:`VIDIOC_S_INPUT <vidioc-s-input>`, &index)) {
>       perror("VIDIOC_S_INPUT");
>       exit(EXIT_FAILURE);
>   }
> 
> But Sphinx didn't allow doing it. I was unable to find any syntax
> on it that would make Sphinx use a monospaced font but still parse
> the references at the code. While for those small examples this would
> be ok, This is something that we do want for the header files that
> we put at V4L and DVB annexes. For DocBook, we do a lot of things like
> this at the DocBook/media/Makefile:
> 	sed -e "s/\(enum *\)v4l2_mpeg_cx2341x_video_\([a-z]*_spatial_filter_type\)/\1<link linkend=\"\2\">v4l2_mpeg_cx2341x_video_\2<\/link>/g" videodev2.h

In Sphinx, the code-block directive is a literal block, no refs or markup
will be interpreted. This is different to what you know from DocBook.
I will look for a solution, that matches your requirement.

> The actual code is a way more complex, but basically the idea is that
> it escapes anything that DocBook might interpret as a command, and adds
> <link> tags for every enum, typedef, ioctl, struct, syscall and define
> it founds at the header files. the xmllint will produce errors when
> links are not solved, and we'll be able to detect that the API is not
> fully documented.

OK, checking dead internal links might be one requirement more. Normally 
sphinx reports internal refs which are broken with a WARNING, but I have to
analyze what xmllint checks and how we could realize something similar
in sphinx / or if it is enough what sphinx reports. 

> We need to do a similar process with Sphinx. A side effect is that, if
> someone looks at the header files in the anexes, it can click on any
> symbol at the API and see the full documentation.

Yes, altogether, it seems to be a nice feature, I have to think about. But
I will point it in a separate thread.

> --
> 
> Btw, in the case of the above example, I had to manually number it as
> "Example 2", because I was unable to find a way with Sphinx to auto
> numerate code-block captions. This is also something we want to fix,
> as it is very hard to manually number things on a 600+ page document.
> 

This needs a sphinx extension, I have seen similar solutions for 
numbering figures on the net ... I will try to find a small solution.

But before I will send out some small patches which are needed 
first (IMHO). E.g. customizing the RTD theme for rendering large 
tables in HTML well and activation of useful extensions like todolist.
I have this in my "chaotic bulk" patch :-) ... I will separate it out
an send it to Jon one by one.

-- Markus --

> Regards,
> Mauro

[docs-next PATCH] Documentation/sphinx: skip build if user requested specific DOCBOOKS

[Jani Nikula]

If the user requested specific DocBooks to be built using 'make
DOCBOOKS=foo.xml htmldocs', assume no Sphinx build is desired. This
check is transitional, and can be removed once we drop the DocBook
build.

Cc: Markus Heiser <markus.heiser@darmarit.de>
Cc: Mauro Carvalho Chehab <mchehab@osg.samsung.com>
Fixes: 22cba31bae9d ("Documentation/sphinx: add basic working Sphinx configuration and build")
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/Makefile.sphinx | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index 6c203745b6c6..d8d13c92a178 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -17,6 +17,12 @@ ifeq ($(HAVE_SPHINX),0)
 	$(warning The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point to the full path of the '$(SPHINXBUILD)' executable.)
 	@echo "  SKIP    Sphinx $@ target."
 
+else ifneq ($(DOCBOOKS),)
+
+# Skip Sphinx build if the user explicitly requested DOCBOOKS.
+.DEFAULT:
+	@echo "  SKIP    Sphinx $@ target (DOCBOOKS specified)."
+
 else # HAVE_SPHINX
 
 # User-friendly check for rst2pdf
-- 
2.1.4

Re: [PATCH] doc: flat-table directive

[Jani Nikula]

On Fri, 01 Jul 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Am 01.07.2016 um 13:56 schrieb Jani Nikula <jani.nikula@intel.com>:
>
>> On Fri, 01 Jul 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>>> Am 01.07.2016 um 12:44 schrieb Jani Nikula <jani.nikula@intel.com>:
>>> 
>>>> On Fri, 01 Jul 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
>>>>> Not being able to compile just one docbook is a regression and breaks
>>>>> my process. This needs to be fixed.
>>>> 
>>>> Do you have a regression with *DocBook XML* on docs-next now? If yes,
>>>> clearly that must be fixed ASAP.
>>> 
>>> Now I see the problem ...
>>> 
>>> It seems that we didn't neglected that the DocBook Makefile 
>>> builds single books, e.g.
>>> 
>>>  make DOCBOOKS=device-drivers.xml htmldocs
>>> 
>>> Since the Makefile.sphinx does not care the environment DOCBOOKS,
>>> the XML is build and the reST also, where only the XML content
>>> should be build.
>>> 
>>> Jani, may could you apply a small fix on the Makefile.sphinx which
>>> takes into account, not to run the sphinx-build if the DOCBOOK 
>>> environment is set .. is it a solution for a start?
>> 
>> That is not the problem. Makefile.sphinx happily ignores DOCBOOKS. For
>> example 'make DOCBOOKS=kernel-api.xml htmldocs' on docs-next works just
>> fine for me. Sphinx build ignores DOCBOOKS, the DocBook build works just
>> like it always did.
>
> Are you sure .. may I do something wrong::
>
> $ make DOCBOOKS=kernel-api.xml htmldocs
>   SPHINX  htmldocs
> Running Sphinx v1.4.4
> loading pickled environment... done
> building [mo]: targets for 0 po files that are out of date
> building [html]: targets for 0 source files that are out of date
> updating environment: 0 added, 0 changed, 0 removed
> looking for now-outdated files... none found
> no targets are out of date.
> build succeeded.
>   DOCPROC Documentation/DocBook/kernel-api.xml
> .//lib/crc32.c:148: warning: No description found for parameter 'tab)[256]'
> .//lib/crc32.c:148: warning: Excess function parameter 'tab' description in 'crc32_le_generic'
> .//lib/crc32.c:293: warning: No description found for parameter 'tab)[256]'
> .//lib/crc32.c:293: warning: Excess function parameter 'tab' description in 'crc32_be_generic'
> .//lib/crc32.c:1: warning: no structured comments found
> .//mm/memory.c:2881: warning: No description found for parameter 'old'
>   XMLREF  Documentation/DocBook/kernel-api.aux.xml
>   HTML    Documentation/DocBook/kernel-api.html
>   ....
>
> In this example, the sphinx build is started as a side effect
> even if it is not rebuild in this example, because nothing
> in reST content has changed: "looking for now-outdated files... none found".
>
> As far as I can see, this is done **because** Makefile.sphinx
> ignores DOCBOOK.

I was referring to the original problem Mauro reported where the build
failed. The above is a different issue, but an issue nonetheless. Patch
sent.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [docs-next PATCH] Documentation/sphinx: skip build if user requested specific DOCBOOKS

[Markus Heiser]

Hi Jani,

tested ... it works, thanks a lot.

-- Markus --

Am 01.07.2016 um 14:24 schrieb Jani Nikula <jani.nikula@intel.com>:

> If the user requested specific DocBooks to be built using 'make
> DOCBOOKS=foo.xml htmldocs', assume no Sphinx build is desired. This
> check is transitional, and can be removed once we drop the DocBook
> build.
> 
> Cc: Markus Heiser <markus.heiser@darmarit.de>
> Cc: Mauro Carvalho Chehab <mchehab@osg.samsung.com>
> Fixes: 22cba31bae9d ("Documentation/sphinx: add basic working Sphinx configuration and build")
> Signed-off-by: Jani Nikula <jani.nikula@intel.com>
> ---
> Documentation/Makefile.sphinx | 6 ++++++
> 1 file changed, 6 insertions(+)
> 
> diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
> index 6c203745b6c6..d8d13c92a178 100644
> --- a/Documentation/Makefile.sphinx
> +++ b/Documentation/Makefile.sphinx
> @@ -17,6 +17,12 @@ ifeq ($(HAVE_SPHINX),0)
> 	$(warning The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point to the full path of the '$(SPHINXBUILD)' executable.)
> 	@echo "  SKIP    Sphinx $@ target."
> 
> +else ifneq ($(DOCBOOKS),)
> +
> +# Skip Sphinx build if the user explicitly requested DOCBOOKS.
> +.DEFAULT:
> +	@echo "  SKIP    Sphinx $@ target (DOCBOOKS specified)."
> +
> else # HAVE_SPHINX
> 
> # User-friendly check for rst2pdf
> -- 
> 2.1.4
> 

Re: [PATCH] doc: flat-table directive

[Mauro Carvalho Chehab]

Hi Jani,

Em Fri, 01 Jul 2016 13:44:17 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Fri, 01 Jul 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> > Not being able to compile just one docbook is a regression and breaks
> > my process. This needs to be fixed.  
> 
> Do you have a regression with *DocBook XML* on docs-next now? If yes,
> clearly that must be fixed ASAP.

No, but as we'll be switching from DocBook to Sphinx, it should be
possible for the maintainers to use the same or similar workflows.

The one requirement from myside, as the media maintainer, is to be
able to compile *just* the media Sphinx book. As I do it patch by
patch, and we usually merge ~500 patches per kernel version, the
build should be fast, and shouldn't be affected by any breakages
on other docs.

> As to Sphinx, I fear your DocBook workflows are so elaborate that we are
> bound to break some of them when switching over. Personally, I don't
> think that's all bad. 

Well, you're not the one that is responsible for fixing the media mess.
this workflow is there since 2006 at the V4L2 side, and has been improved
over the years. Thanks to it, our documentation always reflect the API
supported by the Kernel. At DVB, we didn't have anything like that, and
we had a *huge* gap between documentation and code. So, last year, we
fixed the gap and the DVB APIs that are more relevant are also in good
shape.

We should not go back to the dark ages, where gaps happened and nobody
detected, until several years after the code has been introduced, and
being hard to identify what the hack some obscure API bits really mean.

> It's an opportunity to make things better.

I'm all for making things better, but let's not make them worse.

> But we
> can't make all of that happen overnight for media. This is why we have
> Sphinx and DocBook side by side for now.

My plan is to swith over the media docbook in one Kernel cycle. Maintaining
two documentations side by side, with the high volume of changes they suffer
on each Kernel cycle is not feasible.

> We specifically wanted to make the switch for e.g. GPU documentation
> fast, act as guinea pigs, hit the issues, and make it easier for others
> in the end. We'd also love to get some of those validation things you
> have, for everyone, but using Sphinx instead of a bunch of scripts and
> sed on the input files.

Sphinx doesn't seem to be able to cross-reference the source code to
identify documentation gaps at the public APIs. So, we'll need an external
script to parse the code and get the API identifiers from the headers.

Yeah, the logic we use to parse the media headers could be more
generic and be used on other subsystems, and I'm all for it, but
semantic parsers are tricky. In order to make them generic, it would
very likely need to use a real lexical analyzer, like flex (or some
perl/python module that would increase the toolchain dependencies to build
the Kernel). Also, right now, the logic handles some special cases,
that are specific to the subsystem. By making it generic, we'll need
to either get rid of that or add per-book quirk configuration files.

> This is also one of the reasons why I so much want to keep everything
> behind one configuration file, and build everything in the Sphinx
> toolchain. To keep it all more uniform, to not duplicate stuff, and not
> deviate to some silos like we've done in the past. I think when we have
> things working, we can add dedicated config files for the select few
> things that have additional special needs. Media is probably one of
> them. But that said, I think we should be able to keep including that to
> the main documentation build too.

I agree with the general principle of building everything at the Sphinx
toolchain. But this requires a deep look, and it will take a while for
this goal to happen.

Yet, if you're talking about creating an unique huge document with
everything inside, I don't believe at an "one solution fits all" kind
of approach.

We have at least 3 different sets of documents right now:

set 1) Kernel-doc Kernel internal API documentation

This is actually split into 27 different DocBook templates. Among this set, 
3 of them documents also the uAPI: alsa-driver-api.tmpl, iio.tmpl and
networking.tmpl.

I guess it should be possible to group all of them altogether, as you
seem to want. Yet, I'm not sure if this is the best thing to do. What 
would be the advantage of grouping them? What would be the disadvantage?

Looking at the positive side, they all document APIs and they all share
about the same "DocumentationStyle", with is enforced via
kernel-doc-nano-HOWTO.txt. So, gluing them together won't look too much
like a Frankenstein monster.

set 2) "Pure" DocBooks, each of them organized as a separate book and that
don't use Kernel-doc:
	1. kernel-hacking.tmpl
	2. lsm.tmpl
	3. Media DocBook (media_api.tmpl, media-entities.tmpl, media-indices.tmpl)
	4. uio-howto.tmpl
	5. writing-an-alsa-driver.tmpl
	6. writing_musb_glue_layer.tmpl
	7. writing_usb_driver.tmpl

Each was written on a different time, by a different set of authors
and following their own documentation style. Some work has to be
done before being able to merge them, if ever desired, in order to make
them look as a hole. Yet, they deal different subjects. I suspect 
they'll be better alone than grouped into a single document.

Also, media is clearly big and complex enough to be a separate 
document, and has its own documentation style. Except perhaps
for ALSA, all the other 5 documents handle completely different
matters. I can't think why someone would want them grouped.

Also, I'm almost sure that, at least for what's currently at the
Media API DocBook, we'll need to be able to use a different CSS
template, because we have very long tables there, so we want to
use a different layout to use a "landscape" like style.

set 3) Pure text documents, under Documentation/.

In this set, there are also several different subjects, from very
technical stuff to things like CodingStyle and CodeOfConduct.

There are also documents there for the Kernel end-users, like:
kernel-parameters.txt. 

So, IMHO, we should either keep them alone, or create some
some sets like:
	- API documentation;
	- Userspace documentation;
	- Compilation documentation;
	- Kernel programming best practices.

Looking at the media documentation on set 3, there is a mix of updated
and really old docs under video4linux/ and dvb/. Those require extra
care: some could probably be trashed, others should be merged with the
media kernel-doc stuff. There are things there that won't fit on
either one of the categories, like video4linux/CARDLIST.*,
as they're meant to be a sort of end-users documentation.

In summary, grouping all those things into a single document
seems wrong.

Putting everything altoghether would be like gluing all your C
programming language books with your English books and dictionaries,
just because they all were found on the same library. Let's separate
the books into shelves, as it makes easier for the user to find
the right documentation he needs.

> Also, many thanks for digging into this and reporting issues that you do
> face.

Anytime. I'll keep looking into the issues I'm finding, as the end
goal for all of us is to have a better documented Kernel.

-- 
Thanks,
Mauro

Re: [PATCH] doc: flat-table directive

[Jonathan Corbet]

On Fri, 01 Jul 2016 13:44:17 +0300
Jani Nikula <jani.nikula@intel.com> wrote:

> This is also one of the reasons why I so much want to keep everything
> behind one configuration file, and build everything in the Sphinx
> toolchain. To keep it all more uniform, to not duplicate stuff, and not
> deviate to some silos like we've done in the past. I think when we have
> things working, we can add dedicated config files for the select few
> things that have additional special needs. Media is probably one of
> them. But that said, I think we should be able to keep including that to
> the main documentation build too.

So this is mostly how I see things too.  But it might still be worth
thinking about whether media, in particular, is special and should be
buildable as a standalone document.  I can see there may be settings
where only that is wanted.

Some sort of "only build xxx.rst and see what explodes" makefile option
also seems like a useful thing to have, just as a time saver for
developers and maintainers.

jon

Re: [PATCH] doc: flat-table directive

[Mauro Carvalho Chehab]

Em Fri, 1 Jul 2016 14:18:34 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 01.07.2016 um 11:38 schrieb Mauro Carvalho Chehab <mchehab@osg.samsung.com>:
> 
> 
> > Btw, yesterday, I tried to add references to a C code, at video.rst,
> > just like we did with DocBook:
> > 
> > .. code-block:: c
> >   :caption: Example 2: Switching to the first video input
> > 
> >   int index;
> > 
> >   index = 0;
> > 
> >   if (-1 == ioctl(fd, :ref:`VIDIOC_S_INPUT <vidioc-s-input>`, &index)) {
> >       perror("VIDIOC_S_INPUT");
> >       exit(EXIT_FAILURE);
> >   }
> > 
> > But Sphinx didn't allow doing it. I was unable to find any syntax
> > on it that would make Sphinx use a monospaced font but still parse
> > the references at the code. While for those small examples this would
> > be ok, This is something that we do want for the header files that
> > we put at V4L and DVB annexes. For DocBook, we do a lot of things like
> > this at the DocBook/media/Makefile:
> > 	sed -e "s/\(enum *\)v4l2_mpeg_cx2341x_video_\([a-z]*_spatial_filter_type\)/\1<link linkend=\"\2\">v4l2_mpeg_cx2341x_video_\2<\/link>/g" videodev2.h  
> 
> In Sphinx, the code-block directive is a literal block, no refs or markup
> will be interpreted. This is different to what you know from DocBook.
> I will look for a solution, that matches your requirement.

Ok, thanks!

> > The actual code is a way more complex, but basically the idea is that
> > it escapes anything that DocBook might interpret as a command, and adds
> > <link> tags for every enum, typedef, ioctl, struct, syscall and define
> > it founds at the header files. the xmllint will produce errors when
> > links are not solved, and we'll be able to detect that the API is not
> > fully documented.  
> 
> OK, checking dead internal links might be one requirement more. Normally 
> sphinx reports internal refs which are broken with a WARNING, but I have to
> analyze what xmllint checks and how we could realize something similar
> in sphinx / or if it is enough what sphinx reports. 

An option at Sphinx to handle warnings as errors would be useful. Yet,
it is not hard to modify my build scripts to handle WARNING as errors.

> 
> > We need to do a similar process with Sphinx. A side effect is that, if
> > someone looks at the header files in the anexes, it can click on any
> > symbol at the API and see the full documentation.  
> 
> Yes, altogether, it seems to be a nice feature, I have to think about. But
> I will point it in a separate thread.

Ok, thanks!

> > Btw, in the case of the above example, I had to manually number it as
> > "Example 2", because I was unable to find a way with Sphinx to auto
> > numerate code-block captions. This is also something we want to fix,
> > as it is very hard to manually number things on a 600+ page document.
> >   
> 
> This needs a sphinx extension, I have seen similar solutions for 
> numbering figures on the net ... I will try to find a small solution.

Yeah, we'll need this for figures and tables too.

> But before I will send out some small patches which are needed 
> first (IMHO). E.g. customizing the RTD theme for rendering large 
> tables in HTML well and activation of useful extensions like todolist.
> I have this in my "chaotic bulk" patch :-) ... 

Ah, yes, I noticed that the .. todolist was causing warnings...
I ended by adding a patch removing it internally, as there weren't
anything at the todo block :)

> I will separate it out an send it to Jon one by one.

OK!

Thanks!
Mauro

Re: [PATCH] doc: flat-table directive

[Mauro Carvalho Chehab]

Em Fri, 01 Jul 2016 15:26:45 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> > In this example, the sphinx build is started as a side effect
> > even if it is not rebuild in this example, because nothing
> > in reST content has changed: "looking for now-outdated files... none found".
> >
> > As far as I can see, this is done **because** Makefile.sphinx
> > ignores DOCBOOK.  
> 
> I was referring to the original problem Mauro reported where the build
> failed. The above is a different issue, but an issue nonetheless. Patch
> sent.

Patches look ok, but that's not what I wanted ;) what I want is to
have something that Makefile.sphinx will look into and build just
the book described there. E. g. something like:

make BOOK="linux_tv" htmldocs

Would make sphinx to only build the book under the
Documentation/linux_tv directory.

-- 
Thanks,
Mauro

Re: [PATCH] doc: flat-table directive

[Jani Nikula]

On Fri, 01 Jul 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> In Sphinx, the code-block directive is a literal block, no refs or markup
> will be interpreted. This is different to what you know from DocBook.
> I will look for a solution, that matches your requirement.

Parsed literal block solves the problem, but brings other problems like
requiring escaping for a lot of stuff. And you lose syntax
highlighting. It would be great to have a middle ground, like a
"semi-parsed code block" where the references are parsed but nothing
else.

http://docutils.sourceforge.net/docs/ref/rst/directives.html#parsed-literal-block

> OK, checking dead internal links might be one requirement more. Normally 
> sphinx reports internal refs which are broken with a WARNING, but I have to
> analyze what xmllint checks and how we could realize something similar
> in sphinx / or if it is enough what sphinx reports.

When we turn function() and &struct structure references to Sphinx
references, we pretty much rely on *not* being strict about all the
targets existing. At least for now. In an ideal world we'd aim for -n
and -W sphinx-build options, but we're far from that, and I don't know
if it's really possible ever.

Is it possible to set -n and/or -W on a per-rst file basis, either in
the top config file or from within the rst file itself? Then we could
gradually improve this, and subsystems that really care about this could
be in the forefront.

> But before I will send out some small patches which are needed 
> first (IMHO). E.g. customizing the RTD theme for rendering large 
> tables in HTML well and activation of useful extensions like todolist.
> I have this in my "chaotic bulk" patch :-) ... I will separate it out
> an send it to Jon one by one.

Btw I don't think we are really attached to the RTD theme. It's just
something I picked that was prettier than the default theme, and was
widely available and packaged in distros. Ideally, we should probably
keep the customization at a level where it's possible for people to
easily use different themes. That said, rendering big tables in the RTD
theme is definitely an issue.

I'd also aim to be fairly conservative at first in terms of the rst
features and Sphinx extensions we use. Keep it simple. It's really easy
to go overboard in the beginning. See how things pan out and gradually
extend from there.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [docs-next PATCH] Documentation/sphinx: skip build if user requested specific DOCBOOKS

[Mauro Carvalho Chehab]

Em Fri,  1 Jul 2016 15:24:44 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> If the user requested specific DocBooks to be built using 'make
> DOCBOOKS=foo.xml htmldocs', assume no Sphinx build is desired. This
> check is transitional, and can be removed once we drop the DocBook
> build.

I guess this is actually a wrong assumption. I mean, it is valid to
build multiple documents at the same time. Sometimes I do things
like:
	make DOCBOOKS="media_api.xml device-drivers.xml" htmldocs

When I want both docs to be compiled.

What I would be expecting is that Sphinx would be looking into
the DOCBOOKS targets and see if (some) of them belongs to it.

Alternatively, we could add a separate makefile var for the
Sphinx targets, but the logic would be more complex, as it
should:

not ignore:
	make htmldocs

ignore:
	make DOCBOOKS="device-drivers.xml" htmldocs

not ignore:
	make SPHINXDOCS="linux_tv" DOCBOOKS="device-drivers.xml" htmldocs

not ignore:
	make SPHINXDOCS="linux_tv" htmldocs

Regards,
Mauro

> 
> Cc: Markus Heiser <markus.heiser@darmarit.de>
> Cc: Mauro Carvalho Chehab <mchehab@osg.samsung.com>
> Fixes: 22cba31bae9d ("Documentation/sphinx: add basic working Sphinx configuration and build")
> Signed-off-by: Jani Nikula <jani.nikula@intel.com>
> ---
>  Documentation/Makefile.sphinx | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
> index 6c203745b6c6..d8d13c92a178 100644
> --- a/Documentation/Makefile.sphinx
> +++ b/Documentation/Makefile.sphinx
> @@ -17,6 +17,12 @@ ifeq ($(HAVE_SPHINX),0)
>  	$(warning The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point to the full path of the '$(SPHINXBUILD)' executable.)
>  	@echo "  SKIP    Sphinx $@ target."
>  
> +else ifneq ($(DOCBOOKS),)
> +
> +# Skip Sphinx build if the user explicitly requested DOCBOOKS.
> +.DEFAULT:
> +	@echo "  SKIP    Sphinx $@ target (DOCBOOKS specified)."
> +
>  else # HAVE_SPHINX
>  
>  # User-friendly check for rst2pdf


-- 
Thanks,
Mauro

Re: [docs-next PATCH] Documentation/sphinx: skip build if user requested specific DOCBOOKS

[Jani Nikula]

On Fri, 01 Jul 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> Em Fri,  1 Jul 2016 15:24:44 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
>
>> If the user requested specific DocBooks to be built using 'make
>> DOCBOOKS=foo.xml htmldocs', assume no Sphinx build is desired. This
>> check is transitional, and can be removed once we drop the DocBook
>> build.
>
> I guess this is actually a wrong assumption. I mean, it is valid to
> build multiple documents at the same time. Sometimes I do things
> like:
> 	make DOCBOOKS="media_api.xml device-drivers.xml" htmldocs
>
> When I want both docs to be compiled.
>
> What I would be expecting is that Sphinx would be looking into
> the DOCBOOKS targets and see if (some) of them belongs to it.
>
> Alternatively, we could add a separate makefile var for the
> Sphinx targets, but the logic would be more complex, as it
> should:

Please let's not conflate DOCBOOKS to mean something other than
DocBooks. I think it'll be easier that way.

So I guess we'll need a way to build just a subset of the Sphinx
documentation. I would like that to be a somewhat generic thing, not
requiring a separate conf file for each subset. Is the granularity of a
directory enough? I've been meaning to look into passing different
<sourcedir> and <filenames> to sphinx-build for this, but I don't have
the time now.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH] doc: flat-table directive

[Markus Heiser]

Am 01.07.2016 um 15:09 schrieb Jani Nikula <jani.nikula@intel.com>:

> On Fri, 01 Jul 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>> In Sphinx, the code-block directive is a literal block, no refs or markup
>> will be interpreted. This is different to what you know from DocBook.
>> I will look for a solution, that matches your requirement.
> 
> Parsed literal block solves the problem, but brings other problems like
> requiring escaping for a lot of stuff. And you lose syntax
> highlighting. It would be great to have a middle ground, like a
> "semi-parsed code block" where the references are parsed but nothing
> else.
> 
> http://docutils.sourceforge.net/docs/ref/rst/directives.html#parsed-literal-block

Yes, "parsed literal" blocks is not the solution and I have none 
yet ... and you are right, we need something "semi". I doubt whether we 
will eventually find a solution for this, but I will
think about it ... I don't know how, but it must be a solution that is
transparent to the the pygment highlighter and not distort the
node tree. I have to study the sphinx-writer and pygment first.


>> OK, checking dead internal links might be one requirement more. Normally 
>> sphinx reports internal refs which are broken with a WARNING, but I have to
>> analyze what xmllint checks and how we could realize something similar
>> in sphinx / or if it is enough what sphinx reports.
> 
> When we turn function() and &struct structure references to Sphinx
> references, we pretty much rely on *not* being strict about all the
> targets existing. At least for now. In an ideal world we'd aim for -n
> and -W sphinx-build options, but we're far from that, and I don't know
> if it's really possible ever.
> 
> Is it possible to set -n and/or -W on a per-rst file basis, either in
> the top config file or from within the rst file itself? Then we could
> gradually improve this, and subsystems that really care about this could
> be in the forefront.

There is a nitpick_ignore config, but this will not help.
As far as I can see, if you want similar on a per file basis,
you need to implement a (HTML) builder which checks
on which tree-level the *current* node is and if this node
is in a doctree of one of your files in your *configured file-list*  
then turns the warning into an error ... may we see more
requirements coming into, which needs to implement a HTML-builder
we can implement one. I have implemented a man-page builder
for the kernel-doc comments, because it was inevitable
but I think it is not a good idea to reimplement the
HTML builder in this first usage of sphinx.

May this is the totally wrong way, may it is better
to implement a *lint* builder from scratch (should not
be hard).

>> But before I will send out some small patches which are needed 
>> first (IMHO). E.g. customizing the RTD theme for rendering large 
>> tables in HTML well and activation of useful extensions like todolist.
>> I have this in my "chaotic bulk" patch :-) ... I will separate it out
>> an send it to Jon one by one.
> 
> Btw I don't think we are really attached to the RTD theme. It's just
> something I picked that was prettier than the default theme, and was
> widely available and packaged in distros.

IMHO it is not prefect but the most elaborate you will find in the net.

> Ideally, we should probably
> keep the customization at a level where it's possible for people to
> easily use different themes.

Layout is done in the theme, we have no chance to influence 
the layout out of / before the theme. 

> That said, rendering big tables in the RTD
> theme is definitely an issue.
> 
> I'd also aim to be fairly conservative at first in terms of the rst
> features and Sphinx extensions we use. Keep it simple. It's really easy
> to go overboard in the beginning. See how things pan out and gradually
> extend from there.

Yes, KIS ... I send the theme patch and you will see that it contains only 
a view lines pointing exactly what we need. And with the builtin-extensions,
I think it will be good to activate common used extensions ...

* todolist: http://www.sphinx-doc.org/en/stable/ext/todo.html and
* intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html

should be enabled. 

-- Markus --



> 
> BR,
> Jani.
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center

Re: [PATCH] doc: flat-table directive

[Markus Heiser]

Am 01.07.2016 um 14:58 schrieb Jonathan Corbet <corbet@lwn.net>:

> On Fri, 01 Jul 2016 13:44:17 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
> 
>> This is also one of the reasons why I so much want to keep everything
>> behind one configuration file, and build everything in the Sphinx
>> toolchain. To keep it all more uniform, to not duplicate stuff, and not
>> deviate to some silos like we've done in the past. I think when we have
>> things working, we can add dedicated config files for the select few
>> things that have additional special needs. Media is probably one of
>> them. But that said, I think we should be able to keep including that to
>> the main documentation build too.
> 
> So this is mostly how I see things too.  But it might still be worth
> thinking about whether media, in particular, is special and should be
> buildable as a standalone document.  I can see there may be settings
> where only that is wanted.
> 

I dare hardly say it, but I come back to the sphinx-sub-project
solution ... don't hurt me ... listen, it is conceptual a bit 
different to what I have done first, but hopefully it is a solution
we can all live with:

* media is "special", so let's build it separate.
 
We have this Documentation/index.rst file, where we can place
all "books" in, except those which are specifically.

.. toctree::
   :maxdepth: 2

   kernel-documentation
   gpu
   ...

Folders matching Documentation/*/conf.py are build separate,
media is the one. This is exactly what my sphinx-sub-project
solution does, building books from folders with a conf.py in.
This conf.py inherits all settings from the top config in
Documentation/conf.py and overwrites only those configurations
which are different.

... could this be a solution?

... I'm a bit afraid to press the send button ... ;-)

> Some sort of "only build xxx.rst and see what explodes" makefile option
> also seems like a useful thing to have, just as a time saver for
> developers and maintainers.
> 
> jon

IMHO it is a conceptual problem:

If we assemble "books" into one project, the context of each
node in the doctree is the union. E.g. cross references from
one "book" to another will only work if the context is the union. 

No matter how we do it, but if we are building parts, the context
is reduced to this part and this will always be different to build
the union, so it could never be a entire *lint*.

-- Markus --

Re: [PATCH] doc: flat-table directive

[Mauro Carvalho Chehab]

Em Fri, 1 Jul 2016 16:07:47 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 01.07.2016 um 15:09 schrieb Jani Nikula <jani.nikula@intel.com>:
> 
> > On Fri, 01 Jul 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:  
> >> In Sphinx, the code-block directive is a literal block, no refs or markup
> >> will be interpreted. This is different to what you know from DocBook.
> >> I will look for a solution, that matches your requirement.  
> > 
> > Parsed literal block solves the problem, but brings other problems like
> > requiring escaping for a lot of stuff. And you lose syntax
> > highlighting. It would be great to have a middle ground, like a
> > "semi-parsed code block" where the references are parsed but nothing
> > else.
> > 
> > http://docutils.sourceforge.net/docs/ref/rst/directives.html#parsed-literal-block  
> 
> Yes, "parsed literal" blocks is not the solution and I have none 
> yet ... and you are right, we need something "semi". I doubt whether we 
> will eventually find a solution for this, but I will
> think about it ... I don't know how, but it must be a solution that is
> transparent to the the pygment highlighter and not distort the
> node tree. I have to study the sphinx-writer and pygment first.

Yeah, a "semi" solution would be perfect. In the specific case of the
header files, a parsed literal should work. the code that generates
the *.h.xml files right now already does escaping where it needs to
make docbook happy. OK, the characters that need to be escaped will
be different, but shouldn't hard to do it, I guess. At the media
Makefile, it currently does escaping for <, > and &:

ESCAPE = \
	-e "s/&/\\&amp;/g" \
	-e "s/</\\&lt;/g" \
	-e "s/>/\\&gt;/g"

> 
> 
> >> OK, checking dead internal links might be one requirement more. Normally 
> >> sphinx reports internal refs which are broken with a WARNING, but I have to
> >> analyze what xmllint checks and how we could realize something similar
> >> in sphinx / or if it is enough what sphinx reports.  
> > 
> > When we turn function() and &struct structure references to Sphinx
> > references, we pretty much rely on *not* being strict about all the
> > targets existing. At least for now. In an ideal world we'd aim for -n
> > and -W sphinx-build options, but we're far from that, and I don't know
> > if it's really possible ever.

I guess that will depend on how much care the subsystem maintainer has
with that. When building things on media, we have zero warnings when
building the Kernel with W=1 on x86. We also check both sparse and smatch 
regularly, trying zero the warnings there too (currently, there are just
a few of them, all false positives and not trivial to remove).

We also have zero warnings on media documentation using xmllint
pedantic mode.

> > Is it possible to set -n and/or -W on a per-rst file basis, either in
> > the top config file or from within the rst file itself? Then we could
> > gradually improve this, and subsystems that really care about this could
> > be in the forefront.  
> 
> There is a nitpick_ignore config, but this will not help.
> As far as I can see, if you want similar on a per file basis,
> you need to implement a (HTML) builder which checks
> on which tree-level the *current* node is and if this node
> is in a doctree of one of your files in your *configured file-list*  
> then turns the warning into an error ... may we see more
> requirements coming into, which needs to implement a HTML-builder
> we can implement one. I have implemented a man-page builder
> for the kernel-doc comments, because it was inevitable
> but I think it is not a good idea to reimplement the
> HTML builder in this first usage of sphinx.
> 
> May this is the totally wrong way, may it is better
> to implement a *lint* builder from scratch (should not
> be hard).

That would work.

> 
> >> But before I will send out some small patches which are needed 
> >> first (IMHO). E.g. customizing the RTD theme for rendering large 
> >> tables in HTML well and activation of useful extensions like todolist.
> >> I have this in my "chaotic bulk" patch :-) ... I will separate it out
> >> an send it to Jon one by one.  
> > 
> > Btw I don't think we are really attached to the RTD theme. It's just
> > something I picked that was prettier than the default theme, and was
> > widely available and packaged in distros.  
> 
> IMHO it is not prefect but the most elaborate you will find in the net.
> 
> > Ideally, we should probably
> > keep the customization at a level where it's possible for people to
> > easily use different themes.  
> 
> Layout is done in the theme, we have no chance to influence 
> the layout out of / before the theme. 
> 
> > That said, rendering big tables in the RTD
> > theme is definitely an issue.
> > 
> > I'd also aim to be fairly conservative at first in terms of the rst
> > features and Sphinx extensions we use. Keep it simple. It's really easy
> > to go overboard in the beginning. See how things pan out and gradually
> > extend from there.  
> 
> Yes, KIS ... I send the theme patch and you will see that it contains only 
> a view lines pointing exactly what we need. And with the builtin-extensions,
> I think it will be good to activate common used extensions ...
> 
> * todolist: http://www.sphinx-doc.org/en/stable/ext/todo.html and

I don't see much usage for todos at the media documentation, as we
basically reject patches that don't come with a proper documentation.

Ok, there are a few things that need extra care at the DVB side of the
spec. So, maybe a todo might be useful there for a while.

> * intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html

Yeah, this sounds interesting. We would have one usage for it: to point
to the ALSA spec for the audio part of a media device.

I guess this could also be useful for the media controller, as other
subsystems are starting to use it.

Regards,
Mauro

Re: [PATCH] doc: flat-table directive

[Mauro Carvalho Chehab]

Em Fri, 1 Jul 2016 16:07:47 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 01.07.2016 um 15:09 schrieb Jani Nikula <jani.nikula@intel.com>:
> 
> > On Fri, 01 Jul 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:  
> >> In Sphinx, the code-block directive is a literal block, no refs or markup
> >> will be interpreted. This is different to what you know from DocBook.
> >> I will look for a solution, that matches your requirement.  
> > 
> > Parsed literal block solves the problem, but brings other problems like
> > requiring escaping for a lot of stuff. And you lose syntax
> > highlighting. It would be great to have a middle ground, like a
> > "semi-parsed code block" where the references are parsed but nothing
> > else.
> > 
> > http://docutils.sourceforge.net/docs/ref/rst/directives.html#parsed-literal-block  
> 
> Yes, "parsed literal" blocks is not the solution and I have none 
> yet ... and you are right, we need something "semi". I doubt whether we 
> will eventually find a solution for this, but I will
> think about it ... I don't know how, but it must be a solution that is
> transparent to the the pygment highlighter and not distort the
> node tree. I have to study the sphinx-writer and pygment first.

Yeah, a "semi" solution would be perfect. In the specific case of the
header files, a parsed literal should work. the code that generates
the *.h.xml files right now already does escaping where it needs to
make docbook happy. OK, the characters that need to be escaped will
be different, but shouldn't hard to do it, I guess. At the media
Makefile, it currently does escaping for <, > and &:

ESCAPE = \
	-e "s/&/\\&amp;/g" \
	-e "s/</\\&lt;/g" \
	-e "s/>/\\&gt;/g"

> 
> 
> >> OK, checking dead internal links might be one requirement more. Normally 
> >> sphinx reports internal refs which are broken with a WARNING, but I have to
> >> analyze what xmllint checks and how we could realize something similar
> >> in sphinx / or if it is enough what sphinx reports.  
> > 
> > When we turn function() and &struct structure references to Sphinx
> > references, we pretty much rely on *not* being strict about all the
> > targets existing. At least for now. In an ideal world we'd aim for -n
> > and -W sphinx-build options, but we're far from that, and I don't know
> > if it's really possible ever.

I guess that will depend on how much care the subsystem maintainer has
with that. When building things on media, we have zero warnings when
building the Kernel with W=1 on x86. We also check both sparse and smatch 
regularly, trying zero the warnings there too (currently, there are just
a few of them, all false positives and not trivial to remove).

We also have zero warnings on media documentation using xmllint
pedantic mode.

> > Is it possible to set -n and/or -W on a per-rst file basis, either in
> > the top config file or from within the rst file itself? Then we could
> > gradually improve this, and subsystems that really care about this could
> > be in the forefront.  
> 
> There is a nitpick_ignore config, but this will not help.
> As far as I can see, if you want similar on a per file basis,
> you need to implement a (HTML) builder which checks
> on which tree-level the *current* node is and if this node
> is in a doctree of one of your files in your *configured file-list*  
> then turns the warning into an error ... may we see more
> requirements coming into, which needs to implement a HTML-builder
> we can implement one. I have implemented a man-page builder
> for the kernel-doc comments, because it was inevitable
> but I think it is not a good idea to reimplement the
> HTML builder in this first usage of sphinx.
> 
> May this is the totally wrong way, may it is better
> to implement a *lint* builder from scratch (should not
> be hard).

That would work.

> 
> >> But before I will send out some small patches which are needed 
> >> first (IMHO). E.g. customizing the RTD theme for rendering large 
> >> tables in HTML well and activation of useful extensions like todolist.
> >> I have this in my "chaotic bulk" patch :-) ... I will separate it out
> >> an send it to Jon one by one.  
> > 
> > Btw I don't think we are really attached to the RTD theme. It's just
> > something I picked that was prettier than the default theme, and was
> > widely available and packaged in distros.  
> 
> IMHO it is not prefect but the most elaborate you will find in the net.
> 
> > Ideally, we should probably
> > keep the customization at a level where it's possible for people to
> > easily use different themes.  
> 
> Layout is done in the theme, we have no chance to influence 
> the layout out of / before the theme. 
> 
> > That said, rendering big tables in the RTD
> > theme is definitely an issue.
> > 
> > I'd also aim to be fairly conservative at first in terms of the rst
> > features and Sphinx extensions we use. Keep it simple. It's really easy
> > to go overboard in the beginning. See how things pan out and gradually
> > extend from there.  
> 
> Yes, KIS ... I send the theme patch and you will see that it contains only 
> a view lines pointing exactly what we need. And with the builtin-extensions,
> I think it will be good to activate common used extensions ...
> 
> * todolist: http://www.sphinx-doc.org/en/stable/ext/todo.html and

I don't see much usage for TODOs at the media documentation, as we
basically reject patches that don't come with a proper documentation.

Ok, there are a few things that need extra care at the DVB side of the
spec. So, maybe a todo might be useful there for a while.

> * intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html

Yeah, this sounds interesting. We would have one usage for it: to point
to the ALSA spec for the audio part of a media device.

I guess this could also be useful for the media controller, as other
subsystems are starting to use it.

Regards,
Mauro

Re: [docs-next PATCH] Documentation/sphinx: skip build if user requested specific DOCBOOKS

[Mauro Carvalho Chehab]

Em Fri, 01 Jul 2016 16:31:14 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Fri, 01 Jul 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> > Em Fri,  1 Jul 2016 15:24:44 +0300
> > Jani Nikula <jani.nikula@intel.com> escreveu:
> >  
> >> If the user requested specific DocBooks to be built using 'make
> >> DOCBOOKS=foo.xml htmldocs', assume no Sphinx build is desired. This
> >> check is transitional, and can be removed once we drop the DocBook
> >> build.  
> >
> > I guess this is actually a wrong assumption. I mean, it is valid to
> > build multiple documents at the same time. Sometimes I do things
> > like:
> > 	make DOCBOOKS="media_api.xml device-drivers.xml" htmldocs
> >
> > When I want both docs to be compiled.
> >
> > What I would be expecting is that Sphinx would be looking into
> > the DOCBOOKS targets and see if (some) of them belongs to it.
> >
> > Alternatively, we could add a separate makefile var for the
> > Sphinx targets, but the logic would be more complex, as it
> > should:  
> 
> Please let's not conflate DOCBOOKS to mean something other than
> DocBooks. I think it'll be easier that way.

OK. Works for me.

> So I guess we'll need a way to build just a subset of the Sphinx
> documentation. I would like that to be a somewhat generic thing, not
> requiring a separate conf file for each subset. Is the granularity of a
> directory enough? I've been meaning to look into passing different
> <sourcedir> and <filenames> to sphinx-build for this, but I don't have
> the time now.

In the case of the media doc, it is actually a matter of building
one rst file (linux_tv/index.rst). For that, I'm equally happy
if we specify it via <sourcedir> or via <filename>.

I suspect we'll end by needing filenames granularity in some future,
but, if you find easier to just handle <sorcedir> for now, that's
OK. 

Regards,
Mauro

Re: [PATCH] doc: flat-table directive

[Jani Nikula]

On Fri, 01 Jul 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Am 01.07.2016 um 14:58 schrieb Jonathan Corbet <corbet@lwn.net>:
>
>> On Fri, 01 Jul 2016 13:44:17 +0300
>> Jani Nikula <jani.nikula@intel.com> wrote:
>> 
>>> This is also one of the reasons why I so much want to keep everything
>>> behind one configuration file, and build everything in the Sphinx
>>> toolchain. To keep it all more uniform, to not duplicate stuff, and not
>>> deviate to some silos like we've done in the past. I think when we have
>>> things working, we can add dedicated config files for the select few
>>> things that have additional special needs. Media is probably one of
>>> them. But that said, I think we should be able to keep including that to
>>> the main documentation build too.
>> 
>> So this is mostly how I see things too.  But it might still be worth
>> thinking about whether media, in particular, is special and should be
>> buildable as a standalone document.  I can see there may be settings
>> where only that is wanted.
>> 
>
> I dare hardly say it, but I come back to the sphinx-sub-project
> solution ... don't hurt me ... listen, it is conceptual a bit 
> different to what I have done first, but hopefully it is a solution
> we can all live with:
>
> * media is "special", so let's build it separate.
>  
> We have this Documentation/index.rst file, where we can place
> all "books" in, except those which are specifically.
>
> .. toctree::
>    :maxdepth: 2
>
>    kernel-documentation
>    gpu
>    ...
>
> Folders matching Documentation/*/conf.py are build separate,
> media is the one. This is exactly what my sphinx-sub-project
> solution does, building books from folders with a conf.py in.
> This conf.py inherits all settings from the top config in
> Documentation/conf.py and overwrites only those configurations
> which are different.
>
> ... could this be a solution?
>
> ... I'm a bit afraid to press the send button ... ;-)

Hey, the discussions can sometimes be heated, but you should never have
to worry about expressing your opinions.

You know I'm opposed to adding several configs and building things
separately. Some of the reasons are to keep things (especially the
Makefiles and configs) simple and easy for the wider audience to
understand, and to agressively promote finding generic solutions that
work for everyone. Sure, maybe this won't completely work in the end,
but it sure as hell won't work if we don't even try.

The current media DocBook docs are a prime example. There are really
great features there, but IMHO done in a really complicated way and
specifically for media documentation (sorry Mauro). I think we'd be in a
better position now if all of that had been done in a more generic way
for all the DocBook docs. So I just want to avoid that for Sphinx, now
that we have a chance for a fresh start.

>> Some sort of "only build xxx.rst and see what explodes" makefile option
>> also seems like a useful thing to have, just as a time saver for
>> developers and maintainers.
>> 
>> jon
>
> IMHO it is a conceptual problem:
>
> If we assemble "books" into one project, the context of each
> node in the doctree is the union. E.g. cross references from
> one "book" to another will only work if the context is the union. 
>
> No matter how we do it, but if we are building parts, the context
> is reduced to this part and this will always be different to build
> the union, so it could never be a entire *lint*.

This is another reason why I like the "everything together" approach. I
like the cross-references from kernel-doc comments to work beyond one
subsystem. I like to have the function() and &struct style references
from GPU documentation to the kernel API and device documents just
work. AFAICT intersphinx doesn't solve this because kernel-doc has no
way of knowing where the target exists. For GPU documentation, putting
it into a silo would be giving up on the goals we had.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH] doc: flat-table directive

[Mauro Carvalho Chehab]

Em Fri, 01 Jul 2016 20:24:48 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Fri, 01 Jul 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> > Am 01.07.2016 um 14:58 schrieb Jonathan Corbet <corbet@lwn.net>:
> >  
> >> On Fri, 01 Jul 2016 13:44:17 +0300
> >> Jani Nikula <jani.nikula@intel.com> wrote:
> >>   
> >>> This is also one of the reasons why I so much want to keep everything
> >>> behind one configuration file, and build everything in the Sphinx
> >>> toolchain. To keep it all more uniform, to not duplicate stuff, and not
> >>> deviate to some silos like we've done in the past. I think when we have
> >>> things working, we can add dedicated config files for the select few
> >>> things that have additional special needs. Media is probably one of
> >>> them. But that said, I think we should be able to keep including that to
> >>> the main documentation build too.  
> >> 
> >> So this is mostly how I see things too.  But it might still be worth
> >> thinking about whether media, in particular, is special and should be
> >> buildable as a standalone document.  I can see there may be settings
> >> where only that is wanted.
> >>   
> >
> > I dare hardly say it, but I come back to the sphinx-sub-project
> > solution ... don't hurt me ... listen, it is conceptual a bit 
> > different to what I have done first, but hopefully it is a solution
> > we can all live with:
> >
> > * media is "special", so let's build it separate.
> >  
> > We have this Documentation/index.rst file, where we can place
> > all "books" in, except those which are specifically.
> >
> > .. toctree::
> >    :maxdepth: 2
> >
> >    kernel-documentation
> >    gpu
> >    ...
> >
> > Folders matching Documentation/*/conf.py are build separate,
> > media is the one. This is exactly what my sphinx-sub-project
> > solution does, building books from folders with a conf.py in.
> > This conf.py inherits all settings from the top config in
> > Documentation/conf.py and overwrites only those configurations
> > which are different.
> >
> > ... could this be a solution?
> >
> > ... I'm a bit afraid to press the send button ... ;-)  
> 
> Hey, the discussions can sometimes be heated, but you should never have
> to worry about expressing your opinions.
> 
> You know I'm opposed to adding several configs and building things
> separately. Some of the reasons are to keep things (especially the
> Makefiles and configs) simple and easy for the wider audience to
> understand, and to agressively promote finding generic solutions that
> work for everyone. Sure, maybe this won't completely work in the end,
> but it sure as hell won't work if we don't even try.
> 
> The current media DocBook docs are a prime example. There are really
> great features there, but IMHO done in a really complicated way and
> specifically for media documentation (sorry Mauro).

My view is just the reverse of yours: the non-media documentation is
too simple, and has lots of gaps ;)

Anyway, the media documentation actually started out of the tree,
by the time Alan Cox were the V4L2 maintainer, as two separate documents. 
The first one written 1999 (v4l2) and the second one in 2001 (dvb).
They were merged upstream by 2006 or 2007, after several conversions
(the DVB doc were in LaTex, and the V4L2 one, were on a very old 
DocBook version).

We're doing a real hard work to keep it updated over the last 10
years, and we just don't want to throw it away and do a fresh
start. Whatever we do, we should not lost documentation nor lost
the scripts that have been working so well to enforce the docs
to keep updated as we touch on the code.

> I think we'd be in a
> better position now if all of that had been done in a more generic way
> for all the DocBook docs. So I just want to avoid that for Sphinx, now
> that we have a chance for a fresh start.
> 
> >> Some sort of "only build xxx.rst and see what explodes" makefile option
> >> also seems like a useful thing to have, just as a time saver for
> >> developers and maintainers.
> >> 
> >> jon  
> >
> > IMHO it is a conceptual problem:
> >
> > If we assemble "books" into one project, the context of each
> > node in the doctree is the union. E.g. cross references from
> > one "book" to another will only work if the context is the union. 
> >
> > No matter how we do it, but if we are building parts, the context
> > is reduced to this part and this will always be different to build
> > the union, so it could never be a entire *lint*.  
> 
> This is another reason why I like the "everything together" approach. I
> like the cross-references from kernel-doc comments to work beyond one
> subsystem. I like to have the function() and &struct style references
> from GPU documentation to the kernel API and device documents just
> work. AFAICT intersphinx doesn't solve this because kernel-doc has no
> way of knowing where the target exists. For GPU documentation, putting
> it into a silo would be giving up on the goals we had.

Having the capability of integrating with other documents is a nice
thing. However, the media uAPI book will have *very few* outside
references (right now, I can think on just one reference, at the
introduction that mentions the ALSA project - even so, maybe the
best would be to point to the ALSA project website, but, if we
decide to point to the internal ALSA docs, we could use intersphinx). 
The reason for that is simple: it describes the API from the Kernel
to userspace, and not the Kernel's internal API.

The Media Kernel's internal API is described at the media section of
the device-drivers.xml. There, references to the other Kernel-doc
documented APIs is very welcomed.

Regards,
Mauro

Captions numbering support

[Mauro Carvalho Chehab]

Markus/Jani,

I'm starting this as a new thread, as it is not directly related to the
other subjects.

I'm fixing right now the captions that got missed during the docbook
conversion. See for example:

https://git.linuxtv.org/mchehab/experimental.git/commit/?h=docs-next&id=69f9771c659958ecc7c8ba26e85c2f61f821b022
https://git.linuxtv.org/mchehab/experimental.git/commit/?h=docs-next&id=56bd374200a15e8888d94fd8410eea916bffea87

As I mentioned before, I'm needing to re-numerate by hand the examples.

I'm wandering if are there any way with Sphinx to use the section
numbering. If so, then we could number the examples as:

1.4 section title
...
1.4.1 Example foo
1.4.2 Example bar
...
1.6 section title 2
...
1.6.1 Exampel foobar

With would make a way easier to keep handling it, while we don't
have ":numbered:" support for the code-block.

Do you know if are there anything like that in Sphinx?

Regards,
Mauro

Re: [PATCH] doc: flat-table directive

[Jani Nikula]

On Fri, 01 Jul 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> Em Fri, 01 Jul 2016 20:24:48 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
>> The current media DocBook docs are a prime example. There are really
>> great features there, but IMHO done in a really complicated way and
>> specifically for media documentation (sorry Mauro).
>
> My view is just the reverse of yours: the non-media documentation is
> too simple, and has lots of gaps ;)

Heh, can't argue with that.

> We're doing a real hard work to keep it updated over the last 10
> years, and we just don't want to throw it away and do a fresh
> start. Whatever we do, we should not lost documentation nor lost
> the scripts that have been working so well to enforce the docs
> to keep updated as we touch on the code.

I hear you. I just want to make it work nicely for everybody. Including
media docs.


BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH] doc: flat-table directive

[Mauro Carvalho Chehab]

Em Fri, 1 Jul 2016 16:07:47 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 01.07.2016 um 15:09 schrieb Jani Nikula <jani.nikula@intel.com>:
> 
> > On Fri, 01 Jul 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:  
> >> In Sphinx, the code-block directive is a literal block, no refs or markup
> >> will be interpreted. This is different to what you know from DocBook.
> >> I will look for a solution, that matches your requirement.  
> > 
> > Parsed literal block solves the problem, but brings other problems like
> > requiring escaping for a lot of stuff. And you lose syntax
> > highlighting. It would be great to have a middle ground, like a
> > "semi-parsed code block" where the references are parsed but nothing
> > else.
> > 
> > http://docutils.sourceforge.net/docs/ref/rst/directives.html#parsed-literal-block  

OK, using parsed literal indeed seems to be a solution. The script below
should be doing what's needed to auto-generate the *.h.rst files.
It should be replicating the logic that is done at the media Makefile,
on a cleaner way. It also allows ignoring some symbols. We do that when
we deprecate APIs: they're kept at the header files, but their
descriptions are removed/replaced at the media book.

This is not yet the final version. I need to do some adjustments, as
only the ioctl refs match the ones on media, but it shouldn't be hard to
fix.


Thanks,
Mauro



#!/usr/bin/perl
use strict;

# change to 1 to generate some debug prints
my $debug = 1;

if (scalar @ARGV < 2 || scalar @ARGV > 3) {
	die "Usage:\n\t$0 <file in> <file out> [<exceptions file>]\n";
}

my ($file_in, $file_out, $file_exceptions) = @ARGV;

my $data;
my @ioctls;
my @defines;
my @typedefs;
my @enums;
my @enum_symbols;
my @structs;

#
# read the file and get identifiers
#

my $is_enum = 0;
open IN, $file_in or die "Can't open $file_in";
while (<IN>) {
	$data .= $_;

	if ($is_enum && m/^\s*([^\s\}]+)\s*[\,=]?/) {
		my $s = $1;
		push @enum_symbols, $1;

		$is_enum = 0 if ($is_enum && m/\}/);

		next;
	}
	$is_enum = 0 if ($is_enum && m/\}/);

	if (m/^\s*#\s*define\s+([A-Z]\S+)\s+_IO/) {
		push @ioctls, $1;
		next;
	}

	if (m/^\s*#\s*define\s+([A-Z]\S+)\s+/) {
		push @defines, $1;
		next
	}

	if (m/^\s*typedef\s+.*\s+(\w\S+);/) {
		push @typedefs, $1;
		next;
	}
	if (m/^\s*enum\s+(\S+)\s+\{/ || m/^\s*enum\s+(\S+)$/) {
		my $v = $1;
		push @enums, $v unless grep{$_ eq $v} @enums;
		$is_enum = $1;
		next;
	}
	if (m/^\s*struct\s+(\S+)\s+\{/ || m/^\s*struct\s+(\S+)$/) {
		my $v = $1;
		push @structs, $v unless grep{$_ eq $v} @structs;
		next;
	}
}
close IN;

#
# Handle multi-line typedefs
#

my @matches = $data =~ m/typedef\s+struct\s+\S+\s*\{[^\}]+\}\s*(\S+)\s*\;/g;
foreach my $m (@matches) {
	push @typedefs, $m unless grep{$_ eq $m} @typedefs;
	next;
}

#
# Handle exceptions, if any
#

if ($file_exceptions) {
	open IN, $file_exceptions or die "Can't read $file_exceptions";
	while (<IN>) {
		next if (m/^\s*$/ || m/^\s*#/);
		if (m/^ignore\s+ioctl\s+(\S+)/) {
			@ioctls = grep { $_ != $1 } @ioctls;
			next;
		}
		if (m/^ignore\s+define\s+(\S+)/) {
			@defines = grep { $_ != $1 } @defines;
			next;
		}
		if (m/^ignore\s+typedef\s+(\S+)/) {
			@typedefs = grep { $_ != $1 } @typedefs;
			next;
		}
		if (m/^ignore\s+enum\s+(\S+)/) {
			@enums = grep { $_ != $1 } @enums;
			next;
		}
		if (m/^ignore\s+struct\s+(\S+)/) {
			@structs = grep { $_ != $1 } @structs;
			next;
		}
		die "Can't parse $file_exceptions";
	}
}

print "ioctls: @ioctls\n" if ($debug && @ioctls);
print "defines: @defines\n" if ($debug && @defines);
print "typedefs: @typedefs\n" if ($debug && @typedefs);
print "enums: @enums\n" if ($debug && @enums);
print "structs: @structs\n" if ($debug && @structs);

#
# Add escape codes for special characters
#
$data =~ s,([\_\`\*\<\>\&\\\\:\/]),\\$1,g;

#
# Align block
#
$data = "    " . $data;
$data =~ s/\n/\n    /g;
$data =~ s/\n\s+$/\n/g;

#
# Add references
#

foreach my $r (@ioctls, @defines, @enum_symbols) {
	my $n = $r;
	$n =~ tr/A-Z/a-z/;
	my $s = ":ref:`$r <$n>`";

	$r =~ s,([\_\`\*\<\>\&\\\\:\/]),\\\\$1,g;

	print "$r -> $s\n" if ($debug);

	$data =~ s/([\s])($r)([\s])/$1$s$3/g;
}

foreach my $r (@enums) {
	my $n = $r;
	$n =~ tr/A-Z_/a-z-/;
	my $s = ":ref:`enum $r <$n>`";

	$r =~ s,([\_\`\*\<\>\&\\\\:\/]),\\\\$1,g;

	print "$r -> $s\n" if ($debug);

	$data =~ s/enum\s+($r)([\s])/$s$2/g;
}

foreach my $r (@structs) {
	my $n = $r;
	$n =~ tr/A-Z_/a-z-/;
	my $s = ":ref:`struct $r <$n>`";

	$r =~ s,([\_\`\*\<\>\&\\\\:\/]),\\\\$1,g;

	print "$r -> $s\n" if ($debug);

	$data =~ s/struct\s+($r)([\s])/$s$2/g;
}

## FIXME

foreach my $r (@typedefs) {
	my $n = $r;
	$n =~ tr/A-Z_/a-z-/;
	my $s = ":ref:`$r <$n>`";

	$r =~ s,([\_\`\*\<\>\&\\\\:\/]),\\\\$1,g;

	print "$r -> $s\n" if ($debug);

	$data =~ s/(typedef\s+^\n+\s+)($r)(\s*;)/$1$s$3/g;
	$data =~ s/(typedef\s+struct\s+\S+\s*\{[^\}]+\}\s*)($r)(\s*;)/$1$s$3/g;
}

#
# Generate output file
#

my $title = $file_in;
$title =~ s,.*/,,;

open OUT, "> $file_out" or die "Can't open $file_out";
print OUT ".. -*- coding: utf-8; mode: rst -*-\n\n";
print OUT "$title\n";
print OUT "=" x length($title);
print OUT "\n\n.. parsed-literal::\n\n";
print OUT $data;
close OUT;