[PATCH v2 0/5] docs: automarkup.py: Make automarkup ready for Sphinx 3.1+

[PATCH v2 0/5] docs: automarkup.py: Make automarkup ready for Sphinx 3.1+

[Nícolas F. R. A. Prado]

Hi,

this patch series makes the automatic markup extension ready for Sphinx 3.1+.
It was based on Mauro's Sphinx patch series, and requires it for the namespaces
to work, but could also be merged through the docs tree without regressions
(other than the increased build time explained below).

The first three patches make automarkup compatible with Sphinx 3.1. The first
patch makes use of the new C roles in Sphinx3 instead of the generic type role
from Sphinx 2, while patches 2 and 3 solve the warnings caused by Sphinx3's
stricter C domain.

Patch 4 adds cross-referencing to C macros with parameters for Sphinx 3.

Patch 5 enables cross-referencing inside C namespaces, which are new to Sphinx
3.1.

On an importante note:
In order to be able to support automatic cross-referencing inside C namespaces,
I needed to disable parallel source reading for Sphinx in patch 5. On my
machine, this makes the build process take about 4 additional minutes. This is
very bad, since the documentation building process already takes too long, but I
couldn't think of a way to sidestep this issue. If anyone has any idea, it would
be greatly appreciated.

Also, for some reason, disabling the source read parallelization makes
Sphinx output 2 warnings saying so, which is another annoyance.

Thanks,
Nícolas

Changes in v2:
- Split the single patch into patches 1, 2 and 3
- Change sphinx version verification in patch 1
- Thanks Mauro for the clarifications in v1:
  - Add patches 4 and 5 for the missing functionalities

Nícolas F. R. A. Prado (5):
  docs: automarkup.py: Use new C roles in Sphinx 3
  docs: automarkup.py: Fix regexes to solve sphinx 3 warnings
  docs: automarkup.py: Skip C reserved words when cross-referencing
  docs: automarkup.py: Add cross-reference for parametrized C macros
  docs: automarkup.py: Allow automatic cross-reference inside C
    namespace

 Documentation/sphinx/automarkup.py | 188 +++++++++++++++++++++++------
 1 file changed, 154 insertions(+), 34 deletions(-)

-- 
2.28.0

[PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Nícolas F. R. A. Prado]

While Sphinx 2 used a single c:type role for struct, union, enum and
typedef, Sphinx 3 uses a specific role for each one.
To keep backward compatibility, detect the Sphinx version and use the
correct roles for that version.

Signed-off-by: Nícolas F. R. A. Prado <nfraprado@protonmail.com>
---
 Documentation/sphinx/automarkup.py | 55 ++++++++++++++++++++++++++----
 1 file changed, 49 insertions(+), 6 deletions(-)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index a1b0f554cd82..db13fb15cedc 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -23,7 +23,21 @@ from itertools import chain
 # bit tries to restrict matches to things that won't create trouble.
 #
 RE_function = re.compile(r'(([\w_][\w\d_]+)\(\))')
-RE_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
+
+#
+# Sphinx 2 uses the same :c:type role for struct, union, enum and typedef
+#
+RE_generic_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
+
+#
+# Sphinx 3 uses a different C role for each one of struct, union, enum and
+# typedef
+#
+RE_struct = re.compile(r'\b(struct)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
+RE_union = re.compile(r'\b(union)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
+RE_enum = re.compile(r'\b(enum)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
+RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
+
 #
 # Detects a reference to a documentation page of the form Documentation/... with
 # an optional extension
@@ -48,9 +62,22 @@ def markup_refs(docname, app, node):
     #
     # Associate each regex with the function that will markup its matches
     #
-    markup_func = {RE_type: markup_c_ref,
-                   RE_function: markup_c_ref,
-                   RE_doc: markup_doc_ref}
+    markup_func_sphinx2 = {RE_doc: markup_doc_ref,
+                           RE_function: markup_c_ref,
+                           RE_generic_type: markup_c_ref}
+
+    markup_func_sphinx3 = {RE_doc: markup_doc_ref,
+                           RE_function: markup_c_ref,
+                           RE_struct: markup_c_ref,
+                           RE_union: markup_c_ref,
+                           RE_enum: markup_c_ref,
+                           RE_typedef: markup_c_ref}
+
+    if sphinx.version_info[0] >= 3:
+        markup_func = markup_func_sphinx3
+    else:
+        markup_func = markup_func_sphinx2
+
     match_iterators = [regex.finditer(t) for regex in markup_func]
     #
     # Sort all references by the starting position in text
@@ -79,8 +106,24 @@ def markup_refs(docname, app, node):
 # type_name) with an appropriate cross reference.
 #
 def markup_c_ref(docname, app, match):
-    class_str = {RE_function: 'c-func', RE_type: 'c-type'}
-    reftype_str = {RE_function: 'function', RE_type: 'type'}
+    class_str = {RE_function: 'c-func',
+                 # Sphinx 2 only
+                 RE_generic_type: 'c-type',
+                 # Sphinx 3+ only
+                 RE_struct: 'c-struct',
+                 RE_union: 'c-union',
+                 RE_enum: 'c-enum',
+                 RE_typedef: 'c-type',
+                 }
+    reftype_str = {RE_function: 'function',
+                   # Sphinx 2 only
+                   RE_generic_type: 'type',
+                   # Sphinx 3+ only
+                   RE_struct: 'struct',
+                   RE_union: 'union',
+                   RE_enum: 'enum',
+                   RE_typedef: 'type',
+                   }
 
     cdom = app.env.domains['c']
     #
-- 
2.28.0

[PATCH v2 2/5] docs: automarkup.py: Fix regexes to solve sphinx 3 warnings

[Nícolas F. R. A. Prado]

With the transition to Sphinx 3, new warnings were generated by
automarkup, exposing bugs in the regexes.

The warnings were caused by the expressions matching words in the
translated versions of the documentation, since any unicode character
was matched.

Fix the regular expression by making the C regexes use ASCII and
ensuring the expressions only match the beginning of words.

Signed-off-by: Nícolas F. R. A. Prado <nfraprado@protonmail.com>
---
 Documentation/sphinx/automarkup.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index db13fb15cedc..43dd9025fc77 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -22,12 +22,13 @@ from itertools import chain
 # :c:func: block (i.e. ":c:func:`mmap()`s" flakes out), so the last
 # bit tries to restrict matches to things that won't create trouble.
 #
-RE_function = re.compile(r'(([\w_][\w\d_]+)\(\))')
+RE_function = re.compile(r'\b(([a-zA-Z_]\w+)\(\))', flags=re.ASCII)
 
 #
 # Sphinx 2 uses the same :c:type role for struct, union, enum and typedef
 #
-RE_generic_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
+RE_generic_type = re.compile(r'\b(struct|union|enum|typedef)\s+([a-zA-Z_]\w+)',
+                             flags=re.ASCII)
 
 #
 # Sphinx 3 uses a different C role for each one of struct, union, enum and
@@ -42,7 +43,7 @@ RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
 # Detects a reference to a documentation page of the form Documentation/... with
 # an optional extension
 #
-RE_doc = re.compile(r'Documentation(/[\w\-_/]+)(\.\w+)*')
+RE_doc = re.compile(r'\bDocumentation(/[\w\-_/]+)(\.\w+)*')
 
 #
 # Many places in the docs refer to common system calls.  It is
-- 
2.28.0

[PATCH v2 3/5] docs: automarkup.py: Skip C reserved words when cross-referencing

[Nícolas F. R. A. Prado]

With the transition to Sphinx 3, new warnings were caused by
automarkup, exposing bugs in the name matching.

When automarkup parsed a text like "struct struct" in the documentation,
it tried to cross-reference to a "struct" symbol, which is recognized as
a C reserved word by Sphinx 3, generating a warning.

Add some C reserved words (only the ones that were causing warnings) to
a list and skip them while trying to cross-reference.

Signed-off-by: Nícolas F. R. A. Prado <nfraprado@protonmail.com>
---
 Documentation/sphinx/automarkup.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index 43dd9025fc77..1cc3a2cf2a88 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -45,6 +45,12 @@ RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
 #
 RE_doc = re.compile(r'\bDocumentation(/[\w\-_/]+)(\.\w+)*')
 
+#
+# Reserved C words that we should skip when cross-referencing
+#
+Skipnames = [ 'for', 'if', 'register', 'sizeof', 'struct', 'unsigned' ]
+
+
 #
 # Many places in the docs refer to common system calls.  It is
 # pointless to try to cross-reference them and, as has been known
@@ -133,7 +139,8 @@ def markup_c_ref(docname, app, match):
     target = match.group(2)
     target_text = nodes.Text(match.group(0))
     xref = None
-    if not (match.re == RE_function and target in Skipfuncs):
+    if not ((match.re == RE_function and target in Skipfuncs)
+            or (target in Skipnames)):
         lit_text = nodes.literal(classes=['xref', 'c', class_str[match.re]])
         lit_text += target_text
         pxref = addnodes.pending_xref('', refdomain = 'c',
-- 
2.28.0

[PATCH v2 4/5] docs: automarkup.py: Add cross-reference for parametrized C macros

[Nícolas F. R. A. Prado]

Sphinx 3 added support for declaring C macros with parameters using the
:c:macro role.

To support automarkup for both functions and parametrized macros using
the same regex (words ending in ()), try to cross-reference to both, and
only fall back to regular text if neither exist.

Signed-off-by: Nícolas F. R. A. Prado <nfraprado@protonmail.com>
---
 Documentation/sphinx/automarkup.py | 49 +++++++++++++++++++++++++-----
 1 file changed, 42 insertions(+), 7 deletions(-)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index 1cc3a2cf2a88..409dbc4100de 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -74,7 +74,7 @@ def markup_refs(docname, app, node):
                            RE_generic_type: markup_c_ref}
 
     markup_func_sphinx3 = {RE_doc: markup_doc_ref,
-                           RE_function: markup_c_ref,
+                           RE_function: markup_func_ref_sphinx3,
                            RE_struct: markup_c_ref,
                            RE_union: markup_c_ref,
                            RE_enum: markup_c_ref,
@@ -109,12 +109,47 @@ def markup_refs(docname, app, node):
     return repl
 
 #
-# Try to replace a C reference (function() or struct/union/enum/typedef
-# type_name) with an appropriate cross reference.
+# In sphinx3 we can cross-reference to C macro and function, each one with its
+# own C role, but both match the same regex, so we try both.
 #
+def markup_func_ref_sphinx3(docname, app, match):
+    class_str = ['c-func', 'c-macro']
+    reftype_str = ['function', 'macro']
+
+    cdom = app.env.domains['c']
+    #
+    # Go through the dance of getting an xref out of the C domain
+    #
+    target = match.group(2)
+    target_text = nodes.Text(match.group(0))
+    xref = None
+    if not (target in Skipfuncs or target in Skipnames):
+        for class_s, reftype_s in zip(class_str, reftype_str):
+            lit_text = nodes.literal(classes=['xref', 'c', class_s])
+            lit_text += target_text
+            pxref = addnodes.pending_xref('', refdomain = 'c',
+                                          reftype = reftype_s,
+                                          reftarget = target, modname = None,
+                                          classname = None)
+            #
+            # XXX The Latex builder will throw NoUri exceptions here,
+            # work around that by ignoring them.
+            #
+            try:
+                xref = cdom.resolve_xref(app.env, docname, app.builder,
+                                         reftype_s, target, pxref,
+                                         lit_text)
+            except NoUri:
+                xref = None
+
+            if xref:
+                return xref
+
+    return target_text
+
 def markup_c_ref(docname, app, match):
-    class_str = {RE_function: 'c-func',
-                 # Sphinx 2 only
+    class_str = {# Sphinx 2 only
+                 RE_function: 'c-func',
                  RE_generic_type: 'c-type',
                  # Sphinx 3+ only
                  RE_struct: 'c-struct',
@@ -122,8 +157,8 @@ def markup_c_ref(docname, app, match):
                  RE_enum: 'c-enum',
                  RE_typedef: 'c-type',
                  }
-    reftype_str = {RE_function: 'function',
-                   # Sphinx 2 only
+    reftype_str = {# Sphinx 2 only
+                   RE_function: 'function',
                    RE_generic_type: 'type',
                    # Sphinx 3+ only
                    RE_struct: 'struct',
-- 
2.28.0

[PATCH v2 5/5] docs: automarkup.py: Allow automatic cross-reference inside C namespace

[Nícolas F. R. A. Prado]

Sphinx 3.1 introduced namespaces for C cross-references. With this,
each C domain type/function declaration is put inside the namespace that
was active at the time of its declaration.

To support automatic cross-reference inside C namespaces:
- Save the C namespace used in each doc file (if any) at the source-read
  phase, in the beginning of the Sphinx process.
- When making the automarkup, if any namespace was used in the current
  file, try to cross-reference to the symbol inside of it before trying
  in the global namespace.

To make the first step possible, disable the parallel_read_safe option
in Sphinx, since the dictionary that maps the files to the C namespaces
can't be concurrently updated. This unfortunately increases the build
time of the documentation.

Signed-off-by: Nícolas F. R. A. Prado <nfraprado@protonmail.com>
---
 Documentation/sphinx/automarkup.py | 130 ++++++++++++++++++-----------
 1 file changed, 82 insertions(+), 48 deletions(-)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index 409dbc4100de..bca8cf5f519d 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -45,6 +45,8 @@ RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
 #
 RE_doc = re.compile(r'\bDocumentation(/[\w\-_/]+)(\.\w+)*')
 
+RE_namespace = re.compile(r'^\s*..\s*c:namespace::\s*(\S+)\s*$')
+
 #
 # Reserved C words that we should skip when cross-referencing
 #
@@ -62,6 +64,8 @@ Skipfuncs = [ 'open', 'close', 'read', 'write', 'fcntl', 'mmap',
               'select', 'poll', 'fork', 'execve', 'clone', 'ioctl',
               'socket' ]
 
+c_namespace = {}
+
 def markup_refs(docname, app, node):
     t = node.astext()
     done = 0
@@ -120,30 +124,42 @@ def markup_func_ref_sphinx3(docname, app, match):
     #
     # Go through the dance of getting an xref out of the C domain
     #
-    target = match.group(2)
+    base_target = match.group(2)
     target_text = nodes.Text(match.group(0))
     xref = None
-    if not (target in Skipfuncs or target in Skipnames):
-        for class_s, reftype_s in zip(class_str, reftype_str):
-            lit_text = nodes.literal(classes=['xref', 'c', class_s])
-            lit_text += target_text
-            pxref = addnodes.pending_xref('', refdomain = 'c',
-                                          reftype = reftype_s,
-                                          reftarget = target, modname = None,
-                                          classname = None)
-            #
-            # XXX The Latex builder will throw NoUri exceptions here,
-            # work around that by ignoring them.
-            #
-            try:
-                xref = cdom.resolve_xref(app.env, docname, app.builder,
-                                         reftype_s, target, pxref,
-                                         lit_text)
-            except NoUri:
-                xref = None
-
-            if xref:
-                return xref
+    possible_targets = [base_target]
+    # Check if this document has a namespace, and if so, try
+    # cross-referencing inside it first.
+    try:
+        namespace = c_namespace[docname]
+    except KeyError:
+        pass
+    else:
+        possible_targets.insert(0, namespace + "." + base_target)
+
+    if base_target not in Skipnames:
+        for target in possible_targets:
+            if target not in Skipfuncs:
+                for class_s, reftype_s in zip(class_str, reftype_str):
+                    lit_text = nodes.literal(classes=['xref', 'c', class_s])
+                    lit_text += target_text
+                    pxref = addnodes.pending_xref('', refdomain = 'c',
+                                                  reftype = reftype_s,
+                                                  reftarget = target, modname = None,
+                                                  classname = None)
+                    #
+                    # XXX The Latex builder will throw NoUri exceptions here,
+                    # work around that by ignoring them.
+                    #
+                    try:
+                        xref = cdom.resolve_xref(app.env, docname, app.builder,
+                                                 reftype_s, target, pxref,
+                                                 lit_text)
+                    except NoUri:
+                        xref = None
+
+                    if xref:
+                        return xref
 
     return target_text
 
@@ -171,34 +187,43 @@ def markup_c_ref(docname, app, match):
     #
     # Go through the dance of getting an xref out of the C domain
     #
-    target = match.group(2)
+    base_target = match.group(2)
     target_text = nodes.Text(match.group(0))
     xref = None
-    if not ((match.re == RE_function and target in Skipfuncs)
-            or (target in Skipnames)):
-        lit_text = nodes.literal(classes=['xref', 'c', class_str[match.re]])
-        lit_text += target_text
-        pxref = addnodes.pending_xref('', refdomain = 'c',
-                                      reftype = reftype_str[match.re],
-                                      reftarget = target, modname = None,
-                                      classname = None)
-        #
-        # XXX The Latex builder will throw NoUri exceptions here,
-        # work around that by ignoring them.
-        #
-        try:
-            xref = cdom.resolve_xref(app.env, docname, app.builder,
-                                     reftype_str[match.re], target, pxref,
-                                     lit_text)
-        except NoUri:
-            xref = None
-    #
-    # Return the xref if we got it; otherwise just return the plain text.
-    #
-    if xref:
-        return xref
+    possible_targets = [base_target]
+    # Check if this document has a namespace, and if so, try
+    # cross-referencing inside it first.
+    try:
+        namespace = c_namespace[docname]
+    except KeyError:
+        pass
     else:
-        return target_text
+        possible_targets.insert(0, namespace + "." + base_target)
+
+    if base_target not in Skipnames:
+        for target in possible_targets:
+            if not (match.re == RE_function and target in Skipfuncs):
+                lit_text = nodes.literal(classes=['xref', 'c', class_str[match.re]])
+                lit_text += target_text
+                pxref = addnodes.pending_xref('', refdomain = 'c',
+                                              reftype = reftype_str[match.re],
+                                              reftarget = target, modname = None,
+                                              classname = None)
+                #
+                # XXX The Latex builder will throw NoUri exceptions here,
+                # work around that by ignoring them.
+                #
+                try:
+                    xref = cdom.resolve_xref(app.env, docname, app.builder,
+                                             reftype_str[match.re], target, pxref,
+                                             lit_text)
+                except NoUri:
+                    xref = None
+
+                if xref:
+                    return xref
+
+    return target_text
 
 #
 # Try to replace a documentation reference of the form Documentation/... with a
@@ -246,9 +271,18 @@ def auto_markup(app, doctree, name):
             if not isinstance(node.parent, nodes.literal):
                 node.parent.replace(node, markup_refs(name, app, node))
 
+def save_c_namespace(app, docname, source):
+    lines = iter(source[0].splitlines(True))
+    for l in lines:
+        match = RE_namespace.search(l)
+        if match:
+            c_namespace[docname] = match.group(1)
+            return
+
 def setup(app):
+    app.connect('source-read', save_c_namespace)
     app.connect('doctree-resolved', auto_markup)
     return {
-        'parallel_read_safe': True,
+        'parallel_read_safe': False,
         'parallel_write_safe': True,
         }
-- 
2.28.0

Re: [PATCH v2 5/5] docs: automarkup.py: Allow automatic cross-reference inside C namespace

[Mauro Carvalho Chehab]

Em Tue, 13 Oct 2020 23:13:40 +0000
Nícolas F. R. A. Prado <nfraprado@protonmail.com> escreveu:

> Sphinx 3.1 introduced namespaces for C cross-references. With this,
> each C domain type/function declaration is put inside the namespace that
> was active at the time of its declaration.
> 
> To support automatic cross-reference inside C namespaces:
> - Save the C namespace used in each doc file (if any) at the source-read
>   phase, in the beginning of the Sphinx process.
> - When making the automarkup, if any namespace was used in the current
>   file, try to cross-reference to the symbol inside of it before trying
>   in the global namespace.
> 
> To make the first step possible, disable the parallel_read_safe option
> in Sphinx, since the dictionary that maps the files to the C namespaces
> can't be concurrently updated. This unfortunately increases the build
> time of the documentation.

Disabling parallel_read_safe will make performance very poor.
Doesn't the C domain store the current namespace somewhere?
If so, then, instead of using the source-read phase, something
else could be used instead.

> 
> Signed-off-by: Nícolas F. R. A. Prado <nfraprado@protonmail.com>
> ---
>  Documentation/sphinx/automarkup.py | 130 ++++++++++++++++++-----------
>  1 file changed, 82 insertions(+), 48 deletions(-)
> 
> diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
> index 409dbc4100de..bca8cf5f519d 100644
> --- a/Documentation/sphinx/automarkup.py
> +++ b/Documentation/sphinx/automarkup.py
> @@ -45,6 +45,8 @@ RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
>  #
>  RE_doc = re.compile(r'\bDocumentation(/[\w\-_/]+)(\.\w+)*')
>  
> +RE_namespace = re.compile(r'^\s*..\s*c:namespace::\s*(\S+)\s*$')
> +
>  #
>  # Reserved C words that we should skip when cross-referencing
>  #
> @@ -62,6 +64,8 @@ Skipfuncs = [ 'open', 'close', 'read', 'write', 'fcntl', 'mmap',
>                'select', 'poll', 'fork', 'execve', 'clone', 'ioctl',
>                'socket' ]
>  
> +c_namespace = {}
> +
>  def markup_refs(docname, app, node):
>      t = node.astext()
>      done = 0
> @@ -120,30 +124,42 @@ def markup_func_ref_sphinx3(docname, app, match):
>      #
>      # Go through the dance of getting an xref out of the C domain
>      #
> -    target = match.group(2)
> +    base_target = match.group(2)
>      target_text = nodes.Text(match.group(0))
>      xref = None
> -    if not (target in Skipfuncs or target in Skipnames):
> -        for class_s, reftype_s in zip(class_str, reftype_str):
> -            lit_text = nodes.literal(classes=['xref', 'c', class_s])
> -            lit_text += target_text
> -            pxref = addnodes.pending_xref('', refdomain = 'c',
> -                                          reftype = reftype_s,
> -                                          reftarget = target, modname = None,
> -                                          classname = None)
> -            #
> -            # XXX The Latex builder will throw NoUri exceptions here,
> -            # work around that by ignoring them.
> -            #
> -            try:
> -                xref = cdom.resolve_xref(app.env, docname, app.builder,
> -                                         reftype_s, target, pxref,
> -                                         lit_text)
> -            except NoUri:
> -                xref = None
> -
> -            if xref:
> -                return xref
> +    possible_targets = [base_target]
> +    # Check if this document has a namespace, and if so, try
> +    # cross-referencing inside it first.
> +    try:
> +        namespace = c_namespace[docname]
> +    except KeyError:
> +        pass
> +    else:
> +        possible_targets.insert(0, namespace + "." + base_target)
> +
> +    if base_target not in Skipnames:
> +        for target in possible_targets:
> +            if target not in Skipfuncs:
> +                for class_s, reftype_s in zip(class_str, reftype_str):
> +                    lit_text = nodes.literal(classes=['xref', 'c', class_s])
> +                    lit_text += target_text
> +                    pxref = addnodes.pending_xref('', refdomain = 'c',
> +                                                  reftype = reftype_s,
> +                                                  reftarget = target, modname = None,
> +                                                  classname = None)
> +                    #
> +                    # XXX The Latex builder will throw NoUri exceptions here,
> +                    # work around that by ignoring them.
> +                    #
> +                    try:
> +                        xref = cdom.resolve_xref(app.env, docname, app.builder,
> +                                                 reftype_s, target, pxref,
> +                                                 lit_text)
> +                    except NoUri:
> +                        xref = None
> +
> +                    if xref:
> +                        return xref
>  
>      return target_text
>  
> @@ -171,34 +187,43 @@ def markup_c_ref(docname, app, match):
>      #
>      # Go through the dance of getting an xref out of the C domain
>      #
> -    target = match.group(2)
> +    base_target = match.group(2)
>      target_text = nodes.Text(match.group(0))
>      xref = None
> -    if not ((match.re == RE_function and target in Skipfuncs)
> -            or (target in Skipnames)):
> -        lit_text = nodes.literal(classes=['xref', 'c', class_str[match.re]])
> -        lit_text += target_text
> -        pxref = addnodes.pending_xref('', refdomain = 'c',
> -                                      reftype = reftype_str[match.re],
> -                                      reftarget = target, modname = None,
> -                                      classname = None)
> -        #
> -        # XXX The Latex builder will throw NoUri exceptions here,
> -        # work around that by ignoring them.
> -        #
> -        try:
> -            xref = cdom.resolve_xref(app.env, docname, app.builder,
> -                                     reftype_str[match.re], target, pxref,
> -                                     lit_text)
> -        except NoUri:
> -            xref = None
> -    #
> -    # Return the xref if we got it; otherwise just return the plain text.
> -    #
> -    if xref:
> -        return xref
> +    possible_targets = [base_target]
> +    # Check if this document has a namespace, and if so, try
> +    # cross-referencing inside it first.
> +    try:
> +        namespace = c_namespace[docname]
> +    except KeyError:
> +        pass
>      else:
> -        return target_text
> +        possible_targets.insert(0, namespace + "." + base_target)
> +
> +    if base_target not in Skipnames:
> +        for target in possible_targets:
> +            if not (match.re == RE_function and target in Skipfuncs):
> +                lit_text = nodes.literal(classes=['xref', 'c', class_str[match.re]])
> +                lit_text += target_text
> +                pxref = addnodes.pending_xref('', refdomain = 'c',
> +                                              reftype = reftype_str[match.re],
> +                                              reftarget = target, modname = None,
> +                                              classname = None)
> +                #
> +                # XXX The Latex builder will throw NoUri exceptions here,
> +                # work around that by ignoring them.
> +                #
> +                try:
> +                    xref = cdom.resolve_xref(app.env, docname, app.builder,
> +                                             reftype_str[match.re], target, pxref,
> +                                             lit_text)
> +                except NoUri:
> +                    xref = None
> +
> +                if xref:
> +                    return xref
> +
> +    return target_text
>  
>  #
>  # Try to replace a documentation reference of the form Documentation/... with a
> @@ -246,9 +271,18 @@ def auto_markup(app, doctree, name):
>              if not isinstance(node.parent, nodes.literal):
>                  node.parent.replace(node, markup_refs(name, app, node))
>  
> +def save_c_namespace(app, docname, source):
> +    lines = iter(source[0].splitlines(True))
> +    for l in lines:
> +        match = RE_namespace.search(l)
> +        if match:
> +            c_namespace[docname] = match.group(1)
> +            return
> +
>  def setup(app):
> +    app.connect('source-read', save_c_namespace)
>      app.connect('doctree-resolved', auto_markup)
>      return {
> -        'parallel_read_safe': True,
> +        'parallel_read_safe': False,
>          'parallel_write_safe': True,
>          }



Thanks,
Mauro

Re: [PATCH v2 2/5] docs: automarkup.py: Fix regexes to solve sphinx 3 warnings

[Jonathan Corbet]

On Tue, 13 Oct 2020 23:13:17 +0000
Nícolas F. R. A. Prado <nfraprado@protonmail.com> wrote:

> The warnings were caused by the expressions matching words in the
> translated versions of the documentation, since any unicode character
> was matched.
> 
> Fix the regular expression by making the C regexes use ASCII

I don't quite understand this part, can you give an example of the kinds
of warnings you were seeing?

Thanks,

jon

Re: [PATCH v2 5/5] docs: automarkup.py: Allow automatic cross-reference inside C namespace

[Jonathan Corbet]

On Wed, 14 Oct 2020 11:56:44 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> > To make the first step possible, disable the parallel_read_safe option
> > in Sphinx, since the dictionary that maps the files to the C namespaces
> > can't be concurrently updated. This unfortunately increases the build
> > time of the documentation.  
> 
> Disabling parallel_read_safe will make performance very poor.
> Doesn't the C domain store the current namespace somewhere?
> If so, then, instead of using the source-read phase, something
> else could be used instead.

That seems like the best solution if it exists, yes.  Otherwise a simple
lock could be used around c_namespace to serialize access there, right?

Thanks,

jon

Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Dafna Hirschfeld]

Hi

Am 14.10.20 um 01:13 schrieb Nícolas F. R. A. Prado:
> While Sphinx 2 used a single c:type role for struct, union, enum and
> typedef, Sphinx 3 uses a specific role for each one.
> To keep backward compatibility, detect the Sphinx version and use the
> correct roles for that version.
> 
> Signed-off-by: Nícolas F. R. A. Prado <nfraprado@protonmail.com>
> ---
>   Documentation/sphinx/automarkup.py | 55 ++++++++++++++++++++++++++----
>   1 file changed, 49 insertions(+), 6 deletions(-)
> 
> diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
> index a1b0f554cd82..db13fb15cedc 100644
> --- a/Documentation/sphinx/automarkup.py
> +++ b/Documentation/sphinx/automarkup.py
> @@ -23,7 +23,21 @@ from itertools import chain
>   # bit tries to restrict matches to things that won't create trouble.
>   #
>   RE_function = re.compile(r'(([\w_][\w\d_]+)\(\))')
> -RE_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
> +
> +#
> +# Sphinx 2 uses the same :c:type role for struct, union, enum and typedef
> +#
> +RE_generic_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
> +
> +#
> +# Sphinx 3 uses a different C role for each one of struct, union, enum and
> +# typedef
> +#
> +RE_struct = re.compile(r'\b(struct)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> +RE_union = re.compile(r'\b(union)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> +RE_enum = re.compile(r'\b(enum)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> +RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=re.ASCII)

I use ubuntu 18.04, my default python is 2.7,
when running 'make htmldocs' with that fix I get:

AttributeError: 'module' object has no attribute 'ASCII'

Thanks,
Dafna

> +
>   #
>   # Detects a reference to a documentation page of the form Documentation/... with
>   # an optional extension
> @@ -48,9 +62,22 @@ def markup_refs(docname, app, node):
>       #
>       # Associate each regex with the function that will markup its matches
>       #
> -    markup_func = {RE_type: markup_c_ref,
> -                   RE_function: markup_c_ref,
> -                   RE_doc: markup_doc_ref}
> +    markup_func_sphinx2 = {RE_doc: markup_doc_ref,
> +                           RE_function: markup_c_ref,
> +                           RE_generic_type: markup_c_ref}
> +
> +    markup_func_sphinx3 = {RE_doc: markup_doc_ref,
> +                           RE_function: markup_c_ref,
> +                           RE_struct: markup_c_ref,
> +                           RE_union: markup_c_ref,
> +                           RE_enum: markup_c_ref,
> +                           RE_typedef: markup_c_ref}
> +
> +    if sphinx.version_info[0] >= 3:
> +        markup_func = markup_func_sphinx3
> +    else:
> +        markup_func = markup_func_sphinx2
> +
>       match_iterators = [regex.finditer(t) for regex in markup_func]
>       #
>       # Sort all references by the starting position in text
> @@ -79,8 +106,24 @@ def markup_refs(docname, app, node):
>   # type_name) with an appropriate cross reference.
>   #
>   def markup_c_ref(docname, app, match):
> -    class_str = {RE_function: 'c-func', RE_type: 'c-type'}
> -    reftype_str = {RE_function: 'function', RE_type: 'type'}
> +    class_str = {RE_function: 'c-func',
> +                 # Sphinx 2 only
> +                 RE_generic_type: 'c-type',
> +                 # Sphinx 3+ only
> +                 RE_struct: 'c-struct',
> +                 RE_union: 'c-union',
> +                 RE_enum: 'c-enum',
> +                 RE_typedef: 'c-type',
> +                 }
> +    reftype_str = {RE_function: 'function',
> +                   # Sphinx 2 only
> +                   RE_generic_type: 'type',
> +                   # Sphinx 3+ only
> +                   RE_struct: 'struct',
> +                   RE_union: 'union',
> +                   RE_enum: 'enum',
> +                   RE_typedef: 'type',
> +                   }
>   
>       cdom = app.env.domains['c']
>       #
> 

Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Jonathan Corbet]

On Fri, 30 Oct 2020 14:33:52 +0100
Dafna Hirschfeld <dafna.hirschfeld@collabora.com> wrote:

> > +RE_struct = re.compile(r'\b(struct)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> > +RE_union = re.compile(r'\b(union)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> > +RE_enum = re.compile(r'\b(enum)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> > +RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=re.ASCII)  
> 
> I use ubuntu 18.04, my default python is 2.7,
> when running 'make htmldocs' with that fix I get:
> 
> AttributeError: 'module' object has no attribute 'ASCII'

Argh...it seems that re.ASCII is Python3 only.

For the short term, I guess we'll need to hack in some sort of workaround.
The not-so-long-term intent, though, is to leave Python 2 behind.

Thanks for the report,

jon

Python 2.7 support and automarkup.py - Was: Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Mauro Carvalho Chehab]

Hi Dafna,

Em Fri, 30 Oct 2020 14:33:52 +0100
Dafna Hirschfeld <dafna.hirschfeld@collabora.com> escreveu:

> Hi
> 
> Am 14.10.20 um 01:13 schrieb Nícolas F. R. A. Prado:
> > While Sphinx 2 used a single c:type role for struct, union, enum and
> > typedef, Sphinx 3 uses a specific role for each one.
> > To keep backward compatibility, detect the Sphinx version and use the
> > correct roles for that version.
> > 
> > Signed-off-by: Nícolas F. R. A. Prado <nfraprado@protonmail.com>
> > ---
> >   Documentation/sphinx/automarkup.py | 55 ++++++++++++++++++++++++++----
> >   1 file changed, 49 insertions(+), 6 deletions(-)
> > 
> > diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
> > index a1b0f554cd82..db13fb15cedc 100644
> > --- a/Documentation/sphinx/automarkup.py
> > +++ b/Documentation/sphinx/automarkup.py
> > @@ -23,7 +23,21 @@ from itertools import chain
> >   # bit tries to restrict matches to things that won't create trouble.
> >   #
> >   RE_function = re.compile(r'(([\w_][\w\d_]+)\(\))')
> > -RE_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
> > +
> > +#
> > +# Sphinx 2 uses the same :c:type role for struct, union, enum and typedef
> > +#
> > +RE_generic_type = re.compile(r'(struct|union|enum|typedef)\s+([\w_][\w\d_]+)')
> > +
> > +#
> > +# Sphinx 3 uses a different C role for each one of struct, union, enum and
> > +# typedef
> > +#
> > +RE_struct = re.compile(r'\b(struct)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> > +RE_union = re.compile(r'\b(union)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> > +RE_enum = re.compile(r'\b(enum)\s+([a-zA-Z_]\w+)', flags=re.ASCII)
> > +RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=re.ASCII)  
> 
> I use ubuntu 18.04, my default python is 2.7,
> when running 'make htmldocs' with that fix I get:
> 
> AttributeError: 'module' object has no attribute 'ASCII'

FYI, there's a discussion at kernel-doc ML about dropping support for
python 2.7 at Kernel 5.11. While not explicitly mentioned at the
discussion, this is the rationale:

	https://www.python.org/doc/sunset-python-2/

As this is currently broken with Python 2.7, then perhaps we can
do that for 5.10 :-)

Jon,

What do you think? 

I see a few alternatives:

1) fix automarkup.py for it to work again with python 2.7;

2) conf.py could gain some logic to disable automarkup with
   Python < 3;

3) scripts/sphinx-pre-install already detects Python version. 
   It should likely be easy to ask the user to use python 3.x,
   if an older version is detected.

Doing (1) or (2) will require an additional step when we raise
the bar for Python version.

Thanks,
Mauro

Re: Python 2.7 support and automarkup.py - Was: Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Jonathan Corbet]

On Fri, 30 Oct 2020 15:10:26 +0100
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> I see a few alternatives:
> 
> 1) fix automarkup.py for it to work again with python 2.7;
> 
> 2) conf.py could gain some logic to disable automarkup with
>    Python < 3;
> 
> 3) scripts/sphinx-pre-install already detects Python version. 
>    It should likely be easy to ask the user to use python 3.x,
>    if an older version is detected.
> 
> Doing (1) or (2) will require an additional step when we raise
> the bar for Python version.

We haven't dropped support for Python 2 yet, so this constitutes a
regression.  My own approach would be something like this at the top of
automarkup.py:

	if python2:
	    ascii = 0
	else:
	    ascii = re.ASCII

...then s/re.ASCII/ascii/ throughout.  I can probably put together
something later this morning.

jon

Re: Python 2.7 support and automarkup.py - Was: Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Mauro Carvalho Chehab]

Em Fri, 30 Oct 2020 08:14:40 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> On Fri, 30 Oct 2020 15:10:26 +0100
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> 
> > I see a few alternatives:
> > 
> > 1) fix automarkup.py for it to work again with python 2.7;
> > 
> > 2) conf.py could gain some logic to disable automarkup with
> >    Python < 3;
> > 
> > 3) scripts/sphinx-pre-install already detects Python version. 
> >    It should likely be easy to ask the user to use python 3.x,
> >    if an older version is detected.
> > 
> > Doing (1) or (2) will require an additional step when we raise
> > the bar for Python version.  
> 
> We haven't dropped support for Python 2 yet, so this constitutes a
> regression.  My own approach would be something like this at the top of
> automarkup.py:
> 
> 	if python2:
> 	    ascii = 0
> 	else:
> 	    ascii = re.ASCII
> 
> ...then s/re.ASCII/ascii/ throughout.  I can probably put together
> something later this morning.

Makes sense.

> 
> jon



Thanks,
Mauro

Re: Python 2.7 support and automarkup.py - Was: Re: [PATCH v2 1/5] docs: automarkup.py: Use new C roles in Sphinx 3

[Matthew Wilcox]

On Fri, Oct 30, 2020 at 08:14:40AM -0600, Jonathan Corbet wrote:
> On Fri, 30 Oct 2020 15:10:26 +0100
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> 
> > I see a few alternatives:
> > 
> > 1) fix automarkup.py for it to work again with python 2.7;
> > 
> > 2) conf.py could gain some logic to disable automarkup with
> >    Python < 3;
> > 
> > 3) scripts/sphinx-pre-install already detects Python version. 
> >    It should likely be easy to ask the user to use python 3.x,
> >    if an older version is detected.
> > 
> > Doing (1) or (2) will require an additional step when we raise
> > the bar for Python version.
> 
> We haven't dropped support for Python 2 yet, so this constitutes a
> regression.  My own approach would be something like this at the top of
> automarkup.py:
> 
> 	if python2:
> 	    ascii = 0
> 	else:
> 	    ascii = re.ASCII
> 
> ...then s/re.ASCII/ascii/ throughout.  I can probably put together
> something later this morning.

Could we have a warning somewhere that python 2.7 is going to produce
inferior docs?

Alternatively, https://docs.python.org/2/library/re.html suggests
using "The third-party regex module".

Re: [PATCH v2 5/5] docs: automarkup.py: Allow automatic cross-reference inside C namespace

[Nícolas F. R. A. Prado]

On Wed Oct 14, 2020 at 4:19 PM -03, Jonathan Corbet wrote:
>
> On Wed, 14 Oct 2020 11:56:44 +0200
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>
> > > To make the first step possible, disable the parallel_read_safe option
> > > in Sphinx, since the dictionary that maps the files to the C namespaces
> > > can't be concurrently updated. This unfortunately increases the build
> > > time of the documentation.
> >
> > Disabling parallel_read_safe will make performance very poor.
> > Doesn't the C domain store the current namespace somewhere?
> > If so, then, instead of using the source-read phase, something
> > else could be used instead.

The issue is that C domain parsing happens at an earlier phase in the Sphinx
process, and the current stack containing the C namespace is long gone when we
get to do the automatic cross-referencing at the doctree-resolved phase.

Not only that, but the namespace isn't assigned to the file it's in, and
vice-versa, because Sphinx's interest is in assigning a C directive it is
currently reading to the current namespace, so there isn't any point in saving
which namespaces appeared at a given file. That is exactly what we want, but
Sphinx doesn't have that information.

For instance, printing all symbols from app.env.domaindata['c']['root_symbol']
shows every single C namespace, but the docname field in each of them is None.

That's why the way to go is to assign the namespaces to the files at the
source-read phase on our own.

> That seems like the best solution if it exists, yes. Otherwise a simple
> lock could be used around c_namespace to serialize access there, right?

Actually I was wrong when I said that the issue was that "they can't be
concurrently updated". When parallel_read_safe is enabled, Sphinx spawns
multiple processes rather than multiple threads, to get true concurrency by
sidestepping python's GIL. So the same c_namespace variable isn't even
accessible across the multiple processes.

Reading multiprocessing's documentation [1] it seems that memory could be shared
between the processes using Value or Array, but both would need to be passed to
the processes by the one who spawned them, that is, it would need to be done
from Sphinx's side.

So, at the moment I'm not really seeing a way to have this information be shared
concurrently by the python processes but I will keep searching.

Thanks,
Nícolas

[1] https://docs.python.org/3/library/multiprocessing.html#sharing-state-between-processes