linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
To: Linux Doc Mailing List <linux-doc@vger.kernel.org>
Cc: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>,
	"Jonathan Corbet" <corbet@lwn.net>,
	linux-kernel@vger.kernel.org
Subject: [PATCH v2 16/39] docs: kernel_abi.py: Handle with a lazy Sphinx parser
Date: Fri, 30 Oct 2020 08:40:35 +0100	[thread overview]
Message-ID: <4659b60795739308e34d2d00c57ee0742a9cd2ab.1604042072.git.mchehab+huawei@kernel.org> (raw)
In-Reply-To: <cover.1604042072.git.mchehab+huawei@kernel.org>

The Sphinx docutils parser is lazy: if the content is bigger than
a certain number of lines, it silenlty stops parsing it,
producing an incomplete content. This seems to be worse on newer
Sphinx versions, like 2.0.

So, change the logic to parse the contents per input file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/sphinx/kernel_abi.py | 39 ++++++++++++++++++++----------
 1 file changed, 26 insertions(+), 13 deletions(-)

diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index 096dec482e96..ce5f3b0ae811 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -37,6 +37,7 @@ import os
 import subprocess
 import sys
 import re
+import kernellog
 
 from os import path
 
@@ -80,12 +81,6 @@ class KernelCmd(Directive):
         "debug"     : directives.flag
     }
 
-    def warn(self, message, **replace):
-        replace["fname"]   = self.state.document.current_source
-        replace["line_no"] = replace.get("line_no", self.lineno)
-        message = ("%(fname)s:%(line_no)s: [kernel-abi WARN] : " + message) % replace
-        self.state.document.settings.env.app.warn(message, prefix="")
-
     def run(self):
 
         doc = self.state.document
@@ -111,7 +106,7 @@ class KernelCmd(Directive):
         shell_env["srctree"] = srctree
 
         lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env)
-        nodeList = self.nestedParse(lines, fname)
+        nodeList = self.nestedParse(lines, self.arguments[0])
         return nodeList
 
     def runCmd(self, cmd, **kwargs):
@@ -138,9 +133,9 @@ class KernelCmd(Directive):
                               % (self.name, ErrorString(exc)))
         return out
 
-    def nestedParse(self, lines, f):
+    def nestedParse(self, lines, fname):
         content = ViewList()
-        node    = nodes.section()
+        node = nodes.section()
 
         if "debug" in self.options:
             code_block = "\n\n.. code-block:: rst\n    :linenos:\n"
@@ -150,22 +145,42 @@ class KernelCmd(Directive):
 
         line_regex = re.compile("^#define LINENO (\S+)\#([0-9]+)$")
         ln = 0
+        n = 0
+        f = fname
 
         for line in lines.split("\n"):
+            n = n + 1
             match = line_regex.search(line)
             if match:
-                f = match.group(1)
+                new_f = match.group(1)
+
+                # Sphinx parser is lazy: it stops parsing contents in the
+                # middle, if it is too big. So, handle it per input file
+                if new_f != f and content:
+                    self.do_parse(content, node)
+                    content = ViewList()
+
+                f = new_f
+
                 # sphinx counts lines from 0
                 ln = int(match.group(2)) - 1
             else:
                 content.append(line, f, ln)
 
-        buf  = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter
+        kernellog.info(self.state.document.settings.env.app, "%s: parsed %i lines" % (fname, n))
 
+        if content:
+            self.do_parse(content, node)
+
+        return node.children
+
+    def do_parse(self, content, node):
         if Use_SSI:
             with switch_source_input(self.state, content):
                 self.state.nested_parse(content, 0, node, match_titles=1)
         else:
+            buf  = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter
+
             self.state.memo.title_styles  = []
             self.state.memo.section_level = 0
             self.state.memo.reporter      = AutodocReporter(content, self.state.memo.reporter)
@@ -173,5 +188,3 @@ class KernelCmd(Directive):
                 self.state.nested_parse(content, 0, node, match_titles=1)
             finally:
                 self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf
-
-        return node.children
-- 
2.26.2


  parent reply	other threads:[~2020-10-30  7:43 UTC|newest]

Thread overview: 62+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-30  7:40 [PATCH v2 00/39] ABI: add it to the documentation build system Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 01/39] scripts: get_abi.pl: change script to allow parsing in ReST mode Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 02/39] scripts: get_abi.pl: fix parsing on " Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 03/39] scripts: get_abi.pl: Allow optionally record from where a line came from Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 04/39] scripts: get_abi.pl: improve its parser to better catch up indentation Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 05/39] scripts: get_abi.pl: cleanup ABI cross-reference logic Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 06/39] scripts: get_abi.pl: detect duplicated ABI definitions Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 07/39] scripts: get_abi.pl: output users in ReST format Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 08/39] scripts: get_abi.pl: prevent duplicated file names Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 09/39] scripts: get_abi.pl: use bold font for ABI definitions Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 10/39] scripts: get_abi.pl: auto-generate cross references Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 11/39] docs: kernellog.py: add support for info() Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 12/39] docs: kernel_abi.py: add a script to parse ABI documentation Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 13/39] docs: kernel_abi.py: fix UTF-8 support Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 14/39] docs: kernel_abi.py: make it compatible with Sphinx 1.7+ Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 15/39] docs: kernel_abi.py: use --enable-lineno for get_abi.pl Mauro Carvalho Chehab
2020-10-30  7:40 ` Mauro Carvalho Chehab [this message]
2020-10-30  7:40 ` [PATCH v2 17/39] docs: add ABI documentation to the admin-guide book Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 18/39] docs: ABI: README: specify that files should be ReST compatible Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 19/39] docs: ABI: stable: make files " Mauro Carvalho Chehab
2020-10-30 10:04   ` Srinivas Kandagatla
2020-10-30  7:40 ` [PATCH v2 20/39] docs: ABI: testing: make the files compatible with ReST output Mauro Carvalho Chehab
2020-10-30  9:19   ` Fabrice Gasnier
2020-10-30 10:09     ` Mauro Carvalho Chehab
2020-11-02 11:04       ` Fabrice Gasnier
2020-11-02 12:46         ` Greg Kroah-Hartman
2020-11-02 14:42           ` Mauro Carvalho Chehab
2020-11-08 16:56             ` Jonathan Cameron
2020-11-10  7:26               ` Duplicated ABI entries - Was: " Mauro Carvalho Chehab
2020-11-10 18:18                 ` Randy Dunlap
2020-11-14 15:27                 ` Jonathan Cameron
2020-10-30 17:26   ` Frederic Barrat
2020-10-30  7:40 ` [PATCH v2 21/39] docs: ABI: sysfs-uevent: make it " Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 22/39] docs: ABI: make it parse ABI/stable as ReST-compatible files Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 23/39] docs: ABI: create a 2-depth index for ABI Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 24/39] docs: ABI: don't escape ReST-incompatible chars from obsolete and removed Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 25/39] docs: abi-testing.rst: enable --rst-sources when building docs Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 26/39] docs: Kconfig/Makefile: add a check for broken ABI files Mauro Carvalho Chehab
2020-10-30 16:49   ` Randy Dunlap
2020-10-30  7:40 ` [PATCH v2 27/39] docs: ABI: convert testing/configfs-acpi to ReST Mauro Carvalho Chehab
2020-10-30 14:33   ` Rafael J. Wysocki
2020-10-30  7:40 ` [PATCH v2 28/39] docs: ABI: fix syntax to be parsed using ReST notation Mauro Carvalho Chehab
2020-10-30 14:39   ` Rafael J. Wysocki
2020-11-02  5:50   ` Jinpu Wang
2020-11-02 18:39   ` Joseph, Jithu
2020-10-30  7:40 ` [PATCH v2 29/39] docs: ABI: vdso: use the right format for ABI Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 30/39] docs: ABI: sysfs-bus-nvdimm: " Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 31/39] docs: ABI: cleanup several ABI documents Mauro Carvalho Chehab
2020-10-30  8:27   ` Rojewski, Cezary
2020-10-30  9:49   ` Suzuki K Poulose
2020-10-30 11:11   ` Ilya Dryomov
2020-10-30 16:42   ` Mathieu Poirier
2020-10-30 23:23   ` Peter Chen
2020-11-01  3:43   ` Dmitry Torokhov
2020-10-30  7:40 ` [PATCH v2 32/39] docs: ABI: change read/write attributes Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 33/39] docs: ABI: stable: remove a duplicated documentation Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 34/39] docs: ABI: unify /sys/class/leds/<led>/brightness documentation Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 35/39] docs: ABI: sysfs-class-power: unify duplicated properties Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 36/39] docs: ABI: sysfs-c2port: remove a duplicated entry Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 37/39] docs: ABI: sysfs-class-backlight: unify ABI documentation Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 38/39] docs: ABI: sysfs-class-led-trigger-pattern: remove hw_pattern duplication Mauro Carvalho Chehab
2020-10-30  7:40 ` [PATCH v2 39/39] scripts: get_abi.pl: assume ReST format by default Mauro Carvalho Chehab

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4659b60795739308e34d2d00c57ee0742a9cd2ab.1604042072.git.mchehab+huawei@kernel.org \
    --to=mchehab+huawei@kernel.org \
    --cc=corbet@lwn.net \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).