[PATCH 1/3] docs: ref-manual: ref-system-requirements: update requirements to build Sphinx docs

[PATCH 1/3] docs: ref-manual: ref-system-requirements: update requirements to build Sphinx docs

[Quentin Schulz]

Tested with containers on all supported distributions.

Debian 8 (Jessie) still has Python3.4 and an old pip3, which makes it
impossible to build typing module which is a requirement of "new" Sphinx
python module.

One cannot update to latest pip3 from within pip in Jessie's version.
One cannot get a newer pip from upstream because newer pip don't support
Python3.4 anymore.
One cannot build with python3-sphinx package from Jessie because the
package is too old (1.2.3) and does not have sphinx.ext.autosectionlabel
module which appeared in 1.4 version.

Signed-off-by: Quentin Schulz <foss@0leil.net>
---
 documentation/poky.yaml                       |  1 +
 .../ref-manual/ref-system-requirements.rst    | 24 ++++++++++++-------
 2 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/documentation/poky.yaml b/documentation/poky.yaml
index ec21271a3..946387795 100644
--- a/documentation/poky.yaml
+++ b/documentation/poky.yaml
@@ -87,3 +87,4 @@ CENTOS8_HOST_PACKAGES_ESSENTIAL : "-y epel-release
      socat perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python3-pip \
      python3-GitPython python3-jinja2 python3-pexpect xz which SDL-devel xterm \
      rpcgen mesa-libGL-devel"
+PIP3_HOST_PACKAGES_DOC : "$ sudo pip3 install sphinx sphinx_rtd_theme pyyaml"
diff --git a/documentation/ref-manual/ref-system-requirements.rst b/documentation/ref-manual/ref-system-requirements.rst
index b03a266ef..e0584fa05 100644
--- a/documentation/ref-manual/ref-system-requirements.rst
+++ b/documentation/ref-manual/ref-system-requirements.rst
@@ -141,7 +141,14 @@ supported Ubuntu or Debian Linux distribution:
    Yocto Project documentation manuals:
    ::
 
-      $ sudo apt-get install make xsltproc docbook-utils fop dblatex xmlto
+      $ sudo apt-get install make python3-pip
+      &PIP3_HOST_PACKAGES_DOC;
+
+   .. note::
+
+      It is currently not possible to build out documentation from Debian 8
+      (Jessie) because of outdated ``pip3`` and ``python3``. ``python3-sphinx``
+      is too outdated.
 
 Fedora Packages
 ---------------
@@ -159,8 +166,8 @@ supported Fedora Linux distribution:
    Yocto Project documentation manuals:
    ::
 
-      $ sudo dnf install  docbook-style-dsssl docbook-style-xsl \
-      docbook-dtds docbook-utils fop libxslt dblatex xmlto
+      $ sudo dnf install make python3-pip which
+      &PIP3_HOST_PACKAGES_DOC;
 
 openSUSE Packages
 -----------------
@@ -178,7 +185,8 @@ supported openSUSE Linux distribution:
    Yocto Project documentation manuals:
    ::
 
-      $ sudo zypper install dblatex xmlto
+      $ sudo zypper install make python3-pip which
+      &PIP3_HOST_PACKAGES_DOC;
 
 
 CentOS-7 Packages
@@ -207,8 +215,8 @@ supported CentOS-7 Linux distribution:
    Yocto Project documentation manuals:
    ::
 
-      $ sudo yum install docbook-style-dsssl docbook-style-xsl \
-      docbook-dtds docbook-utils fop libxslt dblatex xmlto
+      $ sudo yum install make python3-pip which
+      &PIP3_HOST_PACKAGES_DOC;
 
 CentOS-8 Packages
 -----------------
@@ -239,8 +247,8 @@ supported CentOS-8 Linux distribution:
    Yocto Project documentation manuals:
    ::
 
-      $ sudo dnf install docbook-style-dsssl docbook-style-xsl \
-      docbook-dtds docbook-utils fop libxslt dblatex xmlto
+      $ sudo dnf install make python3-pip which
+      &PIP3_HOST_PACKAGES_DOC;
 
 Required Git, tar, Python and gcc Versions
 ==========================================
-- 
2.26.2

[PATCH 2/3] docs: sphinx: yocto-vars: rebuild files when poky.yaml has changed

[Quentin Schulz]

poky.yaml changes aren't detected by Sphinx by default.

In order to detect changes in poky.yaml, its md5sum is stored in the
app.outdir (BUILDDIR/html when building html) and checked against the
md5sum of the poky.yaml under use.

If the md5sum has changed, find all rst files in app.srcdir that have at
least an occurence of `&.*;` and mark them as requiring a rebuild.

Signed-off-by: Quentin Schulz <foss@0leil.net>
---
 documentation/sphinx/yocto-vars.py | 43 ++++++++++++++++++++++++++++--
 1 file changed, 41 insertions(+), 2 deletions(-)

diff --git a/documentation/sphinx/yocto-vars.py b/documentation/sphinx/yocto-vars.py
index 8083d7da1..8795eee0a 100644
--- a/documentation/sphinx/yocto-vars.py
+++ b/documentation/sphinx/yocto-vars.py
@@ -1,4 +1,6 @@
 #!/usr/bin/env python
+from hashlib import md5
+from pathlib import Path
 import re
 import sys
 
@@ -20,25 +22,62 @@ __version__  = '1.0'
 # Each .rst file is processed after source-read event (subst_vars_replace runs once per file)
 subst_vars = {}
 
+poky_hash = ""
+
 def subst_vars_replace(app: Sphinx, docname, source):
     result = source[0]
     for k in subst_vars:
         result = result.replace("&"+k+";", subst_vars[k])
     source[0] = result
 
+def yocto_vars_env_get_outdated(app: Sphinx, env, added, changed, removed):
+    '''
+    If poky.yaml changed (BUILDDIR/.poky.yaml.cache does not exist or contains
+    an md5sum different from poky.yaml's current md5sum), force rebuild of all
+    *.rst files in SOURCEDIR whose content has at least one occurence of `&.*;`
+    (see PATTERN global variable).
+    '''
+    try:
+        poky_cache = Path(app.outdir) / ".poky.yaml.cache"
+        cache_hash = poky_cache.read_text()
+    except FileNotFoundError:
+        cache_hash = None
+
+    if poky_hash == cache_hash:
+        return []
+
+    docs = []
+    for p in Path(app.srcdir).rglob("*.rst"):
+        if PATTERN.search(p.read_text()):
+            p_rel_no_ext = p.relative_to(app.srcdir).parent / p.stem
+            docs.append(str(p_rel_no_ext))
+    return docs
+
+def yocto_vars_build_finished(app: Sphinx, exception):
+    poky_cache = Path(app.outdir) / ".poky.yaml.cache"
+    poky_cache.write_text(poky_hash)
+    return []
+
 PATTERN = re.compile(r'&(.*?);')
 def expand(val, src):
     return PATTERN.sub(lambda m: expand(src.get(m.group(1), ''), src), val)
 
 def setup(app: Sphinx):
-    #FIXME: if poky.yaml changes, files are not reprocessed.
+    global poky_hash
+
     with open("poky.yaml") as file:
-        subst_vars.update(yaml.load(file, Loader=yaml.FullLoader))
+        hasher = md5()
+        buff = file.read()
+        hasher.update(buff.encode('utf-8'))
+        poky_hash = hasher.hexdigest()
+        subst_vars.update(yaml.safe_load(buff))
 
     for k in subst_vars:
         subst_vars[k] = expand(subst_vars[k], subst_vars)
 
     app.connect('source-read', subst_vars_replace)
+    app.connect('env-get-outdated', yocto_vars_env_get_outdated)
+    app.connect('build-finished', yocto_vars_build_finished)
 
     return dict(
         version = __version__,
-- 
2.26.2

[PATCH 3/3] docs: poky.yaml: fix identation in host packages variables

[Quentin Schulz]

This unfortunately makes the variables probably unusable for proper
replacement in other indentation blocks.

Signed-off-by: Quentin Schulz <foss@0leil.net>
---
 documentation/poky.yaml | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/documentation/poky.yaml b/documentation/poky.yaml
index 946387795..fc2adb357 100644
--- a/documentation/poky.yaml
+++ b/documentation/poky.yaml
@@ -71,18 +71,18 @@ FEDORA_HOST_PACKAGES_ESSENTIAL : "gawk make wget tar bzip2 gzip python3 unzip pe
 OPENSUSE_HOST_PACKAGES_ESSENTIAL : "python gcc gcc-c++ git chrpath make wget python-xml \
      diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip \
      python3-pexpect xz which python3-Jinja2 Mesa-libEGL1 libSDL-devel xterm rpcgen Mesa-dri-devel
-     $ sudo pip3 install GitPython"
+     \n\      $ sudo pip3 install GitPython"
 CENTOS7_HOST_PACKAGES_ESSENTIAL : "-y epel-release
-     $ sudo yum makecache
-     $ sudo yum install gawk make wget tar bzip2 gzip python3 unzip perl patch \
+     \n\      $ sudo yum makecache
+     \n\      $ sudo yum install gawk make wget tar bzip2 gzip python3 unzip perl patch \
      diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath socat \
      perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python36-pip xz \
      which SDL-devel xterm mesa-libGL-devel
-     $ sudo pip3 install GitPython jinja2"
+     \n\      $ sudo pip3 install GitPython jinja2"
 CENTOS8_HOST_PACKAGES_ESSENTIAL : "-y epel-release
-     $ sudo dnf config-manager --set-enabled PowerTools
-     $ sudo dnf makecache
-     $ sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl patch \
+     \n\      $ sudo dnf config-manager --set-enabled PowerTools
+     \n\      $ sudo dnf makecache
+     \n\      $ sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl patch \
      diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath ccache \
      socat perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python3-pip \
      python3-GitPython python3-jinja2 python3-pexpect xz which SDL-devel xterm \
-- 
2.26.2

Re: [docs] [PATCH 3/3] docs: poky.yaml: fix identation in host packages variables

[Nicolas Dechesne]

On Sun, Oct 4, 2020 at 3:22 PM Quentin Schulz <foss@0leil.net> wrote:
>
> This unfortunately makes the variables probably unusable for proper
> replacement in other indentation blocks.
>
> Signed-off-by: Quentin Schulz <foss@0leil.net>

For the whole series:

Reviewed-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>

really good addition to yocto-vars.py..

> ---
>  documentation/poky.yaml | 14 +++++++-------
>  1 file changed, 7 insertions(+), 7 deletions(-)
>
> diff --git a/documentation/poky.yaml b/documentation/poky.yaml
> index 946387795..fc2adb357 100644
> --- a/documentation/poky.yaml
> +++ b/documentation/poky.yaml
> @@ -71,18 +71,18 @@ FEDORA_HOST_PACKAGES_ESSENTIAL : "gawk make wget tar bzip2 gzip python3 unzip pe
>  OPENSUSE_HOST_PACKAGES_ESSENTIAL : "python gcc gcc-c++ git chrpath make wget python-xml \
>       diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip \
>       python3-pexpect xz which python3-Jinja2 Mesa-libEGL1 libSDL-devel xterm rpcgen Mesa-dri-devel
> -     $ sudo pip3 install GitPython"
> +     \n\      $ sudo pip3 install GitPython"
>  CENTOS7_HOST_PACKAGES_ESSENTIAL : "-y epel-release
> -     $ sudo yum makecache
> -     $ sudo yum install gawk make wget tar bzip2 gzip python3 unzip perl patch \
> +     \n\      $ sudo yum makecache
> +     \n\      $ sudo yum install gawk make wget tar bzip2 gzip python3 unzip perl patch \
>       diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath socat \
>       perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python36-pip xz \
>       which SDL-devel xterm mesa-libGL-devel
> -     $ sudo pip3 install GitPython jinja2"
> +     \n\      $ sudo pip3 install GitPython jinja2"
>  CENTOS8_HOST_PACKAGES_ESSENTIAL : "-y epel-release
> -     $ sudo dnf config-manager --set-enabled PowerTools
> -     $ sudo dnf makecache
> -     $ sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl patch \
> +     \n\      $ sudo dnf config-manager --set-enabled PowerTools
> +     \n\      $ sudo dnf makecache
> +     \n\      $ sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl patch \
>       diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath ccache \
>       socat perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python3-pip \
>       python3-GitPython python3-jinja2 python3-pexpect xz which SDL-devel xterm \
> --
> 2.26.2
>
>
> 
>