[PATCH 0/5 v2] Proposed docs version changes

[PATCH 0/5 v2] Proposed docs version changes

[Richard Purdie]

I've tweaked my patchset with some review feedback from Quentin. The v2:

* uses yaml to load the variables in conf.py
* fixes a bitbake_mapping[i] -> branch
* fixes a comment typo in set_versions.py
* comments the bitbake langdale version mapping

Richard Purdie (5):
  Makefile/set_versions: Allow poky.yaml to be autogenerated
  conf.py/set_versions/poky.yaml: Set version in conf.py from poky.yaml
  set_versions: Add support for setting POKYVERSION found in older
    releases
  set_versions/switchers.js: Allow switchers.js version information to
    be autogenerated
  set_versions: Various improvements

 documentation/.gitignore                      |   2 +
 documentation/Makefile                        |   1 +
 documentation/conf.py                         |  22 +-
 documentation/{poky.yaml => poky.yaml.in}     |   2 +
 documentation/set_versions.py                 | 225 ++++++++++++++++++
 .../{switchers.js => switchers.js.in}         |   8 +-
 6 files changed, 251 insertions(+), 9 deletions(-)
 rename documentation/{poky.yaml => poky.yaml.in} (98%)
 create mode 100755 documentation/set_versions.py
 rename documentation/sphinx-static/{switchers.js => switchers.js.in} (97%)

-- 
2.32.0

[PATCH 1/5] Makefile/set_versions: Allow poky.yaml to be autogenerated

[Richard Purdie]

Use a script to generate the branch/tag information inside poky.yaml.

If the branch isn't a known release branch, include git magic to find
the closest matching release branch we know about.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 documentation/.gitignore                  |   1 +
 documentation/Makefile                    |   1 +
 documentation/{poky.yaml => poky.yaml.in} |   0
 documentation/set_versions.py             | 115 ++++++++++++++++++++++
 4 files changed, 117 insertions(+)
 rename documentation/{poky.yaml => poky.yaml.in} (100%)
 create mode 100755 documentation/set_versions.py

diff --git a/documentation/.gitignore b/documentation/.gitignore
index 35ead8af6..e5e2c1708 100644
--- a/documentation/.gitignore
+++ b/documentation/.gitignore
@@ -1,5 +1,6 @@
 _build/
 Pipfile.lock
+poky.yaml
 .vscode/
 */svg/*.png
 */svg/*.pdf
diff --git a/documentation/Makefile b/documentation/Makefile
index f04f381bd..bec53399c 100644
--- a/documentation/Makefile
+++ b/documentation/Makefile
@@ -57,4 +57,5 @@ all: html epub latexpdf
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %:
+	$(SOURCEDIR)/set_versions.py
 	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/documentation/poky.yaml b/documentation/poky.yaml.in
similarity index 100%
rename from documentation/poky.yaml
rename to documentation/poky.yaml.in
diff --git a/documentation/set_versions.py b/documentation/set_versions.py
new file mode 100755
index 000000000..266ccf464
--- /dev/null
+++ b/documentation/set_versions.py
@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+#
+# Add version information to poky.yaml based upon current git branch/tags
+#
+# Copyright Linux Foundation
+# Author: Richard Purdie <richard.purdie@linuxfoundation.org>
+#
+# SPDX-License-Identifier: MIT
+#
+
+
+import subprocess
+import collections
+import sys
+
+devbranch = "kirkstone"
+#devbranch = "langdale"
+ltsseries = ["kirkstone", "dunfell"]
+
+release_series = collections.OrderedDict()
+#release_series["langdale"] = "4.1"
+release_series["kirkstone"] = "4.0"
+release_series["honister"] = "3.4"
+release_series["hardknott"] = "3.3"
+release_series["gatesgarth"] = "3.2"
+release_series["dunfell"] = "3.1"
+
+ourversion = None
+ourseries = None
+ourbranch = None
+
+# Test tags exist and inform the user to fetch if not
+try:
+    subprocess.run(["git", "show", "yocto-3.4.2"], capture_output=True, check=True)
+except subprocess.CalledProcessError:
+    sys.exit("Please run 'git fetch --tags' before building the documentation")
+
+# Try and figure out what we are
+tags = subprocess.run(["git", "tag", "--points-at", "HEAD"], capture_output=True, text=True).stdout
+for t in tags.split():
+    if t.startswith("yocto-"):
+        ourversion = t[6:]
+
+if ourversion:
+    # We're a tagged release
+    components = ourversion.split(".")
+    baseversion = components[0] + "." + components[1]
+    for i in release_series:
+        if release_series[i] == baseversion:
+            ourseries = i
+            ourbranch = i
+else:
+    # We're floating on a branch
+    branch = subprocess.run(["git", "branch", "--show-current"], capture_output=True, text=True).stdout.strip()
+    ourbranch = branch
+    if branch != "master" and branch not in release_series:
+        possible_branches = []
+        for b in release_series.keys():
+            result = subprocess.run(["git", "show-ref", "heads/" + b], capture_output=True, text=True)
+            if result.returncode == 0:
+                possible_branches.append(b)
+                continue
+            result = subprocess.run(["git", "show-ref", "origin/" + b], capture_output=True, text=True)
+            if result.returncode == 0:
+                possible_branches.append("origin/" + b)
+        nearestbranch = subprocess.run('git show-branch master ' + ' '.join(possible_branches) + ' | grep "*" | grep -v "$(git rev-parse --abbrev-ref HEAD)" | head -n1', shell=True, capture_output=True, text=True).stdout
+        branch = nearestbranch.split('[')[1].split('~')[0]
+        print("Nearest release branch esimtated to be %s" % branch)
+    if branch == "master":
+        ourseries = devbranch
+    elif branch in release_series:
+        ourseries = branch
+    else:
+        sys.exit("Unknown series for branch %s" % branch)
+
+    previoustags = subprocess.run(["git", "tag", "--merged", "HEAD"], capture_output=True, text=True).stdout
+    previoustags = [t[6:] for t in previoustags.split() if t.startswith("yocto-" + release_series[ourseries])]
+    futuretags = subprocess.run(["git", "tag", "--merged", ourbranch], capture_output=True, text=True).stdout
+    futuretags = [t[6:] for t in futuretags.split() if t.startswith("yocto-" + release_series[ourseries])]
+
+    # Append .999 against the last known version
+    if len(previoustags) != len(futuretags):
+        ourversion = previoustags[-1] + ".999"
+    else:
+        ourversion = release_series[ourseries] + ".999"
+
+series = [k for k in release_series]
+previousseries = series[series.index(ourseries)+1:]
+lastlts = [k for k in previousseries if k in ltsseries]
+
+print("Version calculated to be %s" % ourversion)
+print("Release series calculated to be %s" % ourseries)
+
+replacements = {
+    "DISTRO" : ourversion,
+    "DISTRO_NAME_NO_CAP" : ourseries,
+    "DISTRO_NAME" : ourseries.capitalize(),
+    "DISTRO_NAME_NO_CAP_MINUS_ONE" : previousseries[0],
+    "DISTRO_NAME_NO_CAP_LTS" : lastlts[0],
+    "YOCTO_DOC_VERSION" : ourversion,
+    "DISTRO_REL_TAG" : "yocto-" + ourversion,
+}
+
+with open("poky.yaml.in", "r") as r, open("poky.yaml", "w") as w:
+    lines = r.readlines()
+    for line in lines:
+        data = line.split(":")
+        k = data[0].strip()
+        if k in replacements:
+            w.write("%s : \"%s\"\n" % (k, replacements[k]))
+        else:
+            w.write(line)
+
+print("poky.yaml generated from poky.yaml.in")
+
-- 
2.32.0

[PATCH 2/5] conf.py/set_versions/poky.yaml: Set version in conf.py from poky.yaml

[Richard Purdie]

Allow conf.py to read the versions it needs from poky.yaml and have
set_versions.py write this out. This means we don't have to change as
many files when making new releases.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 documentation/conf.py         | 22 ++++++++++++++++++++--
 documentation/poky.yaml.in    |  2 ++
 documentation/set_versions.py | 20 ++++++++++++++++++++
 3 files changed, 42 insertions(+), 2 deletions(-)

diff --git a/documentation/conf.py b/documentation/conf.py
index 3015892d2..a5d7c0cd8 100644
--- a/documentation/conf.py
+++ b/documentation/conf.py
@@ -15,9 +15,27 @@
 import os
 import sys
 import datetime
+try:
+    import yaml
+except ImportError:
+    sys.stderr.write("The Yocto Project Sphinx documentation requires PyYAML.\
+    \nPlease make sure to install pyyaml python package.\n")
+    sys.exit(1)
 
-current_version = "dev"
-bitbake_version = "" # Leave empty for development branch
+# current_version = "dev"
+# bitbake_version = "" # Leave empty for development branch
+# Obtain versions from poky.yaml instead
+with open("poky.yaml") as data:
+    buff = data.read()
+    subst_vars = yaml.safe_load(buff)
+    if "DOCCONF_VERSION" not in subst_vars:
+        sys.stderr.write("Please set DOCCONF_VERSION in poky.yaml")
+        sys.exit(1)
+    current_version = subst_vars["DOCCONF_VERSION"]
+    if "BITBAKE_SERIES" not in subst_vars:
+        sys.stderr.write("Please set BITBAKE_SERIES in poky.yaml")
+        sys.exit(1)
+    bitbake_version = subst_vars["BITBAKE_SERIES"]
 
 # String used in sidebar
 version = 'Version: ' + current_version
diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
index 89a059ef1..a346b7623 100644
--- a/documentation/poky.yaml.in
+++ b/documentation/poky.yaml.in
@@ -5,6 +5,8 @@ DISTRO_NAME_NO_CAP_MINUS_ONE : "hardknott"
 DISTRO_NAME_NO_CAP_LTS : "dunfell"
 YOCTO_DOC_VERSION : "3.4.2"
 DISTRO_REL_TAG : "yocto-3.4.2"
+DOCCONF_VERSION : "dev"
+BITBAKE_SERIES : ""
 YOCTO_DL_URL : "https://downloads.yoctoproject.org"
 YOCTO_AB_URL : "https://autobuilder.yoctoproject.org"
 YOCTO_RELEASE_DL_URL : "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"
diff --git a/documentation/set_versions.py b/documentation/set_versions.py
index 266ccf464..a4bfaf1bf 100755
--- a/documentation/set_versions.py
+++ b/documentation/set_versions.py
@@ -25,9 +25,20 @@ release_series["hardknott"] = "3.3"
 release_series["gatesgarth"] = "3.2"
 release_series["dunfell"] = "3.1"
 
+#    "langdale" : "2.2",
+bitbake_mapping = {
+    "kirkstone" : "2.0",
+    "honister" : "1.52",
+    "hardknott" : "1.50",
+    "gatesgarth" : "1.48",
+    "dunfell" : "1.46",
+}
+
 ourversion = None
 ourseries = None
 ourbranch = None
+bitbakeversion = None
+docconfver = None
 
 # Test tags exist and inform the user to fetch if not
 try:
@@ -45,10 +56,12 @@ if ourversion:
     # We're a tagged release
     components = ourversion.split(".")
     baseversion = components[0] + "." + components[1]
+    docconfver = ourversion
     for i in release_series:
         if release_series[i] == baseversion:
             ourseries = i
             ourbranch = i
+            bitbakeversion = bitbake_mapping[i]
 else:
     # We're floating on a branch
     branch = subprocess.run(["git", "branch", "--show-current"], capture_output=True, text=True).stdout.strip()
@@ -68,8 +81,11 @@ else:
         print("Nearest release branch esimtated to be %s" % branch)
     if branch == "master":
         ourseries = devbranch
+        docconfver = "dev"
+        bitbakeversion = ""
     elif branch in release_series:
         ourseries = branch
+        bitbakeversion = bitbake_mapping[branch]
     else:
         sys.exit("Unknown series for branch %s" % branch)
 
@@ -83,6 +99,8 @@ else:
         ourversion = previoustags[-1] + ".999"
     else:
         ourversion = release_series[ourseries] + ".999"
+    if not docconfver:
+        docconfver = ourversion
 
 series = [k for k in release_series]
 previousseries = series[series.index(ourseries)+1:]
@@ -99,6 +117,8 @@ replacements = {
     "DISTRO_NAME_NO_CAP_LTS" : lastlts[0],
     "YOCTO_DOC_VERSION" : ourversion,
     "DISTRO_REL_TAG" : "yocto-" + ourversion,
+    "DOCCONF_VERSION" : docconfver,
+    "BITBAKE_SERIES" : bitbakeversion,
 }
 
 with open("poky.yaml.in", "r") as r, open("poky.yaml", "w") as w:
-- 
2.32.0

[PATCH 3/5] set_versions: Add support for setting POKYVERSION found in older releases

[Richard Purdie]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 documentation/set_versions.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/documentation/set_versions.py b/documentation/set_versions.py
index a4bfaf1bf..a121125ac 100755
--- a/documentation/set_versions.py
+++ b/documentation/set_versions.py
@@ -34,6 +34,13 @@ bitbake_mapping = {
     "dunfell" : "1.46",
 }
 
+# 3.4 onwards doesn't have poky version
+poky_mapping = {
+    "3.3" : "25.0",
+    "3.2" : "24.0",
+    "3.1" : "23.0",
+}
+
 ourversion = None
 ourseries = None
 ourbranch = None
@@ -121,6 +128,10 @@ replacements = {
     "BITBAKE_SERIES" : bitbakeversion,
 }
 
+if release_series[ourseries] in poky_mapping:
+    pokyversion = poky_mapping[release_series[ourseries]] + "." + ourversion.rsplit(".", 1)[1]
+    replacements["POKYVERSION"] = pokyversion
+
 with open("poky.yaml.in", "r") as r, open("poky.yaml", "w") as w:
     lines = r.readlines()
     for line in lines:
-- 
2.32.0

[PATCH 4/5] set_versions/switchers.js: Allow switchers.js version information to be autogenerated

[Richard Purdie]

A horrible blunt hammer approach to updating the version information in
switchers.js based on the available tag information.

To merge and work correctly, this will need a change to the autobuilder-helper
docs generation code to pull the swicthers.js and script from master, then
to run the script. That should hopefully remove the need for other patching
even on old docs branches though.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 documentation/.gitignore                      |  1 +
 documentation/set_versions.py                 | 20 +++++++++++++++++++
 .../{switchers.js => switchers.js.in}         |  8 +-------
 3 files changed, 22 insertions(+), 7 deletions(-)
 rename documentation/sphinx-static/{switchers.js => switchers.js.in} (97%)

diff --git a/documentation/.gitignore b/documentation/.gitignore
index e5e2c1708..096b97ec2 100644
--- a/documentation/.gitignore
+++ b/documentation/.gitignore
@@ -1,6 +1,7 @@
 _build/
 Pipfile.lock
 poky.yaml
+sphinx-static/switchers.js
 .vscode/
 */svg/*.png
 */svg/*.pdf
diff --git a/documentation/set_versions.py b/documentation/set_versions.py
index a121125ac..668c42e30 100755
--- a/documentation/set_versions.py
+++ b/documentation/set_versions.py
@@ -13,6 +13,7 @@ import subprocess
 import collections
 import sys
 
+activereleases = ["honister", "hardknott", "dunfell"]
 devbranch = "kirkstone"
 #devbranch = "langdale"
 ltsseries = ["kirkstone", "dunfell"]
@@ -144,3 +145,22 @@ with open("poky.yaml.in", "r") as r, open("poky.yaml", "w") as w:
 
 print("poky.yaml generated from poky.yaml.in")
 
+with open("sphinx-static/switchers.js.in", "r") as r, open("sphinx-static/switchers.js", "w") as w:
+    lines = r.readlines()
+    for line in lines:
+        if "VERSIONS_PLACEHOLDER" in line:
+            w.write("    'dev': 'dev (%s)',\n" % release_series[devbranch])
+            for branch in activereleases:
+                if branch == devbranch:
+                    continue
+                versions = subprocess.run('git tag --list yocto-%s*' % (release_series[branch]), shell=True, capture_output=True, text=True).stdout.split()
+                versions = sorted([v.replace("yocto-" +  release_series[branch] + ".", "").replace("yocto-" +  release_series[branch], "0") for v in versions], key=int)
+                version = release_series[branch]
+                if versions[-1] != "0":
+                    version = version + "." + versions[-1]
+                w.write("    '%s': '%s',\n" % (version, version))
+        else:
+            w.write(line)
+
+print("switchers.js generated from switchers.js.in")
+
diff --git a/documentation/sphinx-static/switchers.js b/documentation/sphinx-static/switchers.js.in
similarity index 97%
rename from documentation/sphinx-static/switchers.js
rename to documentation/sphinx-static/switchers.js.in
index 3ea8927d7..5d3a4d793 100644
--- a/documentation/sphinx-static/switchers.js
+++ b/documentation/sphinx-static/switchers.js.in
@@ -10,13 +10,7 @@ by https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/run-docs-b
   'use strict';
 
   var all_versions = {
-    'dev': 'dev (3.5)',
-    '3.4.2': '3.4.2',
-    '3.3.5': '3.3.5',
-    '3.2.4': '3.2.4',
-    '3.1.14': '3.1.14',
-    '3.0.4': '3.0.4',
-    '2.7.4': '2.7.4',
+    VERSIONS_PLACEHOLDER
   };
 
   var all_doctypes = {
-- 
2.32.0

[PATCH 5/5] set_versions: Various improvements

[Richard Purdie]

- Allow specifying the version from the commandline
- Add all previous release series/version mappings (to support transitions branch)
- Add poky mapping for 3.4 as some releases erronously use it
- Improve git branch 'guessing' code to work properly
- Handle poky '.0' release mappings correctly
- Only write poky.yaml if poky.yaml.in exists
- Ensure older non-active releases are shown in the switchers.js
  release list.
- Ensure current version and current release series are shown in switcher.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 documentation/set_versions.py | 119 +++++++++++++++++++++++++---------
 1 file changed, 89 insertions(+), 30 deletions(-)

diff --git a/documentation/set_versions.py b/documentation/set_versions.py
index 668c42e30..354776c16 100755
--- a/documentation/set_versions.py
+++ b/documentation/set_versions.py
@@ -12,10 +12,16 @@
 import subprocess
 import collections
 import sys
+import os
+import itertools
+
+ourversion = None
+if len(sys.argv) == 2:
+    ourversion = sys.argv[1]
 
 activereleases = ["honister", "hardknott", "dunfell"]
-devbranch = "kirkstone"
 #devbranch = "langdale"
+devbranch = "kirkstone"
 ltsseries = ["kirkstone", "dunfell"]
 
 release_series = collections.OrderedDict()
@@ -25,6 +31,27 @@ release_series["honister"] = "3.4"
 release_series["hardknott"] = "3.3"
 release_series["gatesgarth"] = "3.2"
 release_series["dunfell"] = "3.1"
+release_series["zeus"] = "3.0"
+release_series["warrior"] = "2.7"
+release_series["thud"] = "2.6"
+release_series["sumo"] = "2.5"
+release_series["rocko"] = "2.4"
+release_series["pyro"] = "2.3"
+release_series["morty"] = "2.2"
+release_series["krogoth"] = "2.1"
+release_series["jethro"] = "2.0"
+release_series["jethro-pre"] = "1.9"
+release_series["fido"] = "1.8"
+release_series["dizzy"] = "1.7"
+release_series["daisy"] = "1.6"
+release_series["dora"] = "1.5"
+release_series["dylan"] = "1.4"
+release_series["danny"] = "1.3"
+release_series["denzil"] = "1.2"
+release_series["edison"] = "1.1"
+release_series["bernard"] = "1.0"
+release_series["laverne"] = "0.9"
+
 
 #    "langdale" : "2.2",
 bitbake_mapping = {
@@ -36,13 +63,14 @@ bitbake_mapping = {
 }
 
 # 3.4 onwards doesn't have poky version
+# Early 3.4 release docs do reference it though
 poky_mapping = {
+    "3.4" : "26.0",
     "3.3" : "25.0",
     "3.2" : "24.0",
     "3.1" : "23.0",
 }
 
-ourversion = None
 ourseries = None
 ourbranch = None
 bitbakeversion = None
@@ -69,31 +97,38 @@ if ourversion:
         if release_series[i] == baseversion:
             ourseries = i
             ourbranch = i
-            bitbakeversion = bitbake_mapping[i]
+            if i in bitbake_mapping:
+                bitbakeversion = bitbake_mapping[i]
 else:
     # We're floating on a branch
     branch = subprocess.run(["git", "branch", "--show-current"], capture_output=True, text=True).stdout.strip()
     ourbranch = branch
     if branch != "master" and branch not in release_series:
-        possible_branches = []
-        for b in release_series.keys():
-            result = subprocess.run(["git", "show-ref", "heads/" + b], capture_output=True, text=True)
+        # We're not on a known release branch so we have to guess. Compare the numbers of commits
+        # from each release branch and assume the smallest number of commits is the one we're based off
+        possible_branch = None
+        branch_count = 0
+        for b in itertools.chain(release_series.keys(), ["master"]):
+            result = subprocess.run(["git", "log", "--format=oneline",  "HEAD..origin/" + b], capture_output=True, text=True)
             if result.returncode == 0:
-                possible_branches.append(b)
-                continue
-            result = subprocess.run(["git", "show-ref", "origin/" + b], capture_output=True, text=True)
-            if result.returncode == 0:
-                possible_branches.append("origin/" + b)
-        nearestbranch = subprocess.run('git show-branch master ' + ' '.join(possible_branches) + ' | grep "*" | grep -v "$(git rev-parse --abbrev-ref HEAD)" | head -n1', shell=True, capture_output=True, text=True).stdout
-        branch = nearestbranch.split('[')[1].split('~')[0]
-        print("Nearest release branch esimtated to be %s" % branch)
+                count = result.stdout.count('\n')
+                if not possible_branch or count < branch_count:
+                    print("Branch %s has count %s" % (b, count))
+                    possible_branch = b
+                    branch_count = count
+        if possible_branch:
+            branch = possible_branch
+        else:
+            branch = "master"
+        print("Nearest release branch estimated to be %s" % branch)
     if branch == "master":
         ourseries = devbranch
         docconfver = "dev"
         bitbakeversion = ""
     elif branch in release_series:
         ourseries = branch
-        bitbakeversion = bitbake_mapping[branch]
+        if branch in bitbake_mapping:
+            bitbakeversion = bitbake_mapping[branch]
     else:
         sys.exit("Unknown series for branch %s" % branch)
 
@@ -111,8 +146,8 @@ else:
         docconfver = ourversion
 
 series = [k for k in release_series]
-previousseries = series[series.index(ourseries)+1:]
-lastlts = [k for k in previousseries if k in ltsseries]
+previousseries = series[series.index(ourseries)+1:] or [""]
+lastlts = [k for k in previousseries if k in ltsseries] or "dunfell"
 
 print("Version calculated to be %s" % ourversion)
 print("Release series calculated to be %s" % ourseries)
@@ -130,21 +165,40 @@ replacements = {
 }
 
 if release_series[ourseries] in poky_mapping:
-    pokyversion = poky_mapping[release_series[ourseries]] + "." + ourversion.rsplit(".", 1)[1]
+    pokyversion = poky_mapping[release_series[ourseries]]
+    if ourversion != release_series[ourseries]:
+        pokyversion = pokyversion + "." + ourversion.rsplit(".", 1)[1]
+    else:
+        pokyversion = pokyversion + ".0"
     replacements["POKYVERSION"] = pokyversion
 
-with open("poky.yaml.in", "r") as r, open("poky.yaml", "w") as w:
-    lines = r.readlines()
-    for line in lines:
-        data = line.split(":")
-        k = data[0].strip()
-        if k in replacements:
-            w.write("%s : \"%s\"\n" % (k, replacements[k]))
-        else:
-            w.write(line)
-
-print("poky.yaml generated from poky.yaml.in")
-
+if os.path.exists("poky.yaml.in"):
+    with open("poky.yaml.in", "r") as r, open("poky.yaml", "w") as w:
+        lines = r.readlines()
+        for line in lines:
+            data = line.split(":")
+            k = data[0].strip()
+            if k in replacements:
+                w.write("%s : \"%s\"\n" % (k, replacements[k]))
+            else:
+                w.write(line)
+
+    print("poky.yaml generated from poky.yaml.in")
+
+
+# In the switcher list of versions we display:
+#  - latest dev
+#  - latest stable release
+#  - latest LTS
+#  - latest for each releases listed as active
+#  - latest doc version in current series
+#  - current doc version
+# (with duplicates removed)
+
+if ourseries not in activereleases:
+    activereleases.append(ourseries)
+
+versions = []
 with open("sphinx-static/switchers.js.in", "r") as r, open("sphinx-static/switchers.js", "w") as w:
     lines = r.readlines()
     for line in lines:
@@ -155,10 +209,15 @@ with open("sphinx-static/switchers.js.in", "r") as r, open("sphinx-static/switch
                     continue
                 versions = subprocess.run('git tag --list yocto-%s*' % (release_series[branch]), shell=True, capture_output=True, text=True).stdout.split()
                 versions = sorted([v.replace("yocto-" +  release_series[branch] + ".", "").replace("yocto-" +  release_series[branch], "0") for v in versions], key=int)
+                if not versions:
+                    continue
                 version = release_series[branch]
                 if versions[-1] != "0":
                     version = version + "." + versions[-1]
+                versions.append(version)
                 w.write("    '%s': '%s',\n" % (version, version))
+            if ourversion not in versions:
+                w.write("    '%s': '%s',\n" % (ourversion, ourversion))
         else:
             w.write(line)
 
-- 
2.32.0

Re: [docs] [PATCH 0/5 v2] Proposed docs version changes

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 2224 bytes --]

On Mon, Mar 21, 2022 at 6:45 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> I've tweaked my patchset with some review feedback from Quentin. The v2:
>
> * uses yaml to load the variables in conf.py
> * fixes a bitbake_mapping[i] -> branch
> * fixes a comment typo in set_versions.py
> * comments the bitbake langdale version mapping
>
> Richard Purdie (5):
>   Makefile/set_versions: Allow poky.yaml to be autogenerated
>   conf.py/set_versions/poky.yaml: Set version in conf.py from poky.yaml
>   set_versions: Add support for setting POKYVERSION found in older
>     releases
>   set_versions/switchers.js: Allow switchers.js version information to
>     be autogenerated
>   set_versions: Various improvements
>
>  documentation/.gitignore                      |   2 +
>  documentation/Makefile                        |   1 +
>  documentation/conf.py                         |  22 +-
>  documentation/{poky.yaml => poky.yaml.in}     |   2 +
>  documentation/set_versions.py                 | 225 ++++++++++++++++++
>

Would it make sense to implement what you have in set_versions.py directly
in Sphinx 'native' code ? e.g. directly in conf.py, or in a module loaded
from conf.py (see how kernel does it at [1]) or yocto-vars extension, or as
its own Sphinx extension?

poky.yaml only purpose is to be loaded by yocto-vars.py which is a local
Sphinx extension we created.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/conf.py#n26


>  .../{switchers.js => switchers.js.in}         |   8 +-
>  6 files changed, 251 insertions(+), 9 deletions(-)
>  rename documentation/{poky.yaml => poky.yaml.in} (98%)
>  create mode 100755 documentation/set_versions.py
>  rename documentation/sphinx-static/{switchers.js => switchers.js.in}
> (97%)
>
> --
> 2.32.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#2663):
> https://lists.yoctoproject.org/g/docs/message/2663
> Mute This Topic: https://lists.yoctoproject.org/mt/89934133/1279857
> Group Owner: docs+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [
> nicolas.dechesne@linaro.org]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 3884 bytes --]

Re: [docs] [PATCH 0/5 v2] Proposed docs version changes

[Quentin Schulz]

Hi all,

On 3/21/22 23:06, Richard Purdie wrote:
> On Mon, 2022-03-21 at 22:56 +0100, Nicolas Dechesne wrote:
>>
>>
>> On Mon, Mar 21, 2022 at 6:45 PM Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>>> I've tweaked my patchset with some review feedback from Quentin. The v2:
>>>
>>> * uses yaml to load the variables in conf.py
>>> * fixes a bitbake_mapping[i] -> branch
>>> * fixes a comment typo in set_versions.py
>>> * comments the bitbake langdale version mapping
>>>
>>> Richard Purdie (5):
>>>    Makefile/set_versions: Allow poky.yaml to be autogenerated
>>>    conf.py/set_versions/poky.yaml: Set version in conf.py from poky.yaml
>>>    set_versions: Add support for setting POKYVERSION found in older
>>>      releases
>>>    set_versions/switchers.js: Allow switchers.js version information to
>>>      be autogenerated
>>>    set_versions: Various improvements
>>>
>>>   documentation/.gitignore                      |   2 +
>>>   documentation/Makefile                        |   1 +
>>>   documentation/conf.py                         |  22 +-
>>>   documentation/{poky.yaml => poky.yaml.in}     |   2 +
>>>   documentation/set_versions.py                 | 225 ++++++++++++++++++
>>>
>>
>>
>> Would it make sense to implement what you have in set_versions.py directly in
>> Sphinx 'native' code ? e.g. directly in conf.py, or in a module loaded from
>> conf.py (see how kernel does it at [1]) or yocto-vars extension, or as its own
>> Sphinx extension?
>>
>> poky.yaml only purpose is to be loaded by yocto-vars.py which is a local
>> Sphinx extension we created.
>  > Possibly, yes. I've done it like this as it was easier to validate it 
was doing
> the right things and also easier to apply in retrospect to previous docs
> releases. What we want to do going forward is an open question and it does feel
> like that script would be better off as a module that at least for master, the
> docs code called into.
> 
> I think that would make most sense as a follow on to this series if/as/when
> people have the time to work on it and test it accordingly. As it stands the
> series was already quite painful to develop (I got hooked on Friday and had to
> fix it on Saturday whilst I still had the pieces all in cache).
> 

Agreed. I don't think this patch series makes the doc building situation 
worse than it currently is so there's no reason to hold it off. Makes it 
even easier on stable release maintainers so it's good :)

I just would like the behavior to be unchanged wrt versions listed in 
the switchers.js, the rest is fine and can be modified later on.

Which versions are listed in switchers.js can be modified in a separate 
commit to start the discussion (or at least have a "note" somewhere to 
explain what are the expectations).

Cheers,
Quentin

Re: [docs] [PATCH 0/5 v2] Proposed docs version changes

[Richard Purdie]

On Tue, 2022-03-22 at 14:58 +0100, Quentin Schulz wrote:
> Hi all,
> 
> On 3/21/22 23:06, Richard Purdie wrote:
> > On Mon, 2022-03-21 at 22:56 +0100, Nicolas Dechesne wrote:
> > > 
> > > Would it make sense to implement what you have in set_versions.py directly in
> > > Sphinx 'native' code ? e.g. directly in conf.py, or in a module loaded from
> > > conf.py (see how kernel does it at [1]) or yocto-vars extension, or as its own
> > > Sphinx extension?
> > > 
> > > poky.yaml only purpose is to be loaded by yocto-vars.py which is a local
> > > Sphinx extension we created.
> >  > Possibly, yes. I've done it like this as it was easier to validate it 
> was doing
> > the right things and also easier to apply in retrospect to previous docs
> > releases. What we want to do going forward is an open question and it does feel
> > like that script would be better off as a module that at least for master, the
> > docs code called into.
> > 
> > I think that would make most sense as a follow on to this series if/as/when
> > people have the time to work on it and test it accordingly. As it stands the
> > series was already quite painful to develop (I got hooked on Friday and had to
> > fix it on Saturday whilst I still had the pieces all in cache).
> > 
> 
> Agreed. I don't think this patch series makes the doc building situation 
> worse than it currently is so there's no reason to hold it off. Makes it 
> even easier on stable release maintainers so it's good :)
> 
> I just would like the behavior to be unchanged wrt versions listed in 
> the switchers.js, the rest is fine and can be modified later on.

Just to be clear, you mean adding gatesgarth and zeus back?
Or adding the current version and the current version series?

I'm going to assume we're talking about the first and I'll redo things to try
and do that. I've decided I'd better retest everything having done that with the
side be side comparison so it may take a while :/.


Cheers,

Richard

Re: [docs] [PATCH 0/5 v2] Proposed docs version changes

[Quentin Schulz]

On 3/22/22 15:25, Richard Purdie wrote:
> On Tue, 2022-03-22 at 14:58 +0100, Quentin Schulz wrote:
>> Hi all,
>>
>> On 3/21/22 23:06, Richard Purdie wrote:
>>> On Mon, 2022-03-21 at 22:56 +0100, Nicolas Dechesne wrote:
>>>>
>>>> Would it make sense to implement what you have in set_versions.py directly in
>>>> Sphinx 'native' code ? e.g. directly in conf.py, or in a module loaded from
>>>> conf.py (see how kernel does it at [1]) or yocto-vars extension, or as its own
>>>> Sphinx extension?
>>>>
>>>> poky.yaml only purpose is to be loaded by yocto-vars.py which is a local
>>>> Sphinx extension we created.
>>>   > Possibly, yes. I've done it like this as it was easier to validate it
>> was doing
>>> the right things and also easier to apply in retrospect to previous docs
>>> releases. What we want to do going forward is an open question and it does feel
>>> like that script would be better off as a module that at least for master, the
>>> docs code called into.
>>>
>>> I think that would make most sense as a follow on to this series if/as/when
>>> people have the time to work on it and test it accordingly. As it stands the
>>> series was already quite painful to develop (I got hooked on Friday and had to
>>> fix it on Saturday whilst I still had the pieces all in cache).
>>>
>>
>> Agreed. I don't think this patch series makes the doc building situation
>> worse than it currently is so there's no reason to hold it off. Makes it
>> even easier on stable release maintainers so it's good :)
>>
>> I just would like the behavior to be unchanged wrt versions listed in
>> the switchers.js, the rest is fine and can be modified later on.
> 
> Just to be clear, you mean adding gatesgarth and zeus back?

Keeping the current behavior which is the latest dot version of all 
releases up to Warrior.

dev
3.4.2
3.3.5
3.2.4
3.1.14
3.0.4
2.7.4

is what I currently have.

Quentin

Re: [docs] [PATCH 0/5 v2] Proposed docs version changes

[Quentin Schulz]

On 3/22/22 15:54, Richard Purdie wrote:
> On Tue, 2022-03-22 at 15:33 +0100, Quentin Schulz wrote:
>> On 3/22/22 15:25, Richard Purdie wrote:
>>> On Tue, 2022-03-22 at 14:58 +0100, Quentin Schulz wrote:
>>>> Hi all,
>>>>
>>>> On 3/21/22 23:06, Richard Purdie wrote:
>>>>> On Mon, 2022-03-21 at 22:56 +0100, Nicolas Dechesne wrote:
>>>>>>
>>>>>> Would it make sense to implement what you have in set_versions.py directly in
>>>>>> Sphinx 'native' code ? e.g. directly in conf.py, or in a module loaded from
>>>>>> conf.py (see how kernel does it at [1]) or yocto-vars extension, or as its own
>>>>>> Sphinx extension?
>>>>>>
>>>>>> poky.yaml only purpose is to be loaded by yocto-vars.py which is a local
>>>>>> Sphinx extension we created.
>>>>>    > Possibly, yes. I've done it like this as it was easier to validate it
>>>> was doing
>>>>> the right things and also easier to apply in retrospect to previous docs
>>>>> releases. What we want to do going forward is an open question and it does feel
>>>>> like that script would be better off as a module that at least for master, the
>>>>> docs code called into.
>>>>>
>>>>> I think that would make most sense as a follow on to this series if/as/when
>>>>> people have the time to work on it and test it accordingly. As it stands the
>>>>> series was already quite painful to develop (I got hooked on Friday and had to
>>>>> fix it on Saturday whilst I still had the pieces all in cache).
>>>>>
>>>>
>>>> Agreed. I don't think this patch series makes the doc building situation
>>>> worse than it currently is so there's no reason to hold it off. Makes it
>>>> even easier on stable release maintainers so it's good :)
>>>>
>>>> I just would like the behavior to be unchanged wrt versions listed in
>>>> the switchers.js, the rest is fine and can be modified later on.
>>>
>>> Just to be clear, you mean adding gatesgarth and zeus back?
>>
>> Keeping the current behavior which is the latest dot version of all
>> releases up to Warrior.
>>
>> dev
>> 3.4.2
>> 3.3.5
>> 3.2.4
>> 3.1.14
>> 3.0.4
>> 2.7.4
>>
>> is what I currently have.
> 
> Sadly I can't make this work easily since there are no tags for 2.7.2, 2.7.3
> ,2.7.4, 3.0.2, 3.0.3 or 3.0.4. There are a bunch of older releases missing
> too :(
> 

Quite unfortunate...

We could have 3.0.4 and 2.7.4 hardcoded in switchers.js and use git tags 
for the rest?

Quentin

Re: [docs] [PATCH 0/5 v2] Proposed docs version changes

[Michael Halstead]

[-- Attachment #1: Type: text/plain, Size: 3333 bytes --]

These tags are in place. I back-dated them to the release day but I put
them in my name since I've added these without a release engineer involved.



On Tue, Mar 22, 2022 at 8:00 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Tue, 2022-03-22 at 14:54 +0000, Richard Purdie via
> lists.yoctoproject.org
> wrote:
> > On Tue, 2022-03-22 at 15:33 +0100, Quentin Schulz wrote:
> > > On 3/22/22 15:25, Richard Purdie wrote:
> > > > On Tue, 2022-03-22 at 14:58 +0100, Quentin Schulz wrote:
> > > > > Hi all,
> > > > >
> > > > > On 3/21/22 23:06, Richard Purdie wrote:
> > > > > > On Mon, 2022-03-21 at 22:56 +0100, Nicolas Dechesne wrote:
> > > > > > >
> > > > > > > Would it make sense to implement what you have in
> set_versions.py directly in
> > > > > > > Sphinx 'native' code ? e.g. directly in conf.py, or in a
> module loaded from
> > > > > > > conf.py (see how kernel does it at [1]) or yocto-vars
> extension, or as its own
> > > > > > > Sphinx extension?
> > > > > > >
> > > > > > > poky.yaml only purpose is to be loaded by yocto-vars.py which
> is a local
> > > > > > > Sphinx extension we created.
> > > > > >   > Possibly, yes. I've done it like this as it was easier to
> validate it
> > > > > was doing
> > > > > > the right things and also easier to apply in retrospect to
> previous docs
> > > > > > releases. What we want to do going forward is an open question
> and it does feel
> > > > > > like that script would be better off as a module that at least
> for master, the
> > > > > > docs code called into.
> > > > > >
> > > > > > I think that would make most sense as a follow on to this series
> if/as/when
> > > > > > people have the time to work on it and test it accordingly. As
> it stands the
> > > > > > series was already quite painful to develop (I got hooked on
> Friday and had to
> > > > > > fix it on Saturday whilst I still had the pieces all in cache).
> > > > > >
> > > > >
> > > > > Agreed. I don't think this patch series makes the doc building
> situation
> > > > > worse than it currently is so there's no reason to hold it off.
> Makes it
> > > > > even easier on stable release maintainers so it's good :)
> > > > >
> > > > > I just would like the behavior to be unchanged wrt versions listed
> in
> > > > > the switchers.js, the rest is fine and can be modified later on.
> > > >
> > > > Just to be clear, you mean adding gatesgarth and zeus back?
> > >
> > > Keeping the current behavior which is the latest dot version of all
> > > releases up to Warrior.
> > >
> > > dev
> > > 3.4.2
> > > 3.3.5
> > > 3.2.4
> > > 3.1.14
> > > 3.0.4
> > > 2.7.4
> > >
> > > is what I currently have.
> >
> > Sadly I can't make this work easily since there are no tags for 2.7.2,
> 2.7.3
> > ,2.7.4, 3.0.2, 3.0.3 or 3.0.4. There are a bunch of older releases
> missing
> > too :(
>
> Michael(H), would you be able to add some tags:
>
> yocto-2.7.4 3fd1432890f04fd659952c0f20f5750fb8c3a909
> yocto-2.7.3 a06e6bdf1d8f133dfaf6dad9c49be24f688799c4
> yocto-2.7.2 ee29f61b92c5ca7dd56393cef5fcd3ff3fe243aa
> yocto-3.0.2 ac75b463b23ae6258c1e6f3bbdefc398f3a03d43
> yocto-3.0.3 9d522e747d2715aa0b2fe4c81cacc34bfad3e1a3
> yocto-3.0.4 9f51be3a11b7422aba3617a90a98336c3c75f71e
>
> please?
>
> Cheers,
>
> Richard
>
>

-- 
Michael Halstead
Linux Foundation / Yocto Project
Systems Operations Engineer

[-- Attachment #2: Type: text/html, Size: 4666 bytes --]

Re: [docs] [PATCH 1/5] Makefile/set_versions: Allow poky.yaml to be autogenerated

[Quentin Schulz]

Hi Michael,

On 3/23/22 10:46, Michael Opdenacker via lists.yoctoproject.org wrote:
> Hi Richard,
> 
> On 3/21/22 18:44, Richard Purdie wrote:
>> Use a script to generate the branch/tag information inside poky.yaml.
>>
>> If the branch isn't a known release branch, include git magic to find
>> the closest matching release branch we know about.
>>
>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>> ---
>>   documentation/.gitignore                  |   1 +
>>   documentation/Makefile                    |   1 +
>>   documentation/{poky.yaml => poky.yaml.in} |   0
>>   documentation/set_versions.py             | 115 ++++++++++++++++++++++
>>   4 files changed, 117 insertions(+)
>>   rename documentation/{poky.yaml => poky.yaml.in} (100%)
>>   create mode 100755 documentation/set_versions.py
> 
> A minor issue here: "make clean"  doesn't remove "documentation/poky.yaml" .
> 

This is technically not an issue because open("poky.yaml", "w") 
overwrites the file if it already exists.

Though, we should make sure the run-docs-build script in 
yocto-autobuilder-helper repo actually runs git clean -ffdx before 
checking out another branch otherwise there can be a conflict because 
poky.yaml exists locally and would be overridden/overwritten by 
poky.yaml from an earlier release and fail the checkout. IIRC Richard 
had that in his patch so we should be good.

> What do you prefer? I submit a fix later and we can merge the branch, or
> you propose an update, at least for this patch?

I feel like a fixup patch is just fine, does not seem to be a blocker?

Cheers,
Quentin

Re: [docs] [PATCH 0/5 v2] Proposed docs version changes

[Richard Purdie]

On Tue, 2022-03-22 at 17:47 -0700, Michael Halstead wrote:
> These tags are in place. I back-dated them to the release day but I put them
> in my name since I've added these without a release engineer involved.
> > > > 

Thanks, this helps immensely!

Cheers,

Richard