[PATCH v2 0/1] linux: generate refs automatically

[PATCH v2 0/1] linux: generate refs automatically

[Daniel Gomez]

Add support to generate git references automatically using git-ls-remote.

Changes since v1:
* Script generation moved and called from the main Makefile.
	* scripts/gen-refs.Makefile to generate refs.
* Conversion to python.
* Added support for:
	* Command-line arguments to control options.
	* YAML parser to configure extra and static references. Example added for mcgrof and mcgrof-next.
	* Filter for tags and heads. Example to filter semantic versioning would be --filter-tags "*.*.*".
	* File generation based on time. Update it every 24h (hardcoded). Possibility to
	  use --force to always generate the file.
	* git-ls-remote sorting inverted using --sort option and git versionsort.suffix
	  option (to avoid sort --reverse).

Note: Examples for static/extra configs are only applied for mcgrof's dev trees.
I can also add support for the other trees (jlayton, cel...). Let me know if you
want to give this a try for these trees also.

Daniel

Daniel Gomez (1):
  linux: generate refs automatically

 .gitignore                            |   2 +
 Makefile                              |   1 +
 scripts/gen-refs.Makefile             |  48 +++++
 scripts/generate_refs.py              | 242 ++++++++++++++++++++++++++
 workflows/linux/Kconfig               |   8 +-
 workflows/linux/Kconfig.linus         |  39 +----
 workflows/linux/Kconfig.mcgrof        |  42 +----
 workflows/linux/Kconfig.next          |   4 +-
 workflows/linux/Kconfig.stable        |  51 +-----
 workflows/linux/mcgrof-next-refs.yaml |  10 ++
 workflows/linux/mcgrof-refs.yaml      |  10 ++
 11 files changed, 322 insertions(+), 135 deletions(-)
 create mode 100644 scripts/gen-refs.Makefile
 create mode 100755 scripts/generate_refs.py
 create mode 100644 workflows/linux/mcgrof-next-refs.yaml
 create mode 100644 workflows/linux/mcgrof-refs.yaml

-- 
2.43.0

[PATCH v2 1/1] linux: generate refs automatically

[Daniel Gomez]

Generate a Kconfig choice/endchoice menu with a list of the latest git
refs (tags and heads) for upstream repositories.

The script will create the Kconfig choice options to be consumed by
their respective Kconfigs every 24h since last file update. This is to
avoid delaying the makefile execution while expecting an updated Kconfig
file with latest references.

Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
---
 .gitignore                            |   2 +
 Makefile                              |   1 +
 scripts/gen-refs.Makefile             |  48 +++++
 scripts/generate_refs.py              | 242 ++++++++++++++++++++++++++
 workflows/linux/Kconfig               |   8 +-
 workflows/linux/Kconfig.linus         |  39 +----
 workflows/linux/Kconfig.mcgrof        |  42 +----
 workflows/linux/Kconfig.next          |   4 +-
 workflows/linux/Kconfig.stable        |  51 +-----
 workflows/linux/mcgrof-next-refs.yaml |  10 ++
 workflows/linux/mcgrof-refs.yaml      |  10 ++
 11 files changed, 322 insertions(+), 135 deletions(-)
 create mode 100644 scripts/gen-refs.Makefile
 create mode 100755 scripts/generate_refs.py
 create mode 100644 workflows/linux/mcgrof-next-refs.yaml
 create mode 100644 workflows/linux/mcgrof-refs.yaml

diff --git a/.gitignore b/.gitignore
index d0499ddb..f77478f2 100644
--- a/.gitignore
+++ b/.gitignore
@@ -88,3 +88,5 @@ workflows/selftests/results/
 scripts/kconfig/.nconf-cfg
 .dynamic-kconfig.*
 vagrant/Kconfig.passthrough_libvirt.generated
+
+workflows/linux/Kconfig.refs-*
diff --git a/Makefile b/Makefile
index 9ca3a5f3..d1687889 100644
--- a/Makefile
+++ b/Makefile
@@ -180,6 +180,7 @@ endif
 
 include scripts/gen-hosts.Makefile
 include scripts/gen-nodes.Makefile
+include scripts/gen-refs.Makefile
 
 # disable built-in rules for this
 .SUFFIXES:
diff --git a/scripts/gen-refs.Makefile b/scripts/gen-refs.Makefile
new file mode 100644
index 00000000..92d8d238
--- /dev/null
+++ b/scripts/gen-refs.Makefile
@@ -0,0 +1,48 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+export SRC_URI_HTTPS_LINUX_NEXT = https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
+export SRC_URI_HTTPS_LINUX_STABLE = https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
+export SRC_URI_HTTPS_LINUX_LINUS = https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+export SRC_URI_HTTPS_MCGROF_NEXT = https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux-next.git
+
+GENERATED_REFS_NEXT := $(shell \
+	./scripts/generate_refs.py \
+		--repo $(SRC_URI_HTTPS_LINUX_NEXT) \
+		--prefix BOOTLINUX_TREE_NEXT \
+		--refs 15 \
+		--output workflows/linux/Kconfig.refs-next \
+	)
+
+GENERATED_REFS_STABLE := $(shell \
+	./scripts/generate_refs.py \
+		--repo $(SRC_URI_HTTPS_LINUX_STABLE) \
+		--prefix BOOTLINUX_TREE_STABLE \
+		--refs 15 \
+		--output workflows/linux/Kconfig.refs-stable \
+	)
+
+GENERATED_REFS_LINUS := $(shell \
+	./scripts/generate_refs.py \
+		--repo $(SRC_URI_HTTPS_LINUX_LINUS) \
+		--prefix BOOTLINUX_TREE_LINUS \
+		--refs 15 \
+		--output workflows/linux/Kconfig.refs-linus \
+	)
+
+GENERATED_REFS_MCGROF_NEXT := $(shell \
+	./scripts/generate_refs.py \
+		--repo $(SRC_URI_HTTPS_MCGROF_NEXT) \
+		--prefix BOOTLINUX_TREE_MCGROF_NEXT \
+		--refs 15 \
+		--output workflows/linux/Kconfig.refs-mcgrof-next \
+		--extra workflows/linux/mcgrof-next-refs.yaml \
+	)
+
+GENERATED_REFS_MCGROF := $(shell \
+	./scripts/generate_refs.py \
+		--repo $(SRC_URI_HTTPS_MCGROF_NEXT) \
+		--prefix BOOTLINUX_TREE_MCGROF \
+		--refs 15 \
+		--output workflows/linux/Kconfig.refs-mcgrof \
+		--extra workflows/linux/mcgrof-refs.yaml \
+	)
diff --git a/scripts/generate_refs.py b/scripts/generate_refs.py
new file mode 100755
index 00000000..4f096a2a
--- /dev/null
+++ b/scripts/generate_refs.py
@@ -0,0 +1,242 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: copyleft-next-0.3.1
+import os
+import subprocess
+import sys
+import logging
+import argparse
+import time
+import yaml
+
+
+def popen(
+    cmd: list[str], environ: dict[str, str] = {}, comm: bool = True
+) -> subprocess.Popen[bytes]:
+    try:
+        logging.debug(f"{' '.join(cmd)}")
+        p = subprocess.Popen(
+            cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=environ
+        )
+        if comm:
+            stdout, stderr = p.communicate()
+            if stdout:
+                logging.debug(stdout.decode("utf-8"))
+            if stderr:
+                logging.debug(stderr.decode("utf-8"))
+            logging.debug(f"Return code: {p.returncode}")
+        return {"process": p, "stdout": stdout.decode("utf-8")}
+    except subprocess.CalledProcessError as exc:
+        sys.exit(exc.returncode)
+
+
+def parser():
+    parser = argparse.ArgumentParser(description="Git Reference generate tool")
+    parser.add_argument(
+        "--repo",
+        help="the upstream repository to check tags with git-ls-remote",
+        required=True,
+    )
+    parser.add_argument(
+        "--prefix",
+        help="the Kconfig CONFIG prefix to use (e.g. BOOTLINUX_LINUS)",
+        required=True,
+    )
+    parser.add_argument(
+        "--filter-heads",
+        help="user additional ref filter for heads command",
+        default="*",
+    )
+    parser.add_argument(
+        "--filter-tags",
+        help="user additional ref filter for tags command",
+        default="*",
+    )
+    parser.add_argument(
+        "--output",
+        help="output file",
+        required=True,
+    )
+    parser.add_argument(
+        "--extra",
+        help="extra configs (yaml)",
+    )
+    parser.add_argument(
+        "--refs",
+        type=int,
+        default=1,
+        help="number of references",
+        required=True,
+    )
+    parser.add_argument("--debug", action="store_true", help="debug")
+    parser.add_argument(
+        "--force", action="store_true", help="always generate output file"
+    )
+    return parser
+
+
+def check_file_date(file):
+    if not os.path.exists(file):
+        return True
+    modified_time = os.path.getmtime(file)
+    current_time = time.time()
+    if current_time - modified_time > 86400:
+        return True
+    return False
+
+
+def _ref_generator_choices_static(args, conf_name, ref_name, ref_help):
+    refs = {}
+    with open(args.output, "a") as f:
+        logging.debug("static: {}".format(ref_name))
+        refs.update({ref_name: conf_name})
+        f.write("config {config}\n".format(config=conf_name))
+        f.write('\tbool "{ref}"\n'.format(ref=ref_name))
+        if ref_help:
+            f.write("\thelp\n")
+            f.write("\t\t{help}\n\n".format(help=ref_help))
+        else:
+            f.write("\n")
+    return refs
+
+
+def _ref_generator_choices(args, reflist, id) -> None:
+    refs = {}
+    for refline in reflist.splitlines():
+        if "^{}" in refline:
+            continue
+        if len(refs) >= args.refs:
+            break
+        ref_name = refline.split("/")[-1]
+        logging.debug("{}: {}".format(id, ref_name))
+        config = "{prefix}_REF_{id}".format(prefix=args.prefix, id=id)
+        refs.update({ref_name: config})
+        with open(args.output, "a") as f:
+            f.write("config {conf}\n".format(conf=config))
+            f.write('\t bool "{ref}"\n\n'.format(ref=ref_name))
+        id += 1
+    return refs
+
+
+def ref_generator(args, reflist, extraconfs) -> None:
+    """Generate output file. Example:
+
+    choice
+        prompt "Tag or branch to use"
+
+    config BOOTLINUX_TREE_NEXT_REF_0
+             bool "akpm"
+
+    config BOOTLINUX_TREE_NEXT_REF_1
+             bool "akpm-base"
+
+    config BOOTLINUX_TREE_NEXT_REF_2
+             bool "master"
+
+    config BOOTLINUX_TREE_NEXT_REF_3
+             bool "pending-fixes"
+
+    config BOOTLINUX_TREE_NEXT_REF_4
+             bool "stable"
+
+    endchoice
+
+    config BOOTLINUX_TREE_NEXT_REF
+            string
+            default "akpm" if BOOTLINUX_TREE_NEXT_REF_0
+            default "akpm-base" if BOOTLINUX_TREE_NEXT_REF_1
+            default "master" if BOOTLINUX_TREE_NEXT_REF_2
+            default "pending-fixes" if BOOTLINUX_TREE_NEXT_REF_3
+            default "stable" if BOOTLINUX_TREE_NEXT_REF_4
+    """
+    if os.path.exists(args.output):
+        os.remove(args.output)
+    refs = {}
+
+    with open(args.output, "a") as f:
+        f.write("choice\n")
+        f.write('\tprompt "Tag or branch to use"\n\n')
+
+    logging.debug("Generating...")
+    for rl in reflist:
+        refs.update(_ref_generator_choices(args, rl, len(refs)))
+
+    if extraconfs:
+        for config in extraconfs:
+            refs.update(
+                _ref_generator_choices_static(
+                    args, config["config"], config["ref"], config["help"]
+                )
+            )
+
+    with open(args.output, "a") as f:
+        f.write("endchoice\n\n")
+        f.write("config {prefix}_REF\n".format(prefix=args.prefix))
+        f.write("\tstring\n")
+        for r in refs.keys():
+            f.write('\tdefault "{ref}" if {conf}\n'.format(ref=r, conf=refs[r]))
+
+
+def remote(args) -> None:
+    heads = popen(
+        [
+            "git",
+            "-c",
+            "versionsort.suffix=-",
+            "ls-remote",
+            "--sort=-version:refname",
+            "--heads",
+            args.repo,
+            args.filter_heads,
+        ]
+    )
+    tags = popen(
+        [
+            "git",
+            "-c",
+            "versionsort.suffix=-",
+            "ls-remote",
+            "--sort=-version:refname",
+            "--tags",
+            args.repo,
+            args.filter_tags,
+        ]
+    )
+
+    return [heads["stdout"], tags["stdout"]]
+
+
+def main() -> None:
+    """Kconfig choice generator for git refereces"""
+    log = logging.getLogger()
+    log.setLevel(logging.INFO)
+    p = parser()
+    args, _ = p.parse_known_args()
+    if args.debug:
+        log.setLevel(logging.DEBUG)
+
+    if not args.force and not check_file_date(args.output):
+        logging.debug("File already updated")
+        return
+
+    extraconfs = []
+    if args.extra:
+        if os.path.exists(args.extra):
+            with open(args.extra, mode="rt") as f:
+                yml = yaml.safe_load(f)
+                extraconfs = yml["configs"]
+
+    reflist = remote(args)
+    ref_generator(args, reflist, extraconfs)
+    return
+
+
+if __name__ == "__main__":
+    ret = 0
+    try:
+        main()
+    except Exception:
+        ret = 1
+        import traceback
+
+        traceback.print_exc()
+    sys.exit(ret)
diff --git a/workflows/linux/Kconfig b/workflows/linux/Kconfig
index d4dd3abe..62cb5051 100644
--- a/workflows/linux/Kconfig
+++ b/workflows/linux/Kconfig
@@ -220,12 +220,12 @@ config BOOTLINUX_TEST_MESSAGE_ID_INSTALL_B4
 
 config BOOTLINUX_TREE_TAG
 	string
-	default BOOTLINUX_TREE_LINUS_TAG if BOOTLINUX_TREE_LINUS
-	default BOOTLINUX_TREE_STABLE_TAG if BOOTLINUX_STABLE
-	default BOOTLINUX_TREE_NEXT_TAG if BOOTLINUX_TREE_NEXT
+	default BOOTLINUX_TREE_LINUS_REF if BOOTLINUX_TREE_LINUS
+	default BOOTLINUX_TREE_STABLE_REF if BOOTLINUX_STABLE
+	default BOOTLINUX_TREE_NEXT_REF if BOOTLINUX_TREE_NEXT
 	default BOOTLINUX_TREE_BTRFS_DEVEL_TAG if BOOTLINUX_TREE_BTRFS_DEVEL
 	default BOOTLINUX_KDEVOPS_TREE_TAG if BOOTLINUX_TREE_KDEVOPS_LINUS
-	default BOOTLINUX_MCGROF_TREE_TAG if BOOTLINUX_MCGROF_TREE
+	default BOOTLINUX_TREE_MCGROF_REF if BOOTLINUX_MCGROF_TREE
 	default BOOTLINUX_TREE_JLAYTON_LINUX_TAG if BOOTLINUX_TREE_JLAYTON_LINUX
 	default BOOTLINUX_CEL_TREE_TAG if BOOTLINUX_TREE_CEL_LINUX
 	default BOOTLINUX_CUSTOM_TAG if BOOTLINUX_CUSTOM
diff --git a/workflows/linux/Kconfig.linus b/workflows/linux/Kconfig.linus
index 5c524e52..06e2c44d 100644
--- a/workflows/linux/Kconfig.linus
+++ b/workflows/linux/Kconfig.linus
@@ -23,44 +23,7 @@ endchoice
 
 if BOOTLINUX_TREE_LINUS
 
-choice
-	prompt "Linus' tag or branch to use"
-	default BOOTLINUX_LINUS_68_RC2
-
-config BOOTLINUX_LINUS_MASTER
-	bool "master"
-	help
-	  Use the master branch, the top of Linus' tree.
-
-config BOOTLINUX_LINUS_66_RC4
-	bool "v6.6-rc4"
-	help
-	  Use the v6.6-rc4 tag.
-
-config BOOTLINUX_LINUS_66_RC5
-	bool "v6.6-rc5"
-	help
-	  Use the v6.6-rc5 tag.
-
-config BOOTLINUX_LINUS_66_RC2
-	bool "v6.6-rc2"
-	help
-	  Use the v6.6-rc2 tag.
-
-config BOOTLINUX_LINUS_68_RC2
-	bool "v6.8-rc2"
-	help
-	  Use the v6.8-rc2 tag.
-
-endchoice
-
-config BOOTLINUX_TREE_LINUS_TAG
-	string
-	default "master" if BOOTLINUX_LINUS_MASTER
-	default "v6.6-rc2" if BOOTLINUX_LINUS_66_RC2
-	default "v6.6-rc4" if BOOTLINUX_LINUS_66_RC2
-	default "v6.6-rc5" if BOOTLINUX_LINUS_66_RC5
-	default "v6.8-rc2" if BOOTLINUX_LINUS_68_RC2
+source "workflows/linux/Kconfig.refs-linus"
 
 config BOOTLINUX_TREE_LINUS_NAME
 	string
diff --git a/workflows/linux/Kconfig.mcgrof b/workflows/linux/Kconfig.mcgrof
index 4c4be5b7..ec6a3162 100644
--- a/workflows/linux/Kconfig.mcgrof
+++ b/workflows/linux/Kconfig.mcgrof
@@ -1,43 +1,12 @@
 if BOOTLINUX_TREE_MCGROF_LINUS
 
-choice
-	prompt "mcgrof-linus branch to use"
-	default BOOTLINUX_MCGROF_LINUS_LBS
-
-config BOOTLINUX_MCGROF_LINUS_LBS
-	bool "large-block-linus"
-	help
-	  Use the latest LBS branch based on Linus' tree.
-
-config BOOTLINUX_MCGROF_LINUS_LBS_NOBDEV
-	bool "large-block-linus-nobdev"
-	help
-	  Use the latest minimal branch for LBS which let's you use LBS
-	  devices with a sectorsize compatibility based on Linus' tree.
-
-endchoice
+source "workflows/linux/Kconfig.refs-mcgrof"
 
 endif # BOOTLINUX_TREE_MCGROF_LINUS
 
 if BOOTLINUX_TREE_MCGROF_NEXT
 
-choice
-	prompt "mcgrof-next branch to use"
-	default BOOTLINUX_MCGROF_NEXT_LBS
-
-config BOOTLINUX_MCGROF_NEXT_LBS
-	bool "large-block-next"
-	help
-	  Use the latest stable LBS branch which enables all the bells and
-	  whistles for all types of LBS devices.
-
-config BOOTLINUX_MCGROF_NEXT_LBS_NOBDEV
-	bool "large-block-nobdev"
-	help
-	  Use the latest minimal branch for LBS which let's you use LBS
-	  devices with a sectorsize compatibility.
-
-endchoice
+source "workflows/linux/Kconfig.refs-mcgrof-next"
 
 endif # BOOTLINUX_TREE_MCGROF_NEXT
 
@@ -47,13 +16,6 @@ config BOOTLINUX_MCGROF_TREE
 	bool
 	default y
 
-config BOOTLINUX_MCGROF_TREE_TAG
-	string
-	default "large-block-linus" if BOOTLINUX_MCGROF_LINUS_LBS
-	default "large-block-linus-nobdev" if BOOTLINUX_MCGROF_LINUS_LBS_NOBDEV
-	default "large-block-20230915" if BOOTLINUX_MCGROF_NEXT_LBS
-	default "large-block-nobdev-20230915" if BOOTLINUX_MCGROF_NEXT_LBS_NOBDEV
-
 config BOOTLINUX_MCGROF_TREE_NAME
 	string
 	default "mcgrof-linus" if BOOTLINUX_TREE_MCGROF_LINUS
diff --git a/workflows/linux/Kconfig.next b/workflows/linux/Kconfig.next
index 48c86e3d..3ab841fc 100644
--- a/workflows/linux/Kconfig.next
+++ b/workflows/linux/Kconfig.next
@@ -1,8 +1,6 @@
 if BOOTLINUX_TREE_NEXT
 
-config BOOTLINUX_TREE_NEXT_TAG
-	string "linux-next tag to pull"
-	default "next-20240108"
+source "workflows/linux/Kconfig.refs-next"
 
 config BOOTLINUX_TREE_NEXT_NAME
 	string
diff --git a/workflows/linux/Kconfig.stable b/workflows/linux/Kconfig.stable
index 39c1984a..674cf890 100644
--- a/workflows/linux/Kconfig.stable
+++ b/workflows/linux/Kconfig.stable
@@ -13,56 +13,7 @@ endchoice
 
 if BOOTLINUX_TREE_STABLE
 
-choice
-	prompt "Stable branch to use"
-	default BOOTLINUX_STABLE_V66
-
-config BOOTLINUX_STABLE_V419
-	bool "linux-4.19.y"
-	help
-	  Automatically enable the latest supported linux-4.19.y kernel.
-
-config BOOTLINUX_STABLE_V54
-	bool "linux-5.4.y"
-	help
-	  Automatically enable the latest supported linux-5.4.y kernel.
-
-config BOOTLINUX_STABLE_V510
-	bool "linux-5.10.y"
-	help
-	  Automatically enable the latest supported linux-5.10.y kernel.
-
-config BOOTLINUX_STABLE_V514
-	bool "linux-5.14.y"
-	help
-	  Automatically enable the latest supported linux-5.14.y kernel.
-
-config BOOTLINUX_STABLE_V517
-	bool "linux-5.17.y"
-	help
-	  Automatically enable the latest supported linux-5.17.y kernel.
-
-config BOOTLINUX_STABLE_V519
-	bool "linux-5.19.y"
-	help
-	  Automatically enable the latest supported linux-5.19.y kernel.
-
-config BOOTLINUX_STABLE_V60
-	bool "linux-6.0.y"
-	help
-	  Automatically enable the latest supported linux-6.0.y kernel.
-
-config BOOTLINUX_STABLE_V61
-	bool "linux-6.1.y"
-	help
-	  Automatically enable the latest supported linux-6.1.y kernel.
-
-config BOOTLINUX_STABLE_V66
-	bool "linux-6.6.y"
-	help
-	  Automatically enable the latest supported linux-6.6.y kernel.
-
-endchoice
+source "workflows/linux/Kconfig.refs-stable"
 
 endif # BOOTLINUX_TREE_STABLE
 
diff --git a/workflows/linux/mcgrof-next-refs.yaml b/workflows/linux/mcgrof-next-refs.yaml
new file mode 100644
index 00000000..d430a0a9
--- /dev/null
+++ b/workflows/linux/mcgrof-next-refs.yaml
@@ -0,0 +1,10 @@
+---
+configs:
+  - master:
+    config: BOOTLINUX_MCGROF_NEXT_LBS
+    ref: "large-block-next"
+    help: "Use the latest stable LBS branch which enables all the bells and whistles for all types of LBS devices."
+  - main:
+    config: BOOTLINUX_MCGROF_NEXT_LBS_NOBDEV
+    ref: "large-block-nobdev"
+    help: "Use the latest minimal branch for LBS which let's you use LBS devices with a sectorsize compatibility"
diff --git a/workflows/linux/mcgrof-refs.yaml b/workflows/linux/mcgrof-refs.yaml
new file mode 100644
index 00000000..d343b17f
--- /dev/null
+++ b/workflows/linux/mcgrof-refs.yaml
@@ -0,0 +1,10 @@
+---
+configs:
+  - master:
+    config: BOOTLINUX_MCGROF_LINUS_LBS
+    ref: "large-block-linus"
+    help: "Use the latest LBS branch based on Linus' tree."
+  - main:
+    config: BOOTLINUX_MCGROF_LINUS_LBS_NOBDEV
+    ref: "large-block-linus-nobdev"
+    help: "Use the latest minimal branch for LBS which let's you use LBS devices with a sectorsize compatibility based on Linus' tree."
-- 
2.43.0

Re: [PATCH v2 1/1] linux: generate refs automatically

[Luis Chamberlain]

On Fri, Feb 16, 2024 at 09:18:28AM +0000, Daniel Gomez wrote:
> Generate a Kconfig choice/endchoice menu with a list of the latest git
> refs (tags and heads) for upstream repositories.
> 
> The script will create the Kconfig choice options to be consumed by
> their respective Kconfigs every 24h since last file update. This is to
> avoid delaying the makefile execution while expecting an updated Kconfig
> file with latest references.
> 
> Signed-off-by: Daniel Gomez <da.gomez@samsung.com>

This is the bee's knees! I'm excited for us to merge this ASAP. The only
ask I have is we add a yml file for each tree to match existing default
preferences, and allow somehow an option which allows us to lift this to
the "latest-and-greatest". If possible, I am not sure if it is, a new
kconfig option which which lets users enable it so that it uses the
latest and greatest instead ref tag instad of what is in the yml file.

This would allow us to keep existing baseline testers on a known
baseline, but allow us to then work on dynamic baselines which strive to
automation / regression testing to verify nothing breaks as we upgrade.

  Luis

Re: [PATCH v2 1/1] linux: generate refs automatically

[Daniel Gomez]

On Wed, Feb 21, 2024 at 10:12:07AM -0800, Luis Chamberlain wrote:
> On Fri, Feb 16, 2024 at 09:18:28AM +0000, Daniel Gomez wrote:
> > Generate a Kconfig choice/endchoice menu with a list of the latest git
> > refs (tags and heads) for upstream repositories.
> > 
> > The script will create the Kconfig choice options to be consumed by
> > their respective Kconfigs every 24h since last file update. This is to
> > avoid delaying the makefile execution while expecting an updated Kconfig
> > file with latest references.
> > 
> > Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
> 
> This is the bee's knees! I'm excited for us to merge this ASAP. The only
> ask I have is we add a yml file for each tree to match existing default

Ok. I will add all the static refs for all trees that won't be automatically
generated as I did with mcgrof-next refs.

> preferences, and allow somehow an option which allows us to lift this to
> the "latest-and-greatest". If possible, I am not sure if it is, a new
> kconfig option which which lets users enable it so that it uses the
> latest and greatest instead ref tag instad of what is in the yml file.

The script cannot generate the most recent date without first cloning the
repository to include that information. The workaround employed here involves
leveraging alphabetical reverse order with options [1] [2].

[1] git config: versionsort.suffix=-
[2] git-ls-remote option: --sort=-version:refname

If we want, latest-and-greatest, I think the strategy should be different.
Perhaps, an ansible task after git-clone the repo and sort by date. I can
explore that path too but it would be a different method.

> 
> This would allow us to keep existing baseline testers on a known
> baseline, but allow us to then work on dynamic baselines which strive to
> automation / regression testing to verify nothing breaks as we upgrade.
> 
>   Luis

Re: [PATCH v2 1/1] linux: generate refs automatically

[Luis Chamberlain]

+ Konstantin as he may have better ideas.

Konstantin, this patch set enables kdevops to scrape for ref tags
remotely so we can dynamically generate Kconfig files used for
kdevops for tags which users can select. The trees users can
have can be on kernel.org but elsewhere. The main trees obviously
are Linus, the stable tree, and linux-next. For these 3 it would be
good to get guidance on if the below strategy of using

git config: versionsort.suffix=-
git-ls-remote option: --sort=-version:refname

is best.

I see there's an atom feed, but my notes below on that.

On Thu, Feb 22, 2024 at 04:23:42PM +0000, Daniel Gomez wrote:
> On Wed, Feb 21, 2024 at 10:12:07AM -0800, Luis Chamberlain wrote:
> > On Fri, Feb 16, 2024 at 09:18:28AM +0000, Daniel Gomez wrote:
> > > Generate a Kconfig choice/endchoice menu with a list of the latest git
> > > refs (tags and heads) for upstream repositories.
> > > 
> > > The script will create the Kconfig choice options to be consumed by
> > > their respective Kconfigs every 24h since last file update. This is to
> > > avoid delaying the makefile execution while expecting an updated Kconfig
> > > file with latest references.
> > > 
> > > Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
> > 
> > This is the bee's knees! I'm excited for us to merge this ASAP. The only
> > ask I have is we add a yml file for each tree to match existing default
> 
> Ok. I will add all the static refs for all trees that won't be automatically
> generated as I did with mcgrof-next refs.
> 
> > preferences, and allow somehow an option which allows us to lift this to
> > the "latest-and-greatest". If possible, I am not sure if it is, a new
> > kconfig option which which lets users enable it so that it uses the
> > latest and greatest instead ref tag instad of what is in the yml file.
> 
> The script cannot generate the most recent date without first cloning the
> repository to include that information. The workaround employed here involves
> leveraging alphabetical reverse order with options [1] [2].
> 
> [1] git config: versionsort.suffix=-
> [2] git-ls-remote option: --sort=-version:refname

I see we have an atom feed for the kernel:

https://www.kernel.org/feeds/kdist.xml

But that does not seem as easy to parse for our purposes where we have a
source of 3 main kernels:

  * linus
  * stable
  * linux-next

For at least these 3 it would be great to use something simple which might
already be setup to allow less network traffic against kernel.org,
Konstantin, is the atom feed the most automated way to scape these
without having the kernel source git tree cloned?

I checked to see what we did for php-crystall-ball but that just relies
on you having git tree already present, and we just git fetch.

I *think* we should consider a few things then:

Not all users may want to use network traffic to augment Kconfig for
the latest tags, I think by default we want static settings, but it
doesn't mean we can't use tools to generate and update them for those
uses. So step one I think is to use your tools to automate the static files
and let us update them as developers of kdevops. The default tag used
per branch could be set based on the existing yaml file as you have
based on your tool too. We just don't always generate the files.

Step 2 I think would be an annotation which developers can use to hint,
hey, I want you to update my tags for me with the git config and
git-ls-remote. I think the easiest option is to piggy back on the
`make dynconfig` target to hint that we want to update the Kconfig
files before our run, and want to use these new ones instead of the
old static ones. It is not clear how we could easily avoid yet having
these not create a delta on `git diff`, perhaps using a variable on the
Kconfig include which changes when we use the `make dynconfig` target.

That should also let proprietary forks of kdevops take advantage of this
for customer kernels / tags.

> If we want, latest-and-greatest, I think the strategy should be different.

Sure, perhaps `make dynconfig` could first check if you have git trees
for your user in a specific set of paths, if they are there it can use
the git fetch -t, and git tag -l, otherwise fallback to the remote
option you currenlty use. Or rather, since you have the remote option
already implemnted use that first, and later if folks want to enhance
this they can add support for the check for local trees being present.

Curious if Konstantin has other suggestions.

  Luis

Re: [PATCH v2 1/1] linux: generate refs automatically

[Konstantin Ryabitsev]

On Thu, Feb 22, 2024 at 09:03:34AM -0800, Luis Chamberlain wrote:
> + Konstantin as he may have better ideas.
> 
> Konstantin, this patch set enables kdevops to scrape for ref tags
> remotely so we can dynamically generate Kconfig files used for
> kdevops for tags which users can select. The trees users can
> have can be on kernel.org but elsewhere. The main trees obviously
> are Linus, the stable tree, and linux-next. For these 3 it would be
> good to get guidance on if the below strategy of using
> 
> git config: versionsort.suffix=-
> git-ls-remote option: --sort=-version:refname
> 
> is best.

Why not just grab either of these two?

https://kernel.org/finger_banner
https://www.kernel.org/releases.json

-K

Re: [PATCH v2 1/1] linux: generate refs automatically

[Daniel Gomez]

On Thu, Feb 22, 2024 at 12:58:24PM -0500, Konstantin Ryabitsev wrote:
> On Thu, Feb 22, 2024 at 09:03:34AM -0800, Luis Chamberlain wrote:
> > + Konstantin as he may have better ideas.
> > 
> > Konstantin, this patch set enables kdevops to scrape for ref tags
> > remotely so we can dynamically generate Kconfig files used for
> > kdevops for tags which users can select. The trees users can
> > have can be on kernel.org but elsewhere. The main trees obviously
> > are Linus, the stable tree, and linux-next. For these 3 it would be
> > good to get guidance on if the below strategy of using
> > 
> > git config: versionsort.suffix=-
> > git-ls-remote option: --sort=-version:refname
> > 
> > is best.
> 
> Why not just grab either of these two?
> 
> https://kernel.org/finger_banner
> https://www.kernel.org/releases.json

This is great for latest-and-greatest. Thanks Konstantin.

Luis, I think we can integrate this json file as input for mainline, next and
stable trees and combine them with the output of git-ls-remote and yaml inputs
(if any).

> 
> -K