[PATCH 1/5] meta/conf/templates/default/conf-description.txt: add a template description

[PATCH 1/5] meta/conf/templates/default/conf-description.txt: add a template description

[Alexander Kanavin]

Existing conf-notes.txt is not adequate in situations where the user
needs to select a config template from several, or get an overview of them and
is interested only in a brief explanation of what the template is for,
but not extended multi-paragraph instructions for how to use it.

Such descriptions can be provided via conf-description.txt and
this patch adds such a description for the oe-core default template.

Updates to the tools and a tool for selecting templates which
will make specific use of the description will follow.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/conf/templates/default/conf-description.txt | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 meta/conf/templates/default/conf-description.txt

diff --git a/meta/conf/templates/default/conf-description.txt b/meta/conf/templates/default/conf-description.txt
new file mode 100644
index 00000000000..e49172fa0ce
--- /dev/null
+++ b/meta/conf/templates/default/conf-description.txt
@@ -0,0 +1 @@
+This is the default build configuration for the openembedded-core layer.
-- 
2.39.2

[PATCH 2/5] meta/lib/bblayers/buildconf.py: add support for configuration descriptions

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/lib/bblayers/buildconf.py           | 10 ++++++++--
 meta/lib/oeqa/selftest/cases/bblayers.py |  2 +-
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/meta/lib/bblayers/buildconf.py b/meta/lib/bblayers/buildconf.py
index 87a5e5a55bc..a7d06471df5 100644
--- a/meta/lib/bblayers/buildconf.py
+++ b/meta/lib/bblayers/buildconf.py
@@ -20,7 +20,10 @@ def plugin_init(plugins):
     return BuildConfPlugin()
 
 class BuildConfPlugin(LayerPlugin):
-    notes_fixme = """FIXME: Please place here the description of this build configuration.
+    notes_fixme = """FIXME: Please place here the detailed instructions for using this build configuration.
+They will be shown to the users when they set up their builds via TEMPLATECONF.
+"""
+    description_fixme = """FIXME: Please place here the short description of what this build configuration is for.
 It will be shown to the users when they set up their builds via TEMPLATECONF.
 """
 
@@ -41,14 +44,17 @@ It will be shown to the users when they set up their builds via TEMPLATECONF.
                     bblayers_data = bblayers_data.replace(abspath, "##OEROOT##/" + relpath)
                 dest.write(bblayers_data)
 
+        with open(os.path.join(destdir, "conf-description.txt"), 'w') as dest:
+            dest.write(self.description_fixme)
         with open(os.path.join(destdir, "conf-notes.txt"), 'w') as dest:
             dest.write(self.notes_fixme)
 
         logger.info("""Configuration template placed into {}
 Please review the files in there, and particularly provide a configuration description in {}
+and notes in {}
 You can try out the configuration with
 TEMPLATECONF={} . {}/oe-init-build-env build-try-{}"""
-.format(destdir, os.path.join(destdir, "conf-notes.txt"), destdir, oecorepath, templatename))
+.format(destdir, os.path.join(destdir, "conf-description.txt"), os.path.join(destdir, "conf-notes.txt"), destdir, oecorepath, templatename))
 
     def do_save_build_conf(self, args):
         """ Save the currently active build configuration (conf/local.conf, conf/bblayers.conf) as a template into a layer.\n This template can later be used for setting up builds via TEMPLATECONF. """
diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py
index 8faa0602348..819ddf51bcd 100644
--- a/meta/lib/oeqa/selftest/cases/bblayers.py
+++ b/meta/lib/oeqa/selftest/cases/bblayers.py
@@ -121,7 +121,7 @@ class BitbakeLayers(OESelftestTestCase):
         self.assertEqual(bb_vars['BBFILE_PRIORITY_%s' % layername], str(priority), 'BBFILE_PRIORITY_%s != %d' % (layername, priority))
 
         result = runCmd('bitbake-layers save-build-conf {} {}'.format(layerpath, "buildconf-1"))
-        for f in ('local.conf.sample', 'bblayers.conf.sample', 'conf-notes.txt'):
+        for f in ('local.conf.sample', 'bblayers.conf.sample', 'conf-description.txt', 'conf-notes.txt'):
             fullpath = os.path.join(layerpath, "conf", "templates", "buildconf-1", f)
             self.assertTrue(os.path.exists(fullpath), "Template configuration file {} not found".format(fullpath))
 
-- 
2.39.2

[PATCH 3/5] scripts/oe-setup-builddir: add support for configuration descriptions

[Alexander Kanavin]

They are handled exactly same as conf-notes.txt.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 scripts/oe-setup-builddir | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
index 678aeac4be5..1b5dc55743e 100755
--- a/scripts/oe-setup-builddir
+++ b/scripts/oe-setup-builddir
@@ -57,6 +57,7 @@ if [ -n "$TEMPLATECONF" ]; then
     fi
     OECORELAYERCONF="$TEMPLATECONF/bblayers.conf.sample"
     OECORELOCALCONF="$TEMPLATECONF/local.conf.sample"
+    OECOREDESCRCONF="$TEMPLATECONF/conf-description.txt"
     OECORENOTESCONF="$TEMPLATECONF/conf-notes.txt"
 fi
 
@@ -98,6 +99,13 @@ EOM
     SHOWYPDOC=yes
 fi
 
+if [ -z "$OECOREDESCRCONF" ]; then
+    OECOREDESCRCONF="$OEROOT/meta/conf/templates/default/conf-description.txt"
+fi
+if [ ! -r "$BUILDDIR/conf/conf-description.txt" ]; then
+    [ ! -r "$OECOREDESCRCONF" ] || cp "$OECOREDESCRCONF" "$BUILDDIR/conf/conf-description.txt"
+fi
+
 if [ -z "$OECORENOTESCONF" ]; then
     OECORENOTESCONF="$OEROOT/meta/conf/templates/default/conf-notes.txt"
 fi
@@ -108,6 +116,7 @@ fi
 # Prevent disturbing a new GIT clone in same console
 unset OECORELOCALCONF
 unset OECORELAYERCONF
+unset OECOREDESCRCONF
 unset OECORENOTESCONF
 
 # Ending the first-time run message. Show the YP Documentation banner.
@@ -124,6 +133,7 @@ EOM
 #    unset SHOWYPDOC
 fi
 
+[ ! -r "$BUILDDIR/conf/conf-description.txt" ] || cat "$BUILDDIR/conf/conf-description.txt"
 [ ! -r "$BUILDDIR/conf/conf-notes.txt" ] || cat "$BUILDDIR/conf/conf-notes.txt"
 
 if [ ! -f "$BUILDDIR/conf/templateconf.cfg" ]; then
-- 
2.39.2

[PATCH 4/5] scripts/oe-setup-layers: write a list of layer paths into the checkout's top dir

[Alexander Kanavin]

This is beneficial for setting up builds, as this list can be used
to determine reliably where the actual layers are, and discover
available configurations from them.

Also adjust the selftest to check the presence of that file rather
than any specific layer in a hardcoded location.

Sample output (paths are written relative to the file for relocatability
and ease of reading):

{
    "layers": [
        "meta-openembedded/meta-filesystems",
        "meta-openembedded/meta-gnome",
        "meta-openembedded/meta-initramfs",
        "meta-openembedded/meta-multimedia",
        "meta-openembedded/meta-networking",
        "meta-openembedded/meta-oe",
        "meta-openembedded/meta-perl",
        "meta-openembedded/meta-python",
        "meta-openembedded/meta-webserver",
        "meta-openembedded/meta-xfce",
        "poky/bitbake/lib/layerindexlib/tests/testdata/layer1",
        "poky/bitbake/lib/layerindexlib/tests/testdata/layer2",
        "poky/bitbake/lib/layerindexlib/tests/testdata/layer3",
        "poky/bitbake/lib/layerindexlib/tests/testdata/layer4",
        "poky/meta-poky",
        "poky/meta-selftest",
        "poky/meta-skeleton",
        "poky/meta-yocto-bsp",
        "poky/meta"
    ],
    "version": "1.0"
}

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/lib/oeqa/selftest/cases/bblayers.py |  6 ++----
 scripts/oe-setup-layers                  | 15 +++++++++++++++
 2 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py
index 819ddf51bcd..01fe17842e4 100644
--- a/meta/lib/oeqa/selftest/cases/bblayers.py
+++ b/meta/lib/oeqa/selftest/cases/bblayers.py
@@ -163,7 +163,5 @@ class BitbakeLayers(OESelftestTestCase):
 
         testcheckoutdir = os.path.join(self.builddir, 'test-layer-checkout')
         result = runCmd('{}/setup-layers --destdir {}'.format(self.testlayer_path, testcheckoutdir))
-        # May not necessarily be named 'poky' or 'openembedded-core'
-        oecoredir = os.listdir(testcheckoutdir)[0]
-        testcheckoutfile = os.path.join(testcheckoutdir, oecoredir, "oe-init-build-env")
-        self.assertTrue(os.path.exists(testcheckoutfile), "File {} not found in test layer checkout".format(testcheckoutfile))
+        layers_json = os.path.join(testcheckoutdir, ".oe-layers.json")
+        self.assertTrue(os.path.exists(layers_json), "File {} not found in test layer checkout".format(layers_json))
diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers
index 6d49688a32c..bee4ef0fec8 100755
--- a/scripts/oe-setup-layers
+++ b/scripts/oe-setup-layers
@@ -49,11 +49,24 @@ def _is_repo_at_remote_uri(repodir, remote, uri):
 def _contains_submodules(repodir):
     return os.path.exists(os.path.join(repodir,".gitmodules"))
 
+def _write_layer_list(dest, repodirs):
+    layers = []
+    for r in repodirs:
+        for root, dirs, files in os.walk(r):
+            if os.path.basename(root) == 'conf' and 'layer.conf' in files:
+                layers.append(os.path.relpath(os.path.dirname(root), dest))
+    layers_f = os.path.join(dest, ".oe-layers.json")
+    print("Writing list of layers into {}".format(layers_f))
+    with open(layers_f, 'w') as f:
+        json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, indent=4)
+
 def _do_checkout(args, json):
     repos = json['sources']
+    repodirs = []
     for r_name in repos:
         r_data = repos[r_name]
         repodir = os.path.abspath(os.path.join(args['destdir'], r_data['path']))
+        repodirs.append(repodir)
 
         if 'contains_this_file' in r_data.keys():
             force_arg = 'force_bootstraplayer_checkout'
@@ -96,6 +109,8 @@ def _do_checkout(args, json):
             if _contains_submodules(repodir):
                 print("Repo {} contains submodules, use 'git submodule update' to ensure they are up to date".format(repodir))
 
+    _write_layer_list(args['destdir'], repodirs)
+
 parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/")
 
 parser.add_argument('--force-bootstraplayer-checkout', action='store_true',
-- 
2.39.2

[PATCH 5/5] oe-setup-build: add a tool for discovering config templates and setting up builds

[Alexander Kanavin]

This is another piece of the puzzle in setting up builds from nothing
without having to write custom scripts or use external tools.

After layers have been fetched and placed into their respective locations by
oe-setup-layers, one would surely want to proceed to the actual build, and here's how:

1. Without arguments the tool reads available layers
from .oe-layers.json file (written out by oe-setup-layers or a fallback under scripts/),
prints what templates it has found, and asks the user to select one, as seen below.
This will land the user in a shell ready to run bitbake:

=============================================
alex@Zen2:/srv/work/alex$ ./setup-build
Available build configurations:

1. alex-configuration-gadget
This configuration will set up a build for the purposes of supporting gadget.

2. alex-configuration-gizmo
This configuration allows building a gizmo.

3. poky-default
This is the default build configuration for the Poky reference distribution.

Re-run with 'list -v' to see additional information.
Please choose a configuration by its number: 1
Running: TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/configuration-gadget . /srv/work/alex/poky/oe-init-build-env /srv/work/alex/build-alex-configuration-gadget && /bin/bash
You had no conf/local.conf file. This configuration file has therefore been
created for you from /srv/work/alex/meta-alex/conf/templates/configuration-gadget/local.conf.sample
You may wish to edit it to, for example, select a different MACHINE (target
hardware).

You had no conf/bblayers.conf file. This configuration file has therefore been
created for you from /srv/work/alex/meta-alex/conf/templates/configuration-gadget/bblayers.conf.sample
To add additional metadata layers into your configuration please add entries
to conf/bblayers.conf.

The Yocto Project has extensive documentation about OE including a reference
manual which can be found at:
    https://docs.yoctoproject.org

For more information about OpenEmbedded see the website:
    https://www.openembedded.org/

This configuration will set up a build for the purposes of supporting gadget.
Please refer to meta-alex/README for additional details and available bitbake targets.
==============================================

2. It is also possible to list available configurations without selecting one using
'setup-build list' or to select and setup one non-interactively with 'setup-build setup'.

3. The full set of command line options is:

$ ./setup-build --help
usage: setup-build [-h] [--layerlist LAYERLIST] {list,setup} ...

A script that discovers available build configurations and sets up a build environment based on one of them. Run without arguments to choose one interactively.

positional arguments:
  {list,setup}
    list                List available configurations
    setup               Set up a build environment and open a shell session with it, ready to run builds.

optional arguments:
  -h, --help            show this help message and exit
  --layerlist LAYERLIST
                        Where to look for available layers (as written out by setup-layers script) (default is /srv/work/alex/.oe-layers.json).

$ ./setup-build list --help
usage: setup-build list [-h] [-v]

optional arguments:
  -h, --help  show this help message and exit
  -v          Print detailed information and usage notes for each available build configuration.

$ ./setup-build setup --help
usage: setup-build setup [-h] [-c configuration_name] [-b build_path] [--no-shell]

optional arguments:
  -h, --help            show this help message and exit
  -c configuration_name
                        Use a build configuration configuration_name to set up a build environment (run this script with 'list' to see what is available)
  -b build_path         Set up a build directory in build_path (run this script with 'list -v' to see where it would be by default)
  --no-shell            Create a build directory but do not start a shell session with the build environment from it.

4. There's an an added hint in oe-setup-layers about how to proceed (as it is really not user-friendly
to fetch the layer repos successfully and then exit without a word), and a symlink to the script
from the top level layer checkout directory.

5. The selftest to check layer setup has been adjusted to run a basic check for template
discovery and build setup. The revision of poky to be cloned has been bumped to 4.1,
as that's the first version with a default template in a standard location.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/lib/oeqa/selftest/cases/bblayers.py |  26 ++++-
 scripts/.oe-layers.json                  |   7 ++
 scripts/oe-setup-build                   | 122 +++++++++++++++++++++++
 scripts/oe-setup-layers                  |  10 ++
 4 files changed, 163 insertions(+), 2 deletions(-)
 create mode 100644 scripts/.oe-layers.json
 create mode 100755 scripts/oe-setup-build

diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py
index 01fe17842e4..00b1e6ce96f 100644
--- a/meta/lib/oeqa/selftest/cases/bblayers.py
+++ b/meta/lib/oeqa/selftest/cases/bblayers.py
@@ -152,12 +152,12 @@ class BitbakeLayers(OESelftestTestCase):
         self.validate_layersjson(jsonfile)
 
         # The revision-under-test may not necessarily be available on the remote server,
-        # so replace it with a revision that has a yocto-4.0 tag.
+        # so replace it with a revision that has a yocto-4.1 tag.
         import json
         with open(jsonfile) as f:
             data = json.load(f)
         for s in data['sources']:
-            data['sources'][s]['git-remote']['rev'] = '00cfdde791a0176c134f31e5a09eff725e75b905'
+            data['sources'][s]['git-remote']['rev'] = '5200799866b92259e855051112520006e1aaaac0'
         with open(jsonfile, 'w') as f:
             json.dump(data, f)
 
@@ -165,3 +165,25 @@ class BitbakeLayers(OESelftestTestCase):
         result = runCmd('{}/setup-layers --destdir {}'.format(self.testlayer_path, testcheckoutdir))
         layers_json = os.path.join(testcheckoutdir, ".oe-layers.json")
         self.assertTrue(os.path.exists(layers_json), "File {} not found in test layer checkout".format(layers_json))
+
+        # As setup-layers checkout out an old revision of poky, there is no setup-build symlink,
+        # and we need to run oe-setup-build directly from the current poky tree under test
+        oe_setup_build = os.path.join(get_bb_var('COREBASE'), 'scripts/oe-setup-build')
+        oe_setup_build_l = os.path.join(testcheckoutdir, 'setup-build')
+        os.symlink(oe_setup_build,oe_setup_build_l)
+
+        cmd = '{} --layerlist {} list -v'.format(oe_setup_build_l, layers_json)
+        result = runCmd(cmd)
+        cond = "conf/templates/default" in result.output
+        self.assertTrue(cond, "Incorrect output from {}: {}".format(cmd, result.output))
+
+        # rather than hardcode the build setup cmdline here, let's actually run what the tool suggests to the user
+        conf = None
+        if 'poky-default' in result.output:
+            conf = 'poky-default'
+        elif 'meta-default' in result.output:
+            conf = 'meta-default'
+        self.assertIsNotNone(conf, "Could not find the configuration to set up a build in the output: {}".format(result.output))
+
+        cmd = '{} --layerlist {} setup -c {} --no-shell'.format(oe_setup_build_l, layers_json, conf)
+        result = runCmd(cmd)
diff --git a/scripts/.oe-layers.json b/scripts/.oe-layers.json
new file mode 100644
index 00000000000..1b00a84b542
--- /dev/null
+++ b/scripts/.oe-layers.json
@@ -0,0 +1,7 @@
+{
+    "layers": [
+        "../meta-poky",
+        "../meta"
+    ],
+    "version": "1.0"
+}
diff --git a/scripts/oe-setup-build b/scripts/oe-setup-build
new file mode 100755
index 00000000000..074f0b4c13e
--- /dev/null
+++ b/scripts/oe-setup-build
@@ -0,0 +1,122 @@
+#!/usr/bin/env python3
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import argparse
+import json
+import os
+import subprocess
+
+def defaultlayers():
+    return os.path.abspath(os.path.join(os.path.dirname(__file__), '.oe-layers.json'))
+
+def makebuildpath(topdir, template):
+    return os.path.join(topdir, "build-{}".format(template))
+
+def discover_templates(layers_file):
+    if not os.path.exists(layers_file):
+        print("List of layers {} does not exist; were the layers set up using the setup-layers script?".format(layers_file))
+        return None
+
+    templates = []
+    layers_list = json.load(open(layers_file))["layers"]
+    for layer in layers_list:
+        template_dir = os.path.join(os.path.dirname(layers_file), layer, 'conf','templates')
+        if os.path.exists(template_dir):
+            for d in sorted(os.listdir(template_dir)):
+                templatepath = os.path.join(template_dir,d)
+                if not os.path.isfile(os.path.join(templatepath,'local.conf.sample')):
+                    continue
+                layer_base = os.path.basename(layer)
+                templatename = "{}-{}".format(layer_base[5:] if layer_base.startswith("meta-") else layer_base, d)
+                buildpath = makebuildpath(os.getcwd(), templatename)
+                notespath = os.path.join(template_dir, d, 'conf-notes.txt')
+                try: notes = open(notespath).read()
+                except: notes = None
+                try: description = open(os.path.join(template_dir, d, 'conf-description.txt')).read()
+                except: description = None
+                templates.append({"templatename":templatename,"templatepath":templatepath,"buildpath":buildpath,"notespath":notespath,"notes":notes,"description":description})
+
+    return templates
+
+def print_templates(templates, verbose):
+    print("Available build configurations:\n")
+
+    for i in range(len(templates)):
+        t = templates[i]
+        print("{}. {}".format(i+1, t["templatename"]))
+        print("{}".format(t["description"].strip() if t["description"] else "This configuration does not have a description."))
+        if verbose:
+            print("Configuration template path:", t["templatepath"])
+            print("Build path:", t["buildpath"])
+            print("Usage notes:", t["notespath"] if t["notes"] else "This configuration does not have usage notes.")
+        print("")
+    if not verbose:
+        print("Re-run with 'list -v' to see additional information.")
+
+def list_templates(args):
+    templates = discover_templates(args.layerlist)
+    if not templates:
+        return
+
+    verbose = args.v
+    print_templates(templates, verbose)
+
+def find_template(template_name, templates):
+    print_templates(templates, False)
+    if not template_name:
+        n_s = input("Please choose a configuration by its number: ")
+        try: return templates[int(n_s) - 1]
+        except:
+            print("Invalid selection, please try again.")
+            return None
+    else:
+        for t in templates:
+            if t["templatename"] == template_name:
+                return t
+        print("Configuration {} is not one of {}, please try again.".format(tempalte_name, [t["templatename"] for t in templates]))
+        return None
+
+def setup_build_env(args):
+    templates = discover_templates(args.layerlist)
+    if not templates:
+        return
+
+    template = find_template(args.c, templates)
+    if not template:
+        return
+    builddir = args.b if args.b else template["buildpath"]
+    no_shell = args.no_shell
+    coredir = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), '..'))
+    cmd = "TEMPLATECONF={} . {} {}".format(template["templatepath"], os.path.join(coredir, 'oe-init-build-env'), builddir)
+    if not no_shell:
+        cmd = cmd + " && {}".format(os.environ['SHELL'])
+    print("Running:", cmd)
+    subprocess.run(cmd, shell=True, executable=os.environ['SHELL'])
+
+parser = argparse.ArgumentParser(description="A script that discovers available build configurations and sets up a build environment based on one of them. Run without arguments to choose one interactively.")
+parser.add_argument("--layerlist", default=defaultlayers(), help='Where to look for available layers (as written out by setup-layers script) (default is {}).'.format(defaultlayers()))
+
+subparsers = parser.add_subparsers()
+parser_list_templates = subparsers.add_parser('list', help='List available configurations')
+parser_list_templates.add_argument('-v', action='store_true',
+        help='Print detailed information and usage notes for each available build configuration.')
+parser_list_templates.set_defaults(func=list_templates)
+
+parser_setup_env = subparsers.add_parser('setup', help='Set up a build environment and open a shell session with it, ready to run builds.')
+parser_setup_env.add_argument('-c', metavar='configuration_name', help="Use a build configuration configuration_name to set up a build environment (run this script with 'list' to see what is available)")
+parser_setup_env.add_argument('-b', metavar='build_path', help="Set up a build directory in build_path (run this script with 'list -v' to see where it would be by default)")
+parser_setup_env.add_argument('--no-shell', action='store_true',
+        help='Create a build directory but do not start a shell session with the build environment from it.')
+parser_setup_env.set_defaults(func=setup_build_env)
+
+args = parser.parse_args()
+
+if 'func' in args:
+    args.func(args)
+else:
+    from argparse import Namespace
+    setup_build_env(Namespace(layerlist=args.layerlist, c=None, b=None, no_shell=False))
diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers
index bee4ef0fec8..6fbfefd656f 100755
--- a/scripts/oe-setup-layers
+++ b/scripts/oe-setup-layers
@@ -63,6 +63,7 @@ def _write_layer_list(dest, repodirs):
 def _do_checkout(args, json):
     repos = json['sources']
     repodirs = []
+    oesetupbuild = None
     for r_name in repos:
         r_data = repos[r_name]
         repodir = os.path.abspath(os.path.join(args['destdir'], r_data['path']))
@@ -108,9 +109,18 @@ def _do_checkout(args, json):
 
             if _contains_submodules(repodir):
                 print("Repo {} contains submodules, use 'git submodule update' to ensure they are up to date".format(repodir))
+        if os.path.exists(os.path.join(repodir, 'scripts/oe-setup-build')):
+            oesetupbuild = os.path.join(repodir, 'scripts/oe-setup-build')
 
     _write_layer_list(args['destdir'], repodirs)
 
+    if oesetupbuild:
+        oesetupbuild_symlink = os.path.join(args['destdir'], 'setup-build')
+        if os.path.exists(oesetupbuild_symlink):
+            os.remove(oesetupbuild_symlink)
+        os.symlink(os.path.relpath(oesetupbuild,args['destdir']),oesetupbuild_symlink)
+        print("\nRun '{}' to list available build configuration templates and set up a build from one of them.".format(oesetupbuild_symlink))
+
 parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/")
 
 parser.add_argument('--force-bootstraplayer-checkout', action='store_true',
-- 
2.39.2

Re: [PATCH 5/5] oe-setup-build: add a tool for discovering config templates and setting up builds

[Jermain Horsman]

I just tried to run the selftest on bblayers using oe-core and not poky and I noticed some failures,
which I thought I'd share.

>          # The revision-under-test may not necessarily be available on the
> remote server,
> -        # so replace it with a revision that has a yocto-4.0 tag.
> +        # so replace it with a revision that has a yocto-4.1 tag.
>          import json
>          with open(jsonfile) as f:
>              data = json.load(f)
>          for s in data['sources']:
> -            data['sources'][s]['git-remote']['rev'] =
> '00cfdde791a0176c134f31e5a09eff725e75b905'
> +            data['sources'][s]['git-remote']['rev'] =
> '5200799866b92259e855051112520006e1aaaac0'

First I got a failure here, as this is not a valid reference on oe-core, only on poky is this valid.

> +def setup_build_env(args):
> +    templates = discover_templates(args.layerlist)
> +    if not templates:
> +        return
> +
> +    template = find_template(args.c, templates)
> +    if not template:
> +        return
> +    builddir = args.b if args.b else template["buildpath"]
> +    no_shell = args.no_shell
> +    coredir =
> os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)),
> '..'))
> +    cmd = "TEMPLATECONF={} . {} {}".format(template["templatepath"],
> os.path.join(coredir, 'oe-init-build-env'), builddir)
> +    if not no_shell:
> +        cmd = cmd + " && {}".format(os.environ['SHELL'])
> +    print("Running:", cmd)
> +    subprocess.run(cmd, shell=True, executable=os.environ['SHELL'])
> +

2024-02-08 13:14:26,793 - oe-selftest - INFO - ======================================================================
2024-02-08 13:14:26,793 - oe-selftest - INFO - FAIL: test_bitbakelayers_setup (bblayers.BitbakeLayers)
2024-02-08 13:14:26,794 - oe-selftest - INFO - ----------------------------------------------------------------------
2024-02-08 13:14:26,794 - oe-selftest - INFO - Traceback (most recent call last):
  File "/home/builder/workdir/openembedded-core-contrib/meta/lib/oeqa/selftest/cases/bblayers.py", line 189, in test_bitbakelayers_setup
    result = runCmd(cmd)
  File "/home/builder/workdir/openembedded-core-contrib/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command '/home/builder/workdir/build-st/test-layer-checkout/setup-build --layerlist /home/builder/workdir/build-st/test-layer-checkout/.oe-layers.json setup -c meta-default --no-shell' returned non-zero exit status 1:
Available build configurations:

1. meta-default
This configuration does not have a description.

Re-run with 'list -v' to see additional information.
Running: TEMPLATECONF=/home/builder/workdir/build-st/test-layer-checkout/openembedded-core-contrib/meta/conf/templates/default . /home/builder/workdir/openembedded-core-contrib/oe-init-build-env /home/builder/workdir/build-st/build-meta-default
Traceback (most recent call last):
  File "/home/builder/workdir/build-st/test-layer-checkout/setup-build", line 119, in <module>
    args.func(args)
  File "/home/builder/workdir/build-st/test-layer-checkout/setup-build", line 98, in setup_build_env
    subprocess.run(cmd, shell=True, executable=os.environ['SHELL'])
  File "/usr/lib/python3.9/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'SHELL'


I'm not entirely sure what was going on here, the SHELL variable was set in the shell environment, however it was not in os.environ.

Re: [OE-core] [PATCH 5/5] oe-setup-build: add a tool for discovering config templates and setting up builds

[Alexander Kanavin]

On Thu, 8 Feb 2024 at 14:40, Jermain Horsman <jermain.horsman@nedap.com> wrote:
> >          # The revision-under-test may not necessarily be available on the
> > remote server,
> > -        # so replace it with a revision that has a yocto-4.0 tag.
> > +        # so replace it with a revision that has a yocto-4.1 tag.
> >          import json
> >          with open(jsonfile) as f:
> >              data = json.load(f)
> >          for s in data['sources']:
> > -            data['sources'][s]['git-remote']['rev'] =
> > '00cfdde791a0176c134f31e5a09eff725e75b905'
> > +            data['sources'][s]['git-remote']['rev'] =
> > '5200799866b92259e855051112520006e1aaaac0'
>
> First I got a failure here, as this is not a valid reference on oe-core, only on poky is this valid.

Yes, the test is poky-specific and was before the change too.
Something to address later on.


> 1. meta-default
> This configuration does not have a description.
>
> Re-run with 'list -v' to see additional information.
> Running: TEMPLATECONF=/home/builder/workdir/build-st/test-layer-checkout/openembedded-core-contrib/meta/conf/templates/default . /home/builder/workdir/openembedded-core-contrib/oe-init-build-env /home/builder/workdir/build-st/build-meta-default
> Traceback (most recent call last):
>   File "/home/builder/workdir/build-st/test-layer-checkout/setup-build", line 119, in <module>
>     args.func(args)
>   File "/home/builder/workdir/build-st/test-layer-checkout/setup-build", line 98, in setup_build_env
>     subprocess.run(cmd, shell=True, executable=os.environ['SHELL'])
>   File "/usr/lib/python3.9/os.py", line 679, in __getitem__
>     raise KeyError(key) from None
> KeyError: 'SHELL'
>
>
> I'm not entirely sure what was going on here, the SHELL variable was set in the shell environment, however it was not in os.environ.

I ran this on two different computers now to make sure something
didn't regress, and it passed on both. Can you try with a plain poky
checkout?

Alex

RE: [OE-core] [PATCH 5/5] oe-setup-build: add a tool for discovering config templates and setting up builds

[Jermain Horsman]

> Yes, the test is poky-specific and was before the change too.
> Something to address later on.

Alright

> I ran this on two different computers now to make sure something
> didn't regress, and it passed on both. Can you try with a plain poky
> checkout?

So I did a couple of runs, mostly failures until I changed my default SHELL
Variable to '/bin/bash', i.e.:

oe-selftest - FAIL

builder@buildhost:~$ echo $SHELL
/bin/sh
builder@buildhost:~$ export SHELL=/bin/bash
builder@buildhost:~$ echo $SHELL
/bin/bash
builder@buildhost:~$ cd workdir
builder@buildhost:~/workdir$ . poky/oe-init-build-env

oe-selftest - OK

I've never had any issues with this before, and I'm not aware of bash being required,
But I might be missing something, anyway hope this helps.

Sincerely,

Jermain Horsman

Re: [OE-core] [PATCH 1/5] meta/conf/templates/default/conf-description.txt: add a template description

[Richard Purdie]

On Wed, 2024-01-31 at 13:39 +0100, Alexander Kanavin wrote:
> Existing conf-notes.txt is not adequate in situations where the user
> needs to select a config template from several, or get an overview of them and
> is interested only in a brief explanation of what the template is for,
> but not extended multi-paragraph instructions for how to use it.
> 
> Such descriptions can be provided via conf-description.txt and
> this patch adds such a description for the oe-core default template.
> 
> Updates to the tools and a tool for selecting templates which
> will make specific use of the description will follow.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  meta/conf/templates/default/conf-description.txt | 1 +
>  1 file changed, 1 insertion(+)
>  create mode 100644 meta/conf/templates/default/conf-description.txt
> 
> diff --git a/meta/conf/templates/default/conf-description.txt b/meta/conf/templates/default/conf-description.txt
> new file mode 100644
> index 00000000000..e49172fa0ce
> --- /dev/null
> +++ b/meta/conf/templates/default/conf-description.txt
> @@ -0,0 +1 @@
> +This is the default build configuration for the openembedded-core layer.

I know this feedback is long delayed so my apologies for that.

I think I'm ok with the idea of this but I believe the naming isn't
quite right.

"notes" does suggest something longer, so that is fine. My worry is
that in the metadata we have the SUMMARY and DESCRIPTION variables. The
former is the short one, the latter is the longer.

If developers see:

conf-notes.txt
conf-description.txt

then they don't know which is the "short" one.

If you however you have:

conf-notes.txt
conf-summary.txt

then I think that is much clearer?

I think I'm therefore ok with the idea but I'd like to suggest using
"summary" instead of "description"...

Cheers

Richard

Re: [OE-core] [PATCH 1/5] meta/conf/templates/default/conf-description.txt: add a template description

[Alexander Kanavin]

On Thu, 15 Feb 2024 at 16:00, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:

> I know this feedback is long delayed so my apologies for that.
>
> I think I'm ok with the idea of this but I believe the naming isn't
> quite right.
>
> "notes" does suggest something longer, so that is fine. My worry is
> that in the metadata we have the SUMMARY and DESCRIPTION variables. The
> former is the short one, the latter is the longer.
>
> If developers see:
>
> conf-notes.txt
> conf-description.txt
>
> then they don't know which is the "short" one.
>
> If you however you have:
>
> conf-notes.txt
> conf-summary.txt
>
> then I think that is much clearer?
>
> I think I'm therefore ok with the idea but I'd like to suggest using
> "summary" instead of "description"...

No problem. Peter suggested this previously as well, but I didn't
think it's super important back then.

Alex