* [PATCH 2/7] bitbake-layers: add a command to save the active build configuration as a template into a layer
2022-08-31 11:13 [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid Alexander Kanavin
@ 2022-08-31 11:13 ` Alexander Kanavin
2022-08-31 11:13 ` [PATCH 3/7] meta/files: add layer setup JSON schema and example Alexander Kanavin
` (5 subsequent siblings)
6 siblings, 0 replies; 21+ messages in thread
From: Alexander Kanavin @ 2022-08-31 11:13 UTC (permalink / raw)
To: openembedded-core; +Cc: Alexander Kanavin
This is the reverse of setting up a build by pointing TEMPLATECONF to a directory
with a template and running '. oe-init-build-env': this takes the config files from build/conf,
replaces site-specific paths in bblayers.conf with ##OECORE##-relative paths, and copies
the config files into a specified layer under a specified template name.
In many or perhaps most cases such static prefabricated configurations (that require no
further editing) are just enough, and I believe they should be offered by the
official configuration management. On the other hand, generating build configurations with a
sufficiently versatile tool is a far more complex problem, and one we should try to tackle
once we see where and how static configs fall short.
Tooling to discover and select these templates when setting up a build will be provided later on.
How to use:
alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers save-build-conf ../../meta-alex/ test-1
NOTE: Starting bitbake server...
NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
You can try out the configuration with
TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
alex@Zen2:/srv/work/alex/poky/build-layersetup$
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
meta/lib/bblayers/buildconf.py | 85 ++++++++++++++++++++++++
meta/lib/oeqa/selftest/cases/bblayers.py | 5 ++
2 files changed, 90 insertions(+)
create mode 100644 meta/lib/bblayers/buildconf.py
diff --git a/meta/lib/bblayers/buildconf.py b/meta/lib/bblayers/buildconf.py
new file mode 100644
index 0000000000..e07fc534e1
--- /dev/null
+++ b/meta/lib/bblayers/buildconf.py
@@ -0,0 +1,85 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import logging
+import os
+import stat
+import sys
+import shutil
+import json
+
+import bb.utils
+import bb.process
+
+from bblayers.common import LayerPlugin
+
+logger = logging.getLogger('bitbake-layers')
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
+
+import oe.buildcfg
+
+def plugin_init(plugins):
+ return BuildConfPlugin()
+
+class BuildConfPlugin(LayerPlugin):
+ notes_fixme = """FIXME: Please place here the description of this build configuration.
+It will be shown to the users when they set up their builds via TEMPLATECONF.
+"""
+
+ def _save_conf(self, templatename, templatepath, oecorepath, relpaths_to_oecore):
+ confdir = os.path.join(os.environ["BBPATH"], "conf")
+ destdir = os.path.join(templatepath, "conf", "templates", templatename)
+ os.makedirs(destdir, exist_ok=True)
+
+ with open(os.path.join(confdir, "local.conf")) as src:
+ with open(os.path.join(destdir, "local.conf.sample"), 'w') as dest:
+ dest.write(src.read())
+
+ with open(os.path.join(confdir, "bblayers.conf")) as src:
+ with open(os.path.join(destdir, "bblayers.conf.sample"), 'w') as dest:
+ bblayers_data = src.read()
+
+ for (abspath, relpath) in relpaths_to_oecore:
+ bblayers_data = bblayers_data.replace(abspath, "##OEROOT##/" + relpath)
+ dest.write(bblayers_data)
+
+ with open(os.path.join(destdir, "conf-notes.txt"), 'w') as dest:
+ dest.write(self.notes_fixme)
+
+ logger.info("""Configuration template placed into {}
+Please review the files in there, and particularly provide a configuration description in {}
+You can try out the configuration with
+TEMPLATECONF={} . {}/oe-init-build-env build-try-{}"""
+.format(destdir, os.path.join(destdir, "conf-notes.txt"), destdir, oecorepath, templatename))
+
+ def do_save_build_conf(self, args):
+ """ Save the currently active build configuration (conf/local.conf, conf/bblayers.conf) as a template into a layer.\n This template can later be used for setting up builds via TEMPLATECONF. """
+ repos = {}
+ layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data)
+ targetlayer = None
+ oecore = None
+
+ for l in layers:
+ if l[0] == os.path.abspath(args.layerpath):
+ targetlayer = l[0]
+ if l[1] == 'meta':
+ oecore = os.path.dirname(l[0])
+
+ if not targetlayer:
+ logger.error("Layer {} not in one of the currently enabled layers:\n{}".format(args.layerpath, "\n".join([l[0] for l in layers])))
+ elif not oecore:
+ logger.error("Openembedded-core not in one of the currently enabled layers:\n{}".format("\n".join([l[0] for l in layers])))
+ else:
+ relpaths_to_oecore = [(l[0], os.path.relpath(l[0], start=oecore)) for l in layers]
+ self._save_conf(args.templatename, targetlayer, oecore, relpaths_to_oecore)
+
+ def register_commands(self, sp):
+ parser_build_conf = self.add_command(sp, 'save-build-conf', self.do_save_build_conf, parserecipes=False)
+ parser_build_conf.add_argument('layerpath',
+ help='The path to the layer where the configuration template should be saved.')
+ parser_build_conf.add_argument('templatename',
+ help='The name of the configuration template.')
diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py
index 494fa892a3..549abe7d10 100644
--- a/meta/lib/oeqa/selftest/cases/bblayers.py
+++ b/meta/lib/oeqa/selftest/cases/bblayers.py
@@ -113,6 +113,11 @@ class BitbakeLayers(OESelftestTestCase):
self.assertEqual(bb_vars['BBFILE_PRIORITY_%s' % layername], str(priority), 'BBFILE_PRIORITY_%s != %d' % (layername, priority))
+ result = runCmd('bitbake-layers save-build-conf {} {}'.format(layerpath, "buildconf-1"))
+ for f in ('local.conf.sample', 'bblayers.conf.sample', 'conf-notes.txt'):
+ fullpath = os.path.join(layerpath, "conf", "templates", "buildconf-1", f)
+ self.assertTrue(os.path.exists(fullpath), "Template configuration file {} not found".format(fullpath))
+
def get_recipe_basename(self, recipe):
recipe_file = ""
result = runCmd("bitbake-layers show-recipes -f %s" % recipe)
--
2.30.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 3/7] meta/files: add layer setup JSON schema and example
2022-08-31 11:13 [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid Alexander Kanavin
2022-08-31 11:13 ` [PATCH 2/7] bitbake-layers: add a command to save the active build configuration as a template into a layer Alexander Kanavin
@ 2022-08-31 11:13 ` Alexander Kanavin
2022-08-31 11:13 ` [PATCH 4/7] bitbake-layers: add ability to save current layer repository configuration into a file Alexander Kanavin
` (4 subsequent siblings)
6 siblings, 0 replies; 21+ messages in thread
From: Alexander Kanavin @ 2022-08-31 11:13 UTC (permalink / raw)
To: openembedded-core; +Cc: Joshua Watt, Alexander Kanavin
From: Joshua Watt <JPEWhacker@gmail.com>
Defines a common schema for layer setup that can be consumed by tools to
know how to fetch and assemble layers for end users. Also includes an
example of the layer setup that constructs poky/meta-intel/imaginary product layer
for reference.
The schema can be used to validate a layer setup file with the commands:
$ python3 -m pip install jsonschema
$ jsonschema -i meta/files/layers.example.json meta/files/layers.schema.json
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Alex: I made the following modifications to Joshua's original commit:
- moved the files from meta/lib to meta/files
- the example json showcases a multi-repo, multi-layer setup
instead of just poky - closer to a typical product
- added oe-selftest that validates the example json against the schema using python3-jsonschema-native
- the schema is modified so that:
-- all lists (sources, layers, remotes) are replaced by objects keyed by 'name' properties of the list items.
This allows using them as dicts inside Python, and makes the json more compact and readable.
-- added 'contains_this_file' property to source object
-- replaced 'remote' property with a 'oneOf' definition for git with a specific
'git-remote' property. 'oneOf' is problematic when schema validation fails:
the diagnostic is only that none of oneOf variants matched, which is too non-specific.
-- added 'describe' property to 'git-remote' object.
-- removed description property for a layer source: it is not clear how to add that
when auto-generating the json
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
meta/files/layers.example.json | 72 +++++++++++++++++++
meta/files/layers.schema.json | 91 ++++++++++++++++++++++++
meta/lib/oeqa/selftest/cases/bblayers.py | 16 ++++-
3 files changed, 178 insertions(+), 1 deletion(-)
create mode 100644 meta/files/layers.example.json
create mode 100644 meta/files/layers.schema.json
diff --git a/meta/files/layers.example.json b/meta/files/layers.example.json
new file mode 100644
index 0000000000..3772404ec9
--- /dev/null
+++ b/meta/files/layers.example.json
@@ -0,0 +1,72 @@
+{
+ "sources": {
+ "meta-alex": {
+ "contains_this_file": true,
+ "git-remote": {
+ "branch": "master",
+ "describe": "",
+ "remotes": {
+ "remote-alex": {
+ "uri": "https://github.com/kanavin/meta-alex"
+ }
+ },
+ "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5"
+ },
+ "layers": {
+ "meta-alex": {
+ "subpath": ""
+ }
+ },
+ "path": "meta-alex"
+ },
+ "meta-intel": {
+ "git-remote": {
+ "branch": "master",
+ "describe": "15.0-hardknott-3.3-310-g0a96edae",
+ "remotes": {
+ "origin": {
+ "uri": "git://git.yoctoproject.org/meta-intel"
+ }
+ },
+ "rev": "0a96edae609a3f48befac36af82cf1eed6786b4a"
+ },
+ "layers": {
+ "meta-intel": {
+ "subpath": ""
+ }
+ },
+ "path": "meta-intel"
+ },
+ "poky": {
+ "git-remote": {
+ "branch": "akanavin/setup-layers",
+ "describe": "4.1_M1-374-g9dda719b2a",
+ "remotes": {
+ "origin": {
+ "uri": "git://git.yoctoproject.org/poky"
+ },
+ "poky-contrib": {
+ "uri": "ssh://git@push.yoctoproject.org/poky-contrib"
+ }
+ },
+ "rev": "9dda719b2a4727a4d43a6ab8d9e23f8ca68790ec"
+ },
+ "layers": {
+ "meta": {
+ "subpath": "meta"
+ },
+ "meta-poky": {
+ "subpath": "meta-poky"
+ },
+ "meta-selftest": {
+ "subpath": "meta-selftest"
+ },
+ "meta-yocto-bsp": {
+ "subpath": "meta-yocto-bsp"
+ }
+ },
+ "path": "poky"
+ }
+ },
+ "version": "1.0"
+}
diff --git a/meta/files/layers.schema.json b/meta/files/layers.schema.json
new file mode 100644
index 0000000000..cd4ddd3dcd
--- /dev/null
+++ b/meta/files/layers.schema.json
@@ -0,0 +1,91 @@
+{
+ "description": "OpenEmbedder Layer Setup Manifest",
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "version"
+ ],
+ "properties": {
+ "version": {
+ "description": "The version of this document; currently '1.0'",
+ "enum": ["1.0"]
+ },
+ "sources": {
+ "description": "The dict of layer sources",
+ "type": "object",
+ "patternProperties": { ".*" : {
+ "type": "object",
+ "description": "The upstream source from which a set of layers may be fetched",
+ "additionalProperties": false,
+ "required": [
+ "path"
+ ],
+ "properties": {
+ "path": {
+ "description": "The path where this layer source will be placed when fetching",
+ "type": "string"
+ },
+ "contains_this_file": {
+ "description": "Whether the directory with the layer source also contains this json description. Tools may want to skip the checkout of the source then.",
+ "type": "boolean"
+ },
+ "layers": {
+ "description": "The dict of layers to be used from this upstream source",
+ "type": "object",
+ "patternProperties": { ".*" : {
+ "description": "A layer from the upstream source",
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "subpath": {
+ "description": "The subpath (relative to the source root) for this layer. Omit if the source root is the layer path",
+ "type": "string"
+ }
+ }
+ }}
+ },
+ "git-remote": {
+ "description": "A remote git source from which to fetch",
+ "type": "object",
+ "additionalProperties": false,
+ "required": [
+ "rev"
+ ],
+ "properties": {
+ "branch": {
+ "description": "The git branch to fetch (optional)",
+ "type": "string"
+ },
+ "rev": {
+ "description": "The git revision to checkout",
+ "type": "string"
+ },
+ "describe": {
+ "description": "The output of 'git describe' (human readable description of the revision using tags in revision history).",
+ "type": "string"
+ },
+ "remotes": {
+ "description": "The dict of git remotes to add to this repository",
+ "type": "object",
+ "patternProperties": { ".*" : {
+ "description": "A git remote",
+ "type": "object",
+ "addtionalProperties": false,
+ "required": [
+ "uri"
+ ],
+ "properties": {
+ "uri": {
+ "description": "The URI for the remote",
+ "type": "string"
+ }
+ }
+ }}
+ }
+ }
+ }
+ }
+ }
+ }}
+ }
+}
diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py
index 549abe7d10..c753a7b795 100644
--- a/meta/lib/oeqa/selftest/cases/bblayers.py
+++ b/meta/lib/oeqa/selftest/cases/bblayers.py
@@ -8,12 +8,16 @@ import os
import re
import oeqa.utils.ftools as ftools
-from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars
+from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars, bitbake
from oeqa.selftest.case import OESelftestTestCase
class BitbakeLayers(OESelftestTestCase):
+ def setUpLocal(self):
+ bitbake("python3-jsonschema-native")
+ bitbake("-c addto_recipe_sysroot python3-jsonschema-native")
+
def test_bitbakelayers_layerindexshowdepends(self):
result = runCmd('bitbake-layers layerindex-show-depends meta-poky')
find_in_contents = re.search("openembedded-core", result.output)
@@ -128,3 +132,13 @@ class BitbakeLayers(OESelftestTestCase):
self.assertTrue(os.path.isfile(recipe_file), msg = "Can't find recipe file for %s" % recipe)
return os.path.basename(recipe_file)
+
+ def validate_layersjson(self, json):
+ python = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-jsonschema-native'), 'nativepython3')
+ jsonvalidator = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-jsonschema-native'), 'jsonschema')
+ jsonschema = os.path.join(get_bb_var('COREBASE'), 'meta/files/layers.schema.json')
+ result = runCmd("{} {} -i {} {}".format(python, jsonvalidator, json, jsonschema))
+
+ def test_validate_examplelayersjson(self):
+ json = os.path.join(get_bb_var('COREBASE'), "meta/files/layers.example.json")
+ self.validate_layersjson(json)
--
2.30.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 4/7] bitbake-layers: add ability to save current layer repository configuration into a file
2022-08-31 11:13 [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid Alexander Kanavin
2022-08-31 11:13 ` [PATCH 2/7] bitbake-layers: add a command to save the active build configuration as a template into a layer Alexander Kanavin
2022-08-31 11:13 ` [PATCH 3/7] meta/files: add layer setup JSON schema and example Alexander Kanavin
@ 2022-08-31 11:13 ` Alexander Kanavin
2022-08-31 11:13 ` [PATCH 5/7] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file Alexander Kanavin
` (3 subsequent siblings)
6 siblings, 0 replies; 21+ messages in thread
From: Alexander Kanavin @ 2022-08-31 11:13 UTC (permalink / raw)
To: openembedded-core; +Cc: Alexander Kanavin
This addresses a long standing gap in the core offering:
there is no tooling to capture the currently configured layers
with their revisions, or restore the layers from a configuration
file (without using external tools, some of which aren't particularly
suitable for the task). This plugin addresses the 'capture' part.
Note that the actual writing is performed by a sub-plugin; one such
sub-plugin is provided (for the json + python script format), but
more can be added (e.g. kas, repo, etc.).
How to save a layer configuration:
a) Running with default choices:
$ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/
NOTE: Starting bitbake server...
NOTE: Created /srv/work/alex/meta-alex/setup-layers.json
NOTE: Created /srv/work/alex/meta-alex/setup-layers
b) Command line options:
NOTE: Starting bitbake server...
usage: bitbake-layers create-layers-setup [-h] [--output-prefix OUTPUT_PREFIX] [--writer {oe-setup-layers}] [--json-only] destdir
Writes out a configuration file and/or a script that replicate the directory structure and revisions of the layers in a current build.
positional arguments:
destdir Directory where to write the output
(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching).
optional arguments:
-h, --help show this help message and exit
--output-prefix OUTPUT_PREFIX, -o OUTPUT_PREFIX
File name prefix for the output files, if the default (setup-layers) is undesirable.
--writer {oe-setup-layers}, -w {oe-setup-layers}
Choose the output format (defaults to oe-setup-layers).
Currently supported options are:
oe-setup-layers - a self-contained python script and a json config for it.
--json-only When using the oe-setup-layers writer, write only the layer configuruation in json format. Otherwise, also a copy of scripts/oe-setup-layers (from oe-core or poky) is provided, which is a self contained python script that fetches all the needed layers and sets them to correct revisions using the data from the json.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
meta/lib/bblayers/makesetup.py | 108 ++++++++++++++++++
.../bblayers/setupwriters/oe-setup-layers.py | 50 ++++++++
2 files changed, 158 insertions(+)
create mode 100644 meta/lib/bblayers/makesetup.py
create mode 100644 meta/lib/bblayers/setupwriters/oe-setup-layers.py
diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py
new file mode 100644
index 0000000000..bef6da0ea8
--- /dev/null
+++ b/meta/lib/bblayers/makesetup.py
@@ -0,0 +1,108 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import logging
+import os
+import stat
+import sys
+import shutil
+
+import bb.utils
+import bb.process
+
+from bblayers.common import LayerPlugin
+
+logger = logging.getLogger('bitbake-layers')
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
+
+import oe.buildcfg
+
+def plugin_init(plugins):
+ return MakeSetupPlugin()
+
+class MakeSetupPlugin(LayerPlugin):
+
+ def _get_repo_path(self, layer_path):
+ repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path)
+ return repo_path.strip()
+
+ def _get_remotes(self, repo_path):
+ remotes = {}
+ remotes_list,_ = bb.process.run('git remote', cwd=repo_path)
+ for r in remotes_list.split():
+ uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path)
+ remotes[r] = {'uri':uri.strip()}
+ return remotes
+
+ def _get_describe(self, repo_path):
+ try:
+ describe,_ = bb.process.run('git describe --tags', cwd=repo_path)
+ except bb.process.ExecutionError:
+ return ""
+ return describe.strip()
+
+ def make_repo_config(self, destdir):
+ """ This is a helper function for the writer plugins that discovers currently confugured layers.
+ The writers do not have to use it, but it can save a bit of work and avoid duplicated code, hence it is
+ available here. """
+ repos = {}
+ layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data)
+ try:
+ destdir_repo = self._get_repo_path(destdir)
+ except bb.process.ExecutionError:
+ destdir_repo = None
+
+ for (l_path, l_name, l_branch, l_rev, l_ismodified) in layers:
+ if l_name == 'workspace':
+ continue
+ if l_ismodified:
+ logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l_name,path=l_path))
+ return
+ repo_path = self._get_repo_path(l_path)
+ if repo_path not in repos.keys():
+ repos[repo_path] = {'path':os.path.basename(repo_path),'layers':{},'git-remote':{'rev':l_rev, 'branch':l_branch, 'remotes':self._get_remotes(repo_path), 'describe':self._get_describe(repo_path)}}
+ if repo_path == destdir_repo:
+ repos[repo_path]['contains_this_file'] = True
+ if not repos[repo_path]['git-remote']['remotes'] and not repos[repo_path]['contains_this_file']:
+ logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path))
+ return
+ repos[repo_path]['layers'][l_name] = {'subpath':l_path.replace(repo_path,'')[1:]}
+
+ top_path = os.path.commonpath([os.path.dirname(r) for r in repos.keys()])
+
+ repos_nopaths = {}
+ for r in repos.keys():
+ r_nopath = os.path.basename(r)
+ repos_nopaths[r_nopath] = repos[r]
+ r_relpath = os.path.relpath(r, top_path)
+ repos_nopaths[r_nopath]['path'] = r_relpath
+ return repos_nopaths
+
+ def do_make_setup(self, args):
+ """ Writes out a configuration file and/or a script that replicate the directory structure and revisions of the layers in a current build. """
+ for p in self.plugins:
+ if str(p) == args.writer:
+ p.do_write(self, args)
+
+ def register_commands(self, sp):
+ parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False)
+ parser_setup_layers.add_argument('destdir',
+ help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching).')
+ parser_setup_layers.add_argument('--output-prefix', '-o',
+ help='File name prefix for the output files, if the default (setup-layers) is undesirable.')
+
+ self.plugins = []
+
+ for path in (self.tinfoil.config_data.getVar('BBPATH').split(':')):
+ pluginpath = os.path.join(path, 'lib', 'bblayers', 'setupwriters')
+ bb.utils.load_plugins(logger, self.plugins, pluginpath)
+
+ parser_setup_layers.add_argument('--writer', '-w', choices=[str(p) for p in self.plugins], help="Choose the output format (defaults to oe-setup-layers).\n\nCurrently supported options are:\noe-setup-layers - a self-contained python script and a json config for it.\n\n", default="oe-setup-layers")
+
+ for plugin in self.plugins:
+ if hasattr(plugin, 'register_arguments'):
+ plugin.register_arguments(parser_setup_layers)
diff --git a/meta/lib/bblayers/setupwriters/oe-setup-layers.py b/meta/lib/bblayers/setupwriters/oe-setup-layers.py
new file mode 100644
index 0000000000..f6a484b766
--- /dev/null
+++ b/meta/lib/bblayers/setupwriters/oe-setup-layers.py
@@ -0,0 +1,50 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import logging
+import os
+import json
+import stat
+
+logger = logging.getLogger('bitbake-layers')
+
+def plugin_init(plugins):
+ return OeSetupLayersWriter()
+
+class OeSetupLayersWriter():
+
+ def __str__(self):
+ return "oe-setup-layers"
+
+ def _write_python(self, input, output):
+ with open(input) as f:
+ script = f.read()
+ with open(output, 'w') as f:
+ f.write(script)
+ st = os.stat(output)
+ os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)
+
+ def _write_json(self, repos, output):
+ with open(output, 'w') as f:
+ json.dump(repos, f, sort_keys=True, indent=4)
+
+ def do_write(self, parent, args):
+ """ Writes out a python script and a json config that replicate the directory structure and revisions of the layers in a current build. """
+ repos = parent.make_repo_config(args.destdir)
+ json = {"version":"1.0","sources":repos}
+ if not repos:
+ raise Exception("Could not determine layer sources")
+ output = args.output_prefix or "setup-layers"
+ output = os.path.join(os.path.abspath(args.destdir),output)
+ self._write_json(json, output + ".json")
+ logger.info('Created {}.json'.format(output))
+ if not args.json_only:
+ self._write_python(os.path.join(os.path.dirname(__file__),'../../../../scripts/oe-setup-layers'), output)
+ logger.info('Created {}'.format(output))
+
+ def register_arguments(self, parser):
+ parser.add_argument('--json-only', action='store_true',
+ help='When using the oe-setup-layers writer, write only the layer configuruation in json format. Otherwise, also a copy of scripts/oe-setup-layers (from oe-core or poky) is provided, which is a self contained python script that fetches all the needed layers and sets them to correct revisions using the data from the json.')
--
2.30.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 5/7] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file
2022-08-31 11:13 [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid Alexander Kanavin
` (2 preceding siblings ...)
2022-08-31 11:13 ` [PATCH 4/7] bitbake-layers: add ability to save current layer repository configuration into a file Alexander Kanavin
@ 2022-08-31 11:13 ` Alexander Kanavin
2022-08-31 11:14 ` [PATCH 6/7] selftest/bblayers: add a test for creating a layer setup and using it to restore the layers Alexander Kanavin
` (2 subsequent siblings)
6 siblings, 0 replies; 21+ messages in thread
From: Alexander Kanavin @ 2022-08-31 11:13 UTC (permalink / raw)
To: openembedded-core; +Cc: Alexander Kanavin
This script can be used directly from poky or oe-core, or can be copied directly
into alayer or any other repository - it is self-suffucient and requires only python3
and git on the host where it will run. It is also copied by the bitbake-layers
layers-setup plugin together with the json, unless requested otherwise.
1. How to restore the layers from the saved configuration:
a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it.
(use 'bitbake-layers create-layer-setup' from the previous commit to create them)
b) Running with default options:
(note: this will work to update an existing checkout as well)
alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers
Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override.
Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master
Running 'git init -q /srv/work/alex/my-build/meta-intel'
Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel
Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel
Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel
Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers
Running 'git init -q /srv/work/alex/my-build/poky'
Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky
Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky
Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky
Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky
Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky
2. Command line options:
alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h
usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA]
A self contained python script that fetches all the needed layers and sets them to correct revisions
optional arguments:
-h, --help show this help message and exit
--force-bootstraplayer-checkout
Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place).
--destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build).
--jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json).
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
scripts/oe-setup-layers | 88 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 88 insertions(+)
create mode 100755 scripts/oe-setup-layers
diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers
new file mode 100755
index 0000000000..cbd2efb5c7
--- /dev/null
+++ b/scripts/oe-setup-layers
@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+# This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running
+#
+# bitbake-layers create-layers-setup destdir
+#
+# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy.
+
+import argparse
+import json
+import os
+import subprocess
+
+def _do_checkout(args, json):
+ layers = json['sources']
+ buildconfs = []
+ oecorepath = ""
+ for l_name in layers:
+ l_data = layers[l_name]
+ layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path']))
+
+ for ll_name in l_data['layers']:
+ if ll_name == 'meta':
+ oecorepath = layerdir
+ ll_data = l_data['layers'][ll_name]
+ if 'buildconfigs' in ll_data:
+ for c in ll_data['buildconfigs']:
+ buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c))
+
+ if 'contains_this_file' in l_data.keys():
+ force_arg = 'force_bootstraplayer_checkout'
+ if not args[force_arg]:
+ print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout'))
+ continue
+ l_remote = l_data['git-remote']
+ rev = l_remote['rev']
+ desc = l_remote['describe']
+ if not desc:
+ desc = rev[:10]
+ branch = l_remote['branch']
+ remotes = l_remote['remotes']
+
+ print('\nSetting up source {}, revision {}, branch {}'.format(l_name, desc, branch))
+ cmd = 'git init -q {}'.format(layerdir)
+ print("Running '{}'".format(cmd))
+ subprocess.check_output(cmd, shell=True)
+
+ for remote in remotes:
+ cmd = "git remote remove {} > /dev/null 2>&1; git remote add {} {}".format(remote, remote, remotes[remote]['uri'])
+ print("Running '{}' in {}".format(cmd, layerdir))
+ subprocess.check_output(cmd, shell=True, cwd=layerdir)
+
+ cmd = "git fetch -q {} || true".format(remote)
+ print("Running '{}' in {}".format(cmd, layerdir))
+ subprocess.check_output(cmd, shell=True, cwd=layerdir)
+
+ cmd = 'git checkout -q {}'.format(rev)
+ print("Running '{}' in {}".format(cmd, layerdir))
+ subprocess.check_output(cmd, shell=True, cwd=layerdir)
+
+parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/")
+
+parser.add_argument('--force-bootstraplayer-checkout', action='store_true',
+ help='Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place).')
+
+try:
+ defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', universal_newlines=True, shell=True, cwd=os.path.dirname(__file__)))
+except subprocess.CalledProcessError as e:
+ defaultdest = os.path.abspath(".")
+
+parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest))
+parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json"))
+
+args = parser.parse_args()
+
+with open(args.jsondata) as f:
+ json = json.load(f)
+
+supported_versions = ["1.0"]
+if json["version"] not in supported_versions:
+ raise Exception("File {} has version {}, which is not in supported versions: {}".format(args.jsondata, json["version"], supported_versions))
+
+_do_checkout(vars(args), json)
--
2.30.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 6/7] selftest/bblayers: add a test for creating a layer setup and using it to restore the layers
2022-08-31 11:13 [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid Alexander Kanavin
` (3 preceding siblings ...)
2022-08-31 11:13 ` [PATCH 5/7] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file Alexander Kanavin
@ 2022-08-31 11:14 ` Alexander Kanavin
2022-08-31 11:14 ` [PATCH 7/7] selftest/bblayers: adjust the revision for the layer setup test Alexander Kanavin
2022-09-14 3:27 ` [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid ChenQi
6 siblings, 0 replies; 21+ messages in thread
From: Alexander Kanavin @ 2022-08-31 11:14 UTC (permalink / raw)
To: openembedded-core; +Cc: Alexander Kanavin
This does a basic run-through of the bitbake-layers plugin, and the resulting json layer config
and the layer setup script that uses it. Only poky is actually fetched by the script.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
meta/lib/oeqa/selftest/cases/bblayers.py | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py
index c753a7b795..18007764b3 100644
--- a/meta/lib/oeqa/selftest/cases/bblayers.py
+++ b/meta/lib/oeqa/selftest/cases/bblayers.py
@@ -142,3 +142,25 @@ class BitbakeLayers(OESelftestTestCase):
def test_validate_examplelayersjson(self):
json = os.path.join(get_bb_var('COREBASE'), "meta/files/layers.example.json")
self.validate_layersjson(json)
+
+ def test_bitbakelayers_setup(self):
+ result = runCmd('bitbake-layers create-layers-setup {}'.format(self.testlayer_path))
+ jsonfile = os.path.join(self.testlayer_path, "setup-layers.json")
+ self.validate_layersjson(jsonfile)
+
+ # The revision-under-test may not necessarily be available on the remote server,
+ # so replace it with a stable release tag.
+ import json
+ with open(jsonfile) as f:
+ data = json.load(f)
+ for s in data['sources']:
+ data['sources'][s]['git-remote']['rev'] = 'yocto-4.0'
+ with open(jsonfile, 'w') as f:
+ json.dump(data, f)
+
+ testcheckoutdir = os.path.join(self.builddir, 'test-layer-checkout')
+ result = runCmd('{}/setup-layers --destdir {}'.format(self.testlayer_path, testcheckoutdir))
+ # May not necessarily be named 'poky' or 'openembedded-core'
+ oecoredir = os.listdir(testcheckoutdir)[0]
+ testcheckoutfile = os.path.join(testcheckoutdir, oecoredir, "oe-init-build-env")
+ self.assertTrue(os.path.exists(testcheckoutfile), "File {} not found in test layer checkout".format(testcheckoutfile))
--
2.30.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 7/7] selftest/bblayers: adjust the revision for the layer setup test
2022-08-31 11:13 [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid Alexander Kanavin
` (4 preceding siblings ...)
2022-08-31 11:14 ` [PATCH 6/7] selftest/bblayers: add a test for creating a layer setup and using it to restore the layers Alexander Kanavin
@ 2022-08-31 11:14 ` Alexander Kanavin
2022-09-14 3:27 ` [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid ChenQi
6 siblings, 0 replies; 21+ messages in thread
From: Alexander Kanavin @ 2022-08-31 11:14 UTC (permalink / raw)
To: openembedded-core; +Cc: Alexander Kanavin
Using a tag is not actually robust enough, e.g. poky-contrib
checkouts do not come with any tags. So let's use a revision
matching yocto-4.0, that ought to be present.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
meta/lib/oeqa/selftest/cases/bblayers.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py
index 18007764b3..c6bd5a1f6a 100644
--- a/meta/lib/oeqa/selftest/cases/bblayers.py
+++ b/meta/lib/oeqa/selftest/cases/bblayers.py
@@ -149,12 +149,12 @@ class BitbakeLayers(OESelftestTestCase):
self.validate_layersjson(jsonfile)
# The revision-under-test may not necessarily be available on the remote server,
- # so replace it with a stable release tag.
+ # so replace it with a revision that has a yocto-4.0 tag.
import json
with open(jsonfile) as f:
data = json.load(f)
for s in data['sources']:
- data['sources'][s]['git-remote']['rev'] = 'yocto-4.0'
+ data['sources'][s]['git-remote']['rev'] = '00cfdde791a0176c134f31e5a09eff725e75b905'
with open(jsonfile, 'w') as f:
json.dump(data, f)
--
2.30.2
^ permalink raw reply related [flat|nested] 21+ messages in thread* Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-08-31 11:13 [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid Alexander Kanavin
` (5 preceding siblings ...)
2022-08-31 11:14 ` [PATCH 7/7] selftest/bblayers: adjust the revision for the layer setup test Alexander Kanavin
@ 2022-09-14 3:27 ` ChenQi
2022-09-14 5:00 ` Alexander Kanavin
6 siblings, 1 reply; 21+ messages in thread
From: ChenQi @ 2022-09-14 3:27 UTC (permalink / raw)
To: Alexander Kanavin, openembedded-core; +Cc: Alexander Kanavin
Hi Alex,
Why must TEMPLATECONF be under a layer?
Regards,
Qi
On 8/31/22 19:13, Alexander Kanavin wrote:
> specifically that ../../layer.conf exists, and that second-from-last
> component in the path is 'templates'.
>
> This requires tweaking template.conf creation in eSDK bbclass, as
> we need to ensure that the path in it is valid, and exists
> (which may not be the case if the SDK is poky-based).
>
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
> meta/classes-recipe/populate_sdk_ext.bbclass | 3 ++-
> scripts/oe-setup-builddir | 5 +++++
> 2 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass
> index 56e24c4eed..925cb313fc 100644
> --- a/meta/classes-recipe/populate_sdk_ext.bbclass
> +++ b/meta/classes-recipe/populate_sdk_ext.bbclass
> @@ -438,7 +438,8 @@ python copy_buildsystem () {
> else:
> # Write a templateconf.cfg
> with open(baseoutpath + '/conf/templateconf.cfg', 'w') as f:
> - f.write('meta/conf\n')
> + f.write('meta/conf/templates/default\n')
> + os.makedirs(os.path.join(baseoutpath, core_meta_subdir, 'conf/templates/default'), exist_ok=True)
>
> # Ensure any variables set from the external environment (by way of
> # BB_ENV_PASSTHROUGH_ADDITIONS) are set in the SDK's configuration
> diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
> index 5d644168cb..bf832ee0ca 100755
> --- a/scripts/oe-setup-builddir
> +++ b/scripts/oe-setup-builddir
> @@ -61,6 +61,11 @@ if [ -n "$TEMPLATECONF" ]; then
> echo >&2 "Error: TEMPLATECONF value points to nonexistent directory '$TEMPLATECONF'"
> exit 1
> fi
> + templatesdir=$(python3 -c "import sys; print(sys.argv[1].strip('/').split('/')[-2])" $TEMPLATECONF)
> + if [ ! -f "$TEMPLATECONF/../../layer.conf" -o $templatesdir != "templates" ]; then
> + echo >&2 "Error: TEMPLATECONF value (which is $TEMPLATECONF) must point to meta-some-layer/conf/templates/template-name"
> + exit 1
> + fi
> fi
> OECORELAYERCONF="$TEMPLATECONF/bblayers.conf.sample"
> OECORELOCALCONF="$TEMPLATECONF/local.conf.sample"
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#170101): https://lists.openembedded.org/g/openembedded-core/message/170101
> Mute This Topic: https://lists.openembedded.org/mt/93368468/3618072
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [Qi.Chen@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-14 3:27 ` [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid ChenQi
@ 2022-09-14 5:00 ` Alexander Kanavin
2022-09-14 5:49 ` ChenQi
0 siblings, 1 reply; 21+ messages in thread
From: Alexander Kanavin @ 2022-09-14 5:00 UTC (permalink / raw)
To: ChenQi; +Cc: OE-core, Alexander Kanavin
Hello,
this is done for reasons of discoverability and consistency with
machine and distro definitions: they all should be under
meta-somelayer/conf/. This way both tools and humans can easily find
the config templates, and be certain that this is the only location..
Alex
On Wed, 14 Sept 2022 at 05:27, ChenQi <Qi.Chen@windriver.com> wrote:
>
> Hi Alex,
>
> Why must TEMPLATECONF be under a layer?
>
> Regards,
> Qi
>
> On 8/31/22 19:13, Alexander Kanavin wrote:
> > specifically that ../../layer.conf exists, and that second-from-last
> > component in the path is 'templates'.
> >
> > This requires tweaking template.conf creation in eSDK bbclass, as
> > we need to ensure that the path in it is valid, and exists
> > (which may not be the case if the SDK is poky-based).
> >
> > Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> > ---
> > meta/classes-recipe/populate_sdk_ext.bbclass | 3 ++-
> > scripts/oe-setup-builddir | 5 +++++
> > 2 files changed, 7 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass
> > index 56e24c4eed..925cb313fc 100644
> > --- a/meta/classes-recipe/populate_sdk_ext.bbclass
> > +++ b/meta/classes-recipe/populate_sdk_ext.bbclass
> > @@ -438,7 +438,8 @@ python copy_buildsystem () {
> > else:
> > # Write a templateconf.cfg
> > with open(baseoutpath + '/conf/templateconf.cfg', 'w') as f:
> > - f.write('meta/conf\n')
> > + f.write('meta/conf/templates/default\n')
> > + os.makedirs(os.path.join(baseoutpath, core_meta_subdir, 'conf/templates/default'), exist_ok=True)
> >
> > # Ensure any variables set from the external environment (by way of
> > # BB_ENV_PASSTHROUGH_ADDITIONS) are set in the SDK's configuration
> > diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
> > index 5d644168cb..bf832ee0ca 100755
> > --- a/scripts/oe-setup-builddir
> > +++ b/scripts/oe-setup-builddir
> > @@ -61,6 +61,11 @@ if [ -n "$TEMPLATECONF" ]; then
> > echo >&2 "Error: TEMPLATECONF value points to nonexistent directory '$TEMPLATECONF'"
> > exit 1
> > fi
> > + templatesdir=$(python3 -c "import sys; print(sys.argv[1].strip('/').split('/')[-2])" $TEMPLATECONF)
> > + if [ ! -f "$TEMPLATECONF/../../layer.conf" -o $templatesdir != "templates" ]; then
> > + echo >&2 "Error: TEMPLATECONF value (which is $TEMPLATECONF) must point to meta-some-layer/conf/templates/template-name"
> > + exit 1
> > + fi
> > fi
> > OECORELAYERCONF="$TEMPLATECONF/bblayers.conf.sample"
> > OECORELOCALCONF="$TEMPLATECONF/local.conf.sample"
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#170101): https://lists.openembedded.org/g/openembedded-core/message/170101
> > Mute This Topic: https://lists.openembedded.org/mt/93368468/3618072
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [Qi.Chen@windriver.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-14 5:00 ` Alexander Kanavin
@ 2022-09-14 5:49 ` ChenQi
2022-09-14 6:39 ` Alexander Kanavin
0 siblings, 1 reply; 21+ messages in thread
From: ChenQi @ 2022-09-14 5:49 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: OE-core, Alexander Kanavin
On 9/14/22 13:00, Alexander Kanavin wrote:
> Hello,
>
> this is done for reasons of discoverability and consistency with
> machine and distro definitions: they all should be under
> meta-somelayer/conf/. This way both tools and humans can easily find
> the config templates, and be certain that this is the only location..
>
> Alex
Hi Alex,
I'm reluctant to agree that this is like machine and distro, because
it's a hard requirement that machine and distro definitions be under
some layer, otherwise how can bitbake get info about it? But
TEMPLATECONF seems to be a different case, it could be everywhere
because it's used by the project setup script.
In our case, the TEMPLATECONF is <project_dir>/config/, and layers are
<project_dir>/<layer>/. This directory hierarchy has been working for
years until recent changes.
Do you think such directory hierarchy makes sense? How about we deleting
such check if there's no strong technical reason to do so? By 'strong
technical reason', I mean that some codes in oe-core are written based
on this assumption (this is the part I'm sure about).
Regards,
Qi
> On Wed, 14 Sept 2022 at 05:27, ChenQi <Qi.Chen@windriver.com> wrote:
>> Hi Alex,
>>
>> Why must TEMPLATECONF be under a layer?
>>
>> Regards,
>> Qi
>>
>> On 8/31/22 19:13, Alexander Kanavin wrote:
>>> specifically that ../../layer.conf exists, and that second-from-last
>>> component in the path is 'templates'.
>>>
>>> This requires tweaking template.conf creation in eSDK bbclass, as
>>> we need to ensure that the path in it is valid, and exists
>>> (which may not be the case if the SDK is poky-based).
>>>
>>> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
>>> ---
>>> meta/classes-recipe/populate_sdk_ext.bbclass | 3 ++-
>>> scripts/oe-setup-builddir | 5 +++++
>>> 2 files changed, 7 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/meta/classes-recipe/populate_sdk_ext.bbclass b/meta/classes-recipe/populate_sdk_ext.bbclass
>>> index 56e24c4eed..925cb313fc 100644
>>> --- a/meta/classes-recipe/populate_sdk_ext.bbclass
>>> +++ b/meta/classes-recipe/populate_sdk_ext.bbclass
>>> @@ -438,7 +438,8 @@ python copy_buildsystem () {
>>> else:
>>> # Write a templateconf.cfg
>>> with open(baseoutpath + '/conf/templateconf.cfg', 'w') as f:
>>> - f.write('meta/conf\n')
>>> + f.write('meta/conf/templates/default\n')
>>> + os.makedirs(os.path.join(baseoutpath, core_meta_subdir, 'conf/templates/default'), exist_ok=True)
>>>
>>> # Ensure any variables set from the external environment (by way of
>>> # BB_ENV_PASSTHROUGH_ADDITIONS) are set in the SDK's configuration
>>> diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
>>> index 5d644168cb..bf832ee0ca 100755
>>> --- a/scripts/oe-setup-builddir
>>> +++ b/scripts/oe-setup-builddir
>>> @@ -61,6 +61,11 @@ if [ -n "$TEMPLATECONF" ]; then
>>> echo >&2 "Error: TEMPLATECONF value points to nonexistent directory '$TEMPLATECONF'"
>>> exit 1
>>> fi
>>> + templatesdir=$(python3 -c "import sys; print(sys.argv[1].strip('/').split('/')[-2])" $TEMPLATECONF)
>>> + if [ ! -f "$TEMPLATECONF/../../layer.conf" -o $templatesdir != "templates" ]; then
>>> + echo >&2 "Error: TEMPLATECONF value (which is $TEMPLATECONF) must point to meta-some-layer/conf/templates/template-name"
>>> + exit 1
>>> + fi
>>> fi
>>> OECORELAYERCONF="$TEMPLATECONF/bblayers.conf.sample"
>>> OECORELOCALCONF="$TEMPLATECONF/local.conf.sample"
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-
>>> Links: You receive all messages sent to this group.
>>> View/Reply Online (#170101): https://lists.openembedded.org/g/openembedded-core/message/170101
>>> Mute This Topic: https://lists.openembedded.org/mt/93368468/3618072
>>> Group Owner: openembedded-core+owner@lists.openembedded.org
>>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [Qi.Chen@windriver.com]
>>> -=-=-=-=-=-=-=-=-=-=-=-
>>>
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-14 5:49 ` ChenQi
@ 2022-09-14 6:39 ` Alexander Kanavin
2022-09-14 7:42 ` ChenQi
0 siblings, 1 reply; 21+ messages in thread
From: Alexander Kanavin @ 2022-09-14 6:39 UTC (permalink / raw)
To: ChenQi; +Cc: OE-core, Alexander Kanavin
On Wed, 14 Sept 2022 at 07:49, ChenQi <Qi.Chen@windriver.com> wrote:
> I'm reluctant to agree that this is like machine and distro, because
> it's a hard requirement that machine and distro definitions be under
> some layer, otherwise how can bitbake get info about it? But
> TEMPLATECONF seems to be a different case, it could be everywhere
> because it's used by the project setup script.
>
> In our case, the TEMPLATECONF is <project_dir>/config/, and layers are
> <project_dir>/<layer>/. This directory hierarchy has been working for
> years until recent changes.
>
> Do you think such directory hierarchy makes sense? How about we deleting
> such check if there's no strong technical reason to do so? By 'strong
> technical reason', I mean that some codes in oe-core are written based
> on this assumption (this is the part I'm sure about).
Again, it's not about only code. It's about humans too: we benefit
from having things where we expect them to be. If your templates are
in meta-layer/conf/templates/ you do not have to document, explain or
support this; anyone new to the project will simply pick this up from
prior experience or official documentation.
That said, there is already code that makes this assumption too: both
'bitbake-layers save-template-conf' and upcoming 'oe-setup-build' (the
patch was sent for review here) consider only
meta-layer/conf/templates.
I have to ask the same question to you: is there a strong technical
reason that you cannot move the templates to the new standard
location?
Alex
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-14 6:39 ` Alexander Kanavin
@ 2022-09-14 7:42 ` ChenQi
2022-09-14 8:03 ` Alexander Kanavin
0 siblings, 1 reply; 21+ messages in thread
From: ChenQi @ 2022-09-14 7:42 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: OE-core, Alexander Kanavin
On 9/14/22 14:39, Alexander Kanavin wrote:
> On Wed, 14 Sept 2022 at 07:49, ChenQi <Qi.Chen@windriver.com> wrote:
>> I'm reluctant to agree that this is like machine and distro, because
>> it's a hard requirement that machine and distro definitions be under
>> some layer, otherwise how can bitbake get info about it? But
>> TEMPLATECONF seems to be a different case, it could be everywhere
>> because it's used by the project setup script.
>>
>> In our case, the TEMPLATECONF is <project_dir>/config/, and layers are
>> <project_dir>/<layer>/. This directory hierarchy has been working for
>> years until recent changes.
>>
>> Do you think such directory hierarchy makes sense? How about we deleting
>> such check if there's no strong technical reason to do so? By 'strong
>> technical reason', I mean that some codes in oe-core are written based
>> on this assumption (this is the part I'm sure about).
> Again, it's not about only code. It's about humans too: we benefit
> from having things where we expect them to be.
$TEMPLATECONF/bblayers.conf.sample has a list of layers. These layers
may have dependencies on each other or they may not. Which layer do you
think should this TEMPLATECONF locate?
TEMPLATECONF, by its nature, is a project setup variable. It logically
does not belong to any layer.
Why would people expect some project level variable to point to some
directory under a layer?
> If your templates are
> in meta-layer/conf/templates/ you do not have to document, explain or
> support this;
Again, why this meta-layer should have knowledge about the whole
project? It should be the project that has knowledge about layer, not
the other way around.
> anyone new to the project will simply pick this up from
> prior experience or official documentation.
When users see a file in a layer that refers to other layers which this
layer does not depend on and not been dependent upon, they may ask why.
>
> That said, there is already code that makes this assumption too: both
> 'bitbake-layers save-template-conf' and upcoming 'oe-setup-build' (the
> patch was sent for review here) consider only
> meta-layer/conf/templates.
Give it a second thought.
> I have to ask the same question to you: is there a strong technical
> reason that you cannot move the templates to the new standard
> location?
Yes. The bblayers.conf.sample is generated dynamically according to
project setup.
That said, I could of course create a useless layer that does nothing
but only holds these sample files to satisfy this sanity check. But I do
think this sanity check is logically wrong.
Again, project contains layers, project setup could choose to use layers
the project wants. Forcing a project level variable to point to a
location under a layer is not reasonable.
Regards,
Qi
> Alex
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-14 7:42 ` ChenQi
@ 2022-09-14 8:03 ` Alexander Kanavin
2022-09-14 8:17 ` ChenQi
0 siblings, 1 reply; 21+ messages in thread
From: Alexander Kanavin @ 2022-09-14 8:03 UTC (permalink / raw)
To: ChenQi; +Cc: OE-core, Alexander Kanavin
On Wed, 14 Sept 2022 at 09:42, ChenQi <Qi.Chen@windriver.com> wrote:
> Yes. The bblayers.conf.sample is generated dynamically according to
> project setup.
Can we step back for a moment, I'd like to know more about how you do
this. Is the code that does dynamic generation of the list in bblayers
published somewhere? How a decision to include (or not) a layer is
made?
Alex
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-14 8:03 ` Alexander Kanavin
@ 2022-09-14 8:17 ` ChenQi
2022-09-14 8:42 ` Alexander Kanavin
0 siblings, 1 reply; 21+ messages in thread
From: ChenQi @ 2022-09-14 8:17 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: OE-core, Alexander Kanavin
On 9/14/22 16:03, Alexander Kanavin wrote:
> On Wed, 14 Sept 2022 at 09:42, ChenQi <Qi.Chen@windriver.com> wrote:
>
>> Yes. The bblayers.conf.sample is generated dynamically according to
>> project setup.
> Can we step back for a moment, I'd like to know more about how you do
> this. Is the code that does dynamic generation of the list in bblayers
> published somewhere? How a decision to include (or not) a layer is
> made?
>
> Alex
The codes are in: https://github.com/WindRiver-Labs/wrlinux-x
e.g.
./wrlinux-x/setup.sh --all-layers (this pulls down all supported layers)
./wrlinux-x/setup.sh --all-layers --dl-layers (this pulls down all
supported layers, together with their needed tarballs/git repos)
./wrlinux-x/setup.sh (this pulls down the default list of layers)
./wrlinux-x/setup.sh --layers meta-xxx (this pulls down the default list
of layers + meta-xxx and its dependencies)
And the final list of layers are in
<project_dir>/config/bblayers.conf.sample, which will be updated if
needed when re-running the setup.sh script.
Regards,
Qi
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-14 8:17 ` ChenQi
@ 2022-09-14 8:42 ` Alexander Kanavin
2022-09-15 1:07 ` Peter Kjellerstedt
0 siblings, 1 reply; 21+ messages in thread
From: Alexander Kanavin @ 2022-09-14 8:42 UTC (permalink / raw)
To: ChenQi; +Cc: OE-core, Alexander Kanavin
On Wed, 14 Sept 2022 at 10:17, ChenQi <Qi.Chen@windriver.com> wrote:
> The codes are in: https://github.com/WindRiver-Labs/wrlinux-x
>
> e.g.
>
> ./wrlinux-x/setup.sh --all-layers (this pulls down all supported layers)
>
> ./wrlinux-x/setup.sh --all-layers --dl-layers (this pulls down all
> supported layers, together with their needed tarballs/git repos)
>
> ./wrlinux-x/setup.sh (this pulls down the default list of layers)
>
> ./wrlinux-x/setup.sh --layers meta-xxx (this pulls down the default list
> of layers + meta-xxx and its dependencies)
>
> And the final list of layers are in
> <project_dir>/config/bblayers.conf.sample, which will be updated if
> needed when re-running the setup.sh script.
May I suggest that the tool skips the bblayers.conf.sample step
altogether, and simply writes the desired layers directly into
bblayers.conf? It can do this with 'bitbake-layers add-layer'. Is
there a need for doing the intermediate step of generating the
template?
Just to be clear where I am coming from: anything in TEMPLATECONF must
not be auto-generated during build setup, and must be under version
control. We are trying to standardize layer setup and configuration
handling, and this would be a necessary step towards that.
Alex
^ permalink raw reply [flat|nested] 21+ messages in thread
* RE: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-14 8:42 ` Alexander Kanavin
@ 2022-09-15 1:07 ` Peter Kjellerstedt
2022-09-15 9:20 ` Alexander Kanavin
0 siblings, 1 reply; 21+ messages in thread
From: Peter Kjellerstedt @ 2022-09-15 1:07 UTC (permalink / raw)
To: Alexander Kanavin, ChenQi; +Cc: OE-core, Alexander Kanavin
> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Alexander Kanavin
> Sent: den 14 september 2022 09:42
> To: ChenQi <Qi.Chen@windriver.com>
> Cc: OE-core <openembedded-core@lists.openembedded.org>; Alexander Kanavin
> <alex@linutronix.de>
> Subject: Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check
> that TEMPLATECONF is valid
>
> On Wed, 14 Sept 2022 at 10:17, ChenQi <Qi.Chen@windriver.com> wrote:
> > The codes are in: https://github.com/WindRiver-Labs/wrlinux-x
> >
> > e.g.
> >
> > ./wrlinux-x/setup.sh --all-layers (this pulls down all supported layers)
> >
> > ./wrlinux-x/setup.sh --all-layers --dl-layers (this pulls down all
> > supported layers, together with their needed tarballs/git repos)
> >
> > ./wrlinux-x/setup.sh (this pulls down the default list of layers)
> >
> > ./wrlinux-x/setup.sh --layers meta-xxx (this pulls down the default list
> > of layers + meta-xxx and its dependencies)
> >
> > And the final list of layers are in
> > <project_dir>/config/bblayers.conf.sample, which will be updated if
> > needed when re-running the setup.sh script.
>
> May I suggest that the tool skips the bblayers.conf.sample step
> altogether, and simply writes the desired layers directly into
> bblayers.conf? It can do this with 'bitbake-layers add-layer'. Is
> there a need for doing the intermediate step of generating the
> template?
I know you have suggested this to me too as we use a similar setup to
generate the bblayers.conf.sample file based on the layers that repo
has checked out in our environment, and now I realized why this is not
as simple as it seems. The problem (at least in our case) is that we
run as a wrapper around oe-init-build-env. This means that before the
wrapper sources oe-init-build-env, the build directory and thus the
conf directory inside it (where the bblayers.conf file is supposed to
go) may not exist. And after it has sourced oe-init-build-env it is
too late as the bblayers.conf file needs to exists before. So to create
the bblayers.conf file, the wrapper would have to duplicate all the
non-trivial code in oe-setup-builddir that determines what the build
directory should be called. And while this of course is possible to
do, it means duplicating code that is internal to bitbake, and risking
missing important changes to the upstream code in the future. Thus it
is much easier and safer to generate the template files and let
oe-init-build-env do its job.
>
> Just to be clear where I am coming from: anything in TEMPLATECONF must
> not be auto-generated during build setup, and must be under version
> control. We are trying to standardize layer setup and configuration
> handling, and this would be a necessary step towards that.
Where I (and apparently WindRiver) are coming from, the decision on what
layers are part of the configuration is not something a specific layer
is responsible for. I.e., in our setup we fetch the layers we want for
a build, write the list of layers to bblayers.conf.sample and source
oe-init-build-env. With your setup, if I want a project with the two
layers meta-a and meta-b, I would also have to create a layer meta-x
where I create a static bblayers.conf.sample file that lists these
three layers. Then, if another project wants to use meta-a and meta-c,
they would have to add a second static bblayers.conf.sample file in
meta-x that lists those three layers. Basically every time there is a
new combination of layers fetched, there would also have to be a
duplicate of the information in a static bblayers.conf.sample file
somewhere. I am probably missing something here, but to me this seems
suboptimal. If the list of layers in the static file had been used to
fetch the layers, I would better understand it, but then the problem
becomes that the list is now in one of the arbitrary layers that are
supposed to be fetched.
For reference, today when one of our teams start a new development
project, what they do is they create a specific layer (e.g., meta-foo)
for that project where they can do all their development work and
experimentation until it is time for them to integrate to our main
platform (referred to as cfp/cfp.xml below). Then they create a repo
manifest that looks like:
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<include name="cfp/cfp.xml"/>
<project path="meta-foo" name="layers/projects/meta-foo" revision="master"/>
</manifest>
in our manifest repository. This simple manifest is all the project
team needs to create to build with their project specific layer. It
also means that the only input to our Jenkins jobs to build for this
project (instead of the main platform) is the name of the project's
repo manifest. It also means that if we make any changes to the main
platform, e.g., add a new layer (which is a one line change to the
cfp.xml manifest), all projects will automatically get and use that
new layer without having to make any changes to all the project
manifests.
>
> Alex
//Peter
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-15 1:07 ` Peter Kjellerstedt
@ 2022-09-15 9:20 ` Alexander Kanavin
2022-09-15 22:59 ` Peter Kjellerstedt
0 siblings, 1 reply; 21+ messages in thread
From: Alexander Kanavin @ 2022-09-15 9:20 UTC (permalink / raw)
To: Peter Kjellerstedt; +Cc: ChenQi, OE-core, Alexander Kanavin
On Thu, 15 Sept 2022 at 03:07, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> I know you have suggested this to me too as we use a similar setup to
> generate the bblayers.conf.sample file based on the layers that repo
> has checked out in our environment, and now I realized why this is not
> as simple as it seems. The problem (at least in our case) is that we
> run as a wrapper around oe-init-build-env. This means that before the
> wrapper sources oe-init-build-env, the build directory and thus the
> conf directory inside it (where the bblayers.conf file is supposed to
> go) may not exist. And after it has sourced oe-init-build-env it is
> too late as the bblayers.conf file needs to exists before. So to create
> the bblayers.conf file, the wrapper would have to duplicate all the
> non-trivial code in oe-setup-builddir that determines what the build
> directory should be called. And while this of course is possible to
> do, it means duplicating code that is internal to bitbake, and risking
> missing important changes to the upstream code in the future. Thus it
> is much easier and safer to generate the template files and let
> oe-init-build-env do its job.
Wait, I have to ask for details here. What is the non-trivial code
specifically? Isn't it simply
mkdir -p builddir/conf
write_out bblayers.conf
where builddir is the same argument that is passed to
oe-init-build-env? Something doesn't compute here for me.
> Where I (and apparently WindRiver) are coming from, the decision on what
> layers are part of the configuration is not something a specific layer
> is responsible for. I.e., in our setup we fetch the layers we want for
> a build, write the list of layers to bblayers.conf.sample and source
> oe-init-build-env. With your setup, if I want a project with the two
> layers meta-a and meta-b, I would also have to create a layer meta-x
> where I create a static bblayers.conf.sample file that lists these
> three layers. Then, if another project wants to use meta-a and meta-c,
> they would have to add a second static bblayers.conf.sample file in
> meta-x that lists those three layers. Basically every time there is a
> new combination of layers fetched, there would also have to be a
> duplicate of the information in a static bblayers.conf.sample file
> somewhere. I am probably missing something here, but to me this seems
> suboptimal. If the list of layers in the static file had been used to
> fetch the layers, I would better understand it, but then the problem
> becomes that the list is now in one of the arbitrary layers that are
> supposed to be fetched.
The idea is that yes, every possible combination of layers needs its
own template, and it goes under version control into a layer.
Typically this would be a layer that is common to all of your build
configurations (which is meta-a in the above example - *not* meta-x)
and that is where the templates go. Do you have such a layer?
I also have to ask, what prevents you from simply listing all of the
layers in a single unified bblayers.conf.sample. Do they step on each
other? How?
I also need to remind you that bblayers.conf(.sample) does support
includes, so you can implement a structured hierarchy of layer config
fragments, where the common bits are only written once.
Basically, there are alternatives, and I think all of them are better
than writing out the list of layers just before setting up a build
with a custom script.
> in our manifest repository. This simple manifest is all the project
> team needs to create to build with their project specific layer. It
> also means that the only input to our Jenkins jobs to build for this
> project (instead of the main platform) is the name of the project's
> repo manifest. It also means that if we make any changes to the main
> platform, e.g., add a new layer (which is a one line change to the
> cfp.xml manifest), all projects will automatically get and use that
> new layer without having to make any changes to all the project
> manifests.
This is somewhat tangential, as it is about extending the default
layer setup config, not the combinations of layers in the build
configuration. So far, the json is not extensible, but we can think
about how to extend and overlay it too. As far as development in
meta-foo goes, the templates (including layer lists) for it can be
hosted there, until the layer 'graduates' into the common structure.
Alex
^ permalink raw reply [flat|nested] 21+ messages in thread
* RE: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-15 9:20 ` Alexander Kanavin
@ 2022-09-15 22:59 ` Peter Kjellerstedt
2022-09-16 10:57 ` Alexander Kanavin
0 siblings, 1 reply; 21+ messages in thread
From: Peter Kjellerstedt @ 2022-09-15 22:59 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: ChenQi, OE-core, Alexander Kanavin
> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 15 september 2022 10:20
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: ChenQi <Qi.Chen@windriver.com>; OE-core <openembedded-
> core@lists.openembedded.org>; Alexander Kanavin <alex@linutronix.de>
> Subject: Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check
> that TEMPLATECONF is valid
>
> On Thu, 15 Sept 2022 at 03:07, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> > I know you have suggested this to me too as we use a similar setup to
> > generate the bblayers.conf.sample file based on the layers that repo
> > has checked out in our environment, and now I realized why this is not
> > as simple as it seems. The problem (at least in our case) is that we
> > run as a wrapper around oe-init-build-env. This means that before the
> > wrapper sources oe-init-build-env, the build directory and thus the
> > conf directory inside it (where the bblayers.conf file is supposed to
> > go) may not exist. And after it has sourced oe-init-build-env it is
> > too late as the bblayers.conf file needs to exists before. So to create
> > the bblayers.conf file, the wrapper would have to duplicate all the
> > non-trivial code in oe-setup-builddir that determines what the build
> > directory should be called. And while this of course is possible to
> > do, it means duplicating code that is internal to bitbake, and risking
> > missing important changes to the upstream code in the future. Thus it
> > is much easier and safer to generate the template files and let
> > oe-init-build-env do its job.
>
> Wait, I have to ask for details here. What is the non-trivial code
> specifically? Isn't it simply
>
> mkdir -p builddir/conf
> write_out bblayers.conf
>
> where builddir is the same argument that is passed to
> oe-init-build-env? Something doesn't compute here for me.
Since our code is a wrapper around oe-init-build-env, it would have to do
the exact same calculations as are done in oe-setup-builddir, and if you
look at that script, you will see that there are actually quite a lot of
options to determine the name of the build directory.
>
> > Where I (and apparently WindRiver) are coming from, the decision on what
> > layers are part of the configuration is not something a specific layer
> > is responsible for. I.e., in our setup we fetch the layers we want for
> > a build, write the list of layers to bblayers.conf.sample and source
> > oe-init-build-env. With your setup, if I want a project with the two
> > layers meta-a and meta-b, I would also have to create a layer meta-x
> > where I create a static bblayers.conf.sample file that lists these
> > three layers. Then, if another project wants to use meta-a and meta-c,
> > they would have to add a second static bblayers.conf.sample file in
> > meta-x that lists those three layers. Basically every time there is a
> > new combination of layers fetched, there would also have to be a
> > duplicate of the information in a static bblayers.conf.sample file
> > somewhere. I am probably missing something here, but to me this seems
> > suboptimal. If the list of layers in the static file had been used to
> > fetch the layers, I would better understand it, but then the problem
> > becomes that the list is now in one of the arbitrary layers that are
> > supposed to be fetched.
>
> The idea is that yes, every possible combination of layers needs its
> own template, and it goes under version control into a layer.
This seems incredibly inflexible if you have many optional layers.
We use repo's --groups option to handle optional layers. This means
we can, e.g., use one manifest to build the main firmware, but also
to build special firmware for production testing by adding an option
to repo, telling it to fetch an extra layer. This also means that all
the project manifests don't need to exist in two versions (one for
the main firmware and one for the production testing) as this is
handled by the base manifest.
With your solution, every project would have to create two templates
to cover this. And if we add another optional layer (we currently don't
have one, but we used to do), then we would have to add one or two
more templates and so on.
Also, if we decide to add a layer to the base platform, then all the
project templates are broken until someone updates them all. But since
they would be in the project layers, that is not something that I as
the maintainer of the base platform can do.
> Typically this would be a layer that is common to all of your build
> configurations (which is meta-a in the above example - *not* meta-x)
> and that is where the templates go. Do you have such a layer?
Well, we do for the base platform, but then the project templates would
have to be in the project layers.
I know you are currently not solving the problem of actually fetching
the layers, but I do not understand how you are planning to that when
the definition of what layers to use is inside some random layer. And
a solution that would fetch all layers would definitely not be usable
by us at least. Which means that to setup a random product I would
actually need to have inside knowledge of which layer actually happens
to have the list of what layers are needed for that product.
Or are you planning on having some definition of what layers to fetch
outside the layers (where I think it belongs)? Then why would one want
to have duplicate information in one of the fetched layers to setup
the BBLAYERS variable?
>
> I also have to ask, what prevents you from simply listing all of the
> layers in a single unified bblayers.conf.sample. Do they step on each
> other? How?
The project layers can definitely step on each other. We also have some
layers that modify the build completely by being included (they are
obviously not Yocto Compatible as there is no reason for them to be,
given that we only fetch and include the layers in our builds that are
actually used).
>
> I also need to remind you that bblayers.conf(.sample) does support
> includes, so you can implement a structured hierarchy of layer config
> fragments, where the common bits are only written once.
AFAIK, oe-init-build-env does not support that. It will only replace
##OEROOT## in the bblayers.conf.sample file.
>
> Basically, there are alternatives, and I think all of them are better
> than writing out the list of layers just before setting up a build
> with a custom script.
I cannot for the life of me see how your solution is better in any way
than the one we have. In our solution we have simple, hierarchical
manifests that specify what layers are to be used. Then the init scripts
automatically set up the bitbake environment to use those layers. All
the manifests are in one repository. Adding a layer to the platform means
adding one line to one manifest. Adding a new project means adding one
manifest (with basically one line specifying the project layer).
>
> > in our manifest repository. This simple manifest is all the project
> > team needs to create to build with their project specific layer. It
> > also means that the only input to our Jenkins jobs to build for this
> > project (instead of the main platform) is the name of the project's
> > repo manifest. It also means that if we make any changes to the main
> > platform, e.g., add a new layer (which is a one line change to the
> > cfp.xml manifest), all projects will automatically get and use that
> > new layer without having to make any changes to all the project
> > manifests.
>
> This is somewhat tangential, as it is about extending the default
> layer setup config, not the combinations of layers in the build
> configuration. So far, the json is not extensible, but we can think
> about how to extend and overlay it too. As far as development in
> meta-foo goes, the templates (including layer lists) for it can be
> hosted there, until the layer 'graduates' into the common structure.
>
> Alex
//Peter
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-15 22:59 ` Peter Kjellerstedt
@ 2022-09-16 10:57 ` Alexander Kanavin
2022-09-17 0:06 ` Peter Kjellerstedt
0 siblings, 1 reply; 21+ messages in thread
From: Alexander Kanavin @ 2022-09-16 10:57 UTC (permalink / raw)
To: Peter Kjellerstedt; +Cc: ChenQi, OE-core, Alexander Kanavin
On Fri, 16 Sept 2022 at 00:59, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
>
> Since our code is a wrapper around oe-init-build-env, it would have to do
> the exact same calculations as are done in oe-setup-builddir, and if you
> look at that script, you will see that there are actually quite a lot of
> options to determine the name of the build directory.
Line numbers as of current master HEAD please. I am looking at the
script right now and I am not seeing any options: it simply takes
$BUILDDIR var passed in from oe-init-build-env. And oe-init-build-env
sets that to either what was given on the command line, or defaults to
'build'.
Again, to the best of my knowledge this will work in the wrapper, and
you need to show otherwise:
mkdir -p $builddir/conf
write_out $builddir/conf/bblayers.conf
. oe-init-build-env $builddir
Something still doesn't compute here and we need to get to the bottom
of it. Let's focus on this for now, please.
Alex
^ permalink raw reply [flat|nested] 21+ messages in thread
* RE: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-16 10:57 ` Alexander Kanavin
@ 2022-09-17 0:06 ` Peter Kjellerstedt
2022-09-17 8:17 ` Alexander Kanavin
0 siblings, 1 reply; 21+ messages in thread
From: Peter Kjellerstedt @ 2022-09-17 0:06 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: ChenQi, OE-core, Alexander Kanavin
> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 16 september 2022 11:57
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: ChenQi <Qi.Chen@windriver.com>; OE-core <openembedded-
> core@lists.openembedded.org>; Alexander Kanavin <alex@linutronix.de>
> Subject: Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check
> that TEMPLATECONF is valid
>
> On Fri, 16 Sept 2022 at 00:59, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> >
> > Since our code is a wrapper around oe-init-build-env, it would have to do
> > the exact same calculations as are done in oe-setup-builddir, and if you
> > look at that script, you will see that there are actually quite a lot of
> > options to determine the name of the build directory.
>
> Line numbers as of current master HEAD please. I am looking at the
> script right now and I am not seeing any options: it simply takes
> $BUILDDIR var passed in from oe-init-build-env. And oe-init-build-env
> sets that to either what was given on the command line, or defaults to
> 'build'.
Sorry, my fault. I forgot that while the build directory is created
in oe-setup-builddir, the path that is used is actually determined in
oe-buildenv-internal, line 42-81. And this is the part that I would
have to duplicate. Now, I do not need all of it (it can be reduced to
19 lines), but it still means I am duplicating the logic from OE-Core
and if that changes, we may or may not notice...
That said, unless I can convince you that the right thing to do is to
remove templateconf.cfg for all builds if it does not contain a valid
path according to the new rules (and not only for OE-Core and Poky),
then I still need to duplicate the code to determine the build
directory to be able to remove the templateconf.cfg file to avoid
having to make all our developers do it manually...
>
> Again, to the best of my knowledge this will work in the wrapper, and
> you need to show otherwise:
>
> mkdir -p $builddir/conf
> write_out $builddir/conf/bblayers.conf
> . oe-init-build-env $builddir
>
> Something still doesn't compute here and we need to get to the bottom
> of it. Let's focus on this for now, please.
>
> Alex
//Peter
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid
2022-09-17 0:06 ` Peter Kjellerstedt
@ 2022-09-17 8:17 ` Alexander Kanavin
0 siblings, 0 replies; 21+ messages in thread
From: Alexander Kanavin @ 2022-09-17 8:17 UTC (permalink / raw)
To: Peter Kjellerstedt; +Cc: ChenQi, OE-core, Alexander Kanavin
On Sat, 17 Sept 2022 at 02:06, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> Sorry, my fault. I forgot that while the build directory is created
> in oe-setup-builddir, the path that is used is actually determined in
> oe-buildenv-internal, line 42-81. And this is the part that I would
> have to duplicate. Now, I do not need all of it (it can be reduced to
> 19 lines), but it still means I am duplicating the logic from OE-Core
> and if that changes, we may or may not notice...
Then you can do it in the opposite order through the official tooling:
. oe-init-build-env (against a template with a lowest common
denominator set of layers, perhaps even one from poky)
bitbake-layers add-layer /path/to/meta-b
bitbake-layers add-layer /path/to/meta-c
...
This is better, as it avoids having to write bblayers.conf[.template]
directly. You say you don't want to duplicate the script logic; you
should try to avoid duplicating the content of config files too then.
Alex
^ permalink raw reply [flat|nested] 21+ messages in thread