* [PATCH 1/5] bitbake-layers: add a command to save the active build configuration as a template into a layer @ 2022-08-17 13:10 Alexander Kanavin 2022-08-17 13:10 ` [PATCH 2/5] meta/files: add layer setup JSON schema and example Alexander Kanavin ` (3 more replies) 0 siblings, 4 replies; 18+ messages in thread From: Alexander Kanavin @ 2022-08-17 13:10 UTC (permalink / raw) To: openembedded-core; +Cc: Alexander Kanavin This is the reverse of setting up a build by pointing TEMPLATECONF to a directory with a template and running '. oe-init-build-env': this takes the config files from build/conf, replaces site-specific paths in bblayers.conf with ##OECORE##-relative paths, and copies the config files into a specified layer under a specified template name. In many or perhaps most cases such static prefabricated configurations (that require no further editing) are just enough, and I believe they should be offered by the official configuration management. On the other hand, generating build configurations with a sufficiently versatile tool is a far more complex problem, and one we should try to tackle once we see where and how static configs fall short. Tooling to discover and select these templates when setting up a build will be provided later on. How to use: alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers save-build-conf ../../meta-alex/ test-1 NOTE: Starting bitbake server... NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1 Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt You can try out the configuration with TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1 alex@Zen2:/srv/work/alex/poky/build-layersetup$ Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- meta/lib/bblayers/buildconf.py | 83 ++++++++++++++++++++++++ meta/lib/oeqa/selftest/cases/bblayers.py | 5 ++ 2 files changed, 88 insertions(+) create mode 100644 meta/lib/bblayers/buildconf.py diff --git a/meta/lib/bblayers/buildconf.py b/meta/lib/bblayers/buildconf.py new file mode 100644 index 0000000000..ece8836d5a --- /dev/null +++ b/meta/lib/bblayers/buildconf.py @@ -0,0 +1,83 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import stat +import sys +import shutil +import json + +import bb.utils +import bb.process + +from bblayers.common import LayerPlugin + +logger = logging.getLogger('bitbake-layers') + +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) + +import oe.buildcfg + +def plugin_init(plugins): + return BuildConfPlugin() + +class BuildConfPlugin(LayerPlugin): + notes_fixme = """FIXME: Please place here the description of this build configuration. +It will be shown to the users when they set up their builds via TEMPLATECONF. +""" + + def _save_conf(self, templatename, templatepath, oecorepath, relpaths_to_oecore): + confdir = os.path.join(os.environ["BBPATH"], "conf") + destdir = os.path.join(templatepath, "conf", "templates", templatename) + os.makedirs(destdir, exist_ok=True) + + with open(os.path.join(confdir, "local.conf")) as src: + with open(os.path.join(destdir, "local.conf.sample"), 'w') as dest: + dest.write(src.read()) + + with open(os.path.join(confdir, "bblayers.conf")) as src: + with open(os.path.join(destdir, "bblayers.conf.sample"), 'w') as dest: + bblayers_data = src.read() + + for (abspath, relpath) in relpaths_to_oecore: + bblayers_data = bblayers_data.replace(abspath, "##OEROOT##/" + relpath) + dest.write(bblayers_data) + + with open(os.path.join(destdir, "conf-notes.txt"), 'w') as dest: + dest.write(self.notes_fixme) + + logger.info("""Configuration template placed into {} +Please review the files in there, and particularly provide a configuration description in {} +You can try out the configuration with +TEMPLATECONF={} . {}/oe-init-build-env build-try-{}""" +.format(destdir, os.path.join(destdir, "conf-notes.txt"), destdir, oecorepath, templatename)) + + def do_save_build_conf(self, args): + """ Save the currently active build configuration (conf/local.conf, conf/bblayers.conf) as a template into a layer.\n This template can later be used for setting up builds via TEMPLATECONF. """ + repos = {} + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) + targetlayer = None + oecore = None + + for l in layers: + if l[0] == os.path.abspath(args.layerpath): + targetlayer = l[0] + if l[1] == 'meta': + oecore = os.path.dirname(l[0]) + + if not targetlayer: + logger.error("Layer {} not in one of the currently enabled layers:\n{}".format(args.layerpath, "\n".join([l[0] for l in layers]))) + elif not oecore: + logger.error("Openembedded-core not in one of the currently enabled layers:\n{}".format("\n".join([l[0] for l in layers]))) + else: + relpaths_to_oecore = [(l[0], os.path.relpath(l[0], start=oecore)) for l in layers] + self._save_conf(args.templatename, targetlayer, oecore, relpaths_to_oecore) + + def register_commands(self, sp): + parser_build_conf = self.add_command(sp, 'save-build-conf', self.do_save_build_conf, parserecipes=False) + parser_build_conf.add_argument('layerpath', + help='The path to the layer where the configuration template should be saved.') + parser_build_conf.add_argument('templatename', + help='The name of the configuration template.') diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 494fa892a3..549abe7d10 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -113,6 +113,11 @@ class BitbakeLayers(OESelftestTestCase): self.assertEqual(bb_vars['BBFILE_PRIORITY_%s' % layername], str(priority), 'BBFILE_PRIORITY_%s != %d' % (layername, priority)) + result = runCmd('bitbake-layers save-build-conf {} {}'.format(layerpath, "buildconf-1")) + for f in ('local.conf.sample', 'bblayers.conf.sample', 'conf-notes.txt'): + fullpath = os.path.join(layerpath, "conf", "templates", "buildconf-1", f) + self.assertTrue(os.path.exists(fullpath), "Template configuration file {} not found".format(fullpath)) + def get_recipe_basename(self, recipe): recipe_file = "" result = runCmd("bitbake-layers show-recipes -f %s" % recipe) -- 2.30.2 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 2/5] meta/files: add layer setup JSON schema and example 2022-08-17 13:10 [PATCH 1/5] bitbake-layers: add a command to save the active build configuration as a template into a layer Alexander Kanavin @ 2022-08-17 13:10 ` Alexander Kanavin 2022-08-17 20:52 ` [OE-core] " Richard Purdie 2022-08-17 13:10 ` [PATCH 3/5] bitbake-layers: add ability to save current layer repository configuration into a json file Alexander Kanavin ` (2 subsequent siblings) 3 siblings, 1 reply; 18+ messages in thread From: Alexander Kanavin @ 2022-08-17 13:10 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt, Alexander Kanavin From: Joshua Watt <JPEWhacker@gmail.com> Defines a common schema for layer setup that can be consumed by tools to know how to fetch and assemble layers for end users. Also includes an example of the layer setup that constructs poky/meta-intel/imaginary product layer for reference. The schema can be used to validate a layer setup file with the commands: $ python3 -m pip install jsonschema $ jsonschema -i meta/files/layers.example.json meta/files/layers.schema.json Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Alex: I made the following modifications to Joshua's original commit: - moved the files from meta/lib to meta/files - the example json showcases a multi-repo, multi-layer setup with additional configurations and machines, instead of just poky - closer to a typical product - added oe-selftest that validates the example json against the schema using python3-jsonschema-native - the schema is modified so that: -- all lists (sources, layers, remotes) are replaced by objects keyed by 'name' properties of the list items. This allows using them as dicts inside Python, and makes the json more compact and readable. -- added 'contains_this_file' property to source object -- added 'buildconfigs', 'machines' and 'distros' properties to layer objects. -- replaced 'remote' property with a 'oneOf' definition for git with a specific 'git-remote' property. 'oneOf' is problematic when schema validation fails: the diagnostic is only that none of oneOf variants matched, which is too non-specific. -- added 'describe' property to 'git-remote' object. -- removed description property for a layer source: it is not clear how to add that when auto-generating the json Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- meta/files/layers.example.json | 115 +++++++++++++++++++++ meta/files/layers.schema.json | 121 +++++++++++++++++++++++ meta/lib/oeqa/selftest/cases/bblayers.py | 16 ++- 3 files changed, 251 insertions(+), 1 deletion(-) create mode 100644 meta/files/layers.example.json create mode 100644 meta/files/layers.schema.json diff --git a/meta/files/layers.example.json b/meta/files/layers.example.json new file mode 100644 index 0000000000..b6e35c615d --- /dev/null +++ b/meta/files/layers.example.json @@ -0,0 +1,115 @@ +{ + "sources": { + "meta-alex": { + "contains_this_file": true, + "git-remote": { + "branch": "master", + "describe": "", + "remotes": { + "remote-alex": { + "uri": "https://github.com/kanavin/meta-alex" + } + }, + "rev": "05b25605fb8b2399e4706d7323828676bf0da0b5" + }, + "layers": { + "meta-alex": { + "buildconfigs": { + "conf/templates/configuration-gadget": {}, + "conf/templates/configuration-gizmo": {} + }, + "subpath": "" + } + }, + "path": "meta-alex" + }, + "meta-intel": { + "git-remote": { + "branch": "master", + "describe": "15.0-hardknott-3.3-310-g0a96edae", + "remotes": { + "origin": { + "uri": "git://git.yoctoproject.org/meta-intel" + } + }, + "rev": "0a96edae609a3f48befac36af82cf1eed6786b4a" + }, + "layers": { + "meta-intel": { + "machines": { + "intel-core2-32": {}, + "intel-corei7-64": {}, + "intel-skylake-64": {} + }, + "subpath": "" + } + }, + "path": "meta-intel" + }, + "poky": { + "git-remote": { + "branch": "akanavin/setup-layers", + "describe": "4.1_M1-374-g9dda719b2a", + "remotes": { + "origin": { + "uri": "git://git.yoctoproject.org/poky" + }, + "poky-contrib": { + "uri": "ssh://git@push.yoctoproject.org/poky-contrib" + } + }, + "rev": "9dda719b2a4727a4d43a6ab8d9e23f8ca68790ec" + }, + "layers": { + "meta": { + "distros": { + "defaultsetup": {} + }, + "machines": { + "qemuarm": {}, + "qemuarm64": {}, + "qemuarmv5": {}, + "qemumips": {}, + "qemumips64": {}, + "qemuppc": {}, + "qemuppc64": {}, + "qemuriscv32": {}, + "qemuriscv64": {}, + "qemux86": {}, + "qemux86-64": {} + }, + "subpath": "meta" + }, + "meta-poky": { + "buildconfigs": { + "conf": {} + }, + "distros": { + "poky": {}, + "poky-altcfg": {}, + "poky-bleeding": {}, + "poky-tiny": {} + }, + "subpath": "meta-poky" + }, + "meta-selftest": { + "machines": { + "qemux86copy": {} + }, + "subpath": "meta-selftest" + }, + "meta-yocto-bsp": { + "machines": { + "beaglebone-yocto": {}, + "edgerouter": {}, + "genericx86": {}, + "genericx86-64": {} + }, + "subpath": "meta-yocto-bsp" + } + }, + "path": "poky" + } + }, + "version": "1.0" +} diff --git a/meta/files/layers.schema.json b/meta/files/layers.schema.json new file mode 100644 index 0000000000..2603fae8f7 --- /dev/null +++ b/meta/files/layers.schema.json @@ -0,0 +1,121 @@ +{ + "description": "OpenEmbedder Layer Setup Manifest", + "type": "object", + "additionalProperties": false, + "required": [ + "version" + ], + "properties": { + "version": { + "description": "The version of this document; currently '1.0'", + "enum": ["1.0"] + }, + "sources": { + "description": "The dict of layer sources", + "type": "object", + "patternProperties": { ".*" : { + "type": "object", + "description": "The upstream source from which a set of layers may be fetched", + "additionalProperties": false, + "required": [ + "path" + ], + "properties": { + "path": { + "description": "The path where this layer source will be placed when fetching", + "type": "string" + }, + "contains_this_file": { + "description": "Whether the directory with the layer source also contains this json description. Tools may want to skip the checkout of the source then.", + "type": "boolean" + }, + "layers": { + "description": "The dict of layers to be used from this upstream source", + "type": "object", + "patternProperties": { ".*" : { + "description": "A layer from the upstream source", + "type": "object", + "additionalProperties": false, + "properties": { + "subpath": { + "description": "The subpath (relative to the source root) for this layer. Omit if the source root is the layer path", + "type": "string" + }, + "buildconfigs": { + "description": "A dict of template build configurations in the layer, keyed by relative path to the layer root", + "type": "object", + "patternProperties": { ".*" : { + "description": "Properties of a build confuguration template", + "type": "object", + "additionalProperties": false, + "properties":{} + }} + }, + "machines": { + "description": "A dict of machine definitions in the layer", + "type": "object", + "patternProperties": { ".*" : { + "description": "Properties of a machine definition", + "type": "object", + "additionalProperties": false, + "properties":{} + }} + }, + "distros": { + "description": "A dict of distro definitions in the layer", + "type": "object", + "patternProperties": { ".*" : { + "description": "Properties of a distro definition", + "type": "object", + "additionalProperties": false, + "properties":{} + }} + } + } + }} + }, + "git-remote": { + "description": "A remote git source from which to fetch", + "type": "object", + "additionalProperties": false, + "required": [ + "rev" + ], + "properties": { + "branch": { + "description": "The git branch to fetch (optional)", + "type": "string" + }, + "rev": { + "description": "The git revision to checkout", + "type": "string" + }, + "describe": { + "description": "The output of 'git describe' (human readable description of the revision using tags in revision history).", + "type": "string" + }, + "remotes": { + "description": "The dict of git remotes to add to this repository", + "type": "object", + "patternProperties": { ".*" : { + "description": "A git remote", + "type": "object", + "addtionalProperties": false, + "required": [ + "uri" + ], + "properties": { + "uri": { + "description": "The URI for the remote", + "type": "string" + } + } + }} + } + } + } + } + } + }} + } +} diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index 549abe7d10..c753a7b795 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -8,12 +8,16 @@ import os import re import oeqa.utils.ftools as ftools -from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars +from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars, bitbake from oeqa.selftest.case import OESelftestTestCase class BitbakeLayers(OESelftestTestCase): + def setUpLocal(self): + bitbake("python3-jsonschema-native") + bitbake("-c addto_recipe_sysroot python3-jsonschema-native") + def test_bitbakelayers_layerindexshowdepends(self): result = runCmd('bitbake-layers layerindex-show-depends meta-poky') find_in_contents = re.search("openembedded-core", result.output) @@ -128,3 +132,13 @@ class BitbakeLayers(OESelftestTestCase): self.assertTrue(os.path.isfile(recipe_file), msg = "Can't find recipe file for %s" % recipe) return os.path.basename(recipe_file) + + def validate_layersjson(self, json): + python = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-jsonschema-native'), 'nativepython3') + jsonvalidator = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-jsonschema-native'), 'jsonschema') + jsonschema = os.path.join(get_bb_var('COREBASE'), 'meta/files/layers.schema.json') + result = runCmd("{} {} -i {} {}".format(python, jsonvalidator, json, jsonschema)) + + def test_validate_examplelayersjson(self): + json = os.path.join(get_bb_var('COREBASE'), "meta/files/layers.example.json") + self.validate_layersjson(json) -- 2.30.2 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [OE-core] [PATCH 2/5] meta/files: add layer setup JSON schema and example 2022-08-17 13:10 ` [PATCH 2/5] meta/files: add layer setup JSON schema and example Alexander Kanavin @ 2022-08-17 20:52 ` Richard Purdie 2022-08-17 21:36 ` Alexander Kanavin 2022-08-17 22:16 ` [OE-core] [PATCH 2/5] meta/files: add layer setup JSON schema and example Alexander Kanavin 0 siblings, 2 replies; 18+ messages in thread From: Richard Purdie @ 2022-08-17 20:52 UTC (permalink / raw) To: Alexander Kanavin, openembedded-core; +Cc: Joshua Watt, Alexander Kanavin On Wed, 2022-08-17 at 15:10 +0200, Alexander Kanavin wrote: > From: Joshua Watt <JPEWhacker@gmail.com> > > Defines a common schema for layer setup that can be consumed by tools to > know how to fetch and assemble layers for end users. Also includes an > example of the layer setup that constructs poky/meta-intel/imaginary product layer > for reference. > > The schema can be used to validate a layer setup file with the commands: > > $ python3 -m pip install jsonschema > $ jsonschema -i meta/files/layers.example.json meta/files/layers.schema.json > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > Alex: I made the following modifications to Joshua's original commit: > > - moved the files from meta/lib to meta/files > > - the example json showcases a multi-repo, multi-layer setup with additional > configurations and machines, instead of just poky - closer to a typical product > > - added oe-selftest that validates the example json against the schema using python3-jsonschema-native > > - the schema is modified so that: > > -- all lists (sources, layers, remotes) are replaced by objects keyed by 'name' properties of the list items. > This allows using them as dicts inside Python, and makes the json more compact and readable. > > -- added 'contains_this_file' property to source object > > -- added 'buildconfigs', 'machines' and 'distros' properties to layer objects. > > -- replaced 'remote' property with a 'oneOf' definition for git with a specific > 'git-remote' property. 'oneOf' is problematic when schema validation fails: > the diagnostic is only that none of oneOf variants matched, which is too non-specific. > > -- added 'describe' property to 'git-remote' object. > > -- removed description property for a layer source: it is not clear how to add that > when auto-generating the json I finally got some time to think about this a little bit more. One thing that concerns me a little is that this mixes two things, layer setup (as in repos) and configuration. I'm nervous about json config which effectively has to duplicate the list of machines/distros in layers. Where is the distro/machine data used? Are these the only config options people will want to add or will we have others? init system? libc? Or feature information about which configurations are expected to work? I worry this is a magnet for future feature creep and duplication of information. I can see where the buildconfigs are used but again, does the json file need to encode those or could you determine them by inspection of the layers once setup? I'm still worried about bolting a format directly into one of our core tools as it effectively means it is a definitive standard. We do probably need one but I'm not convinced this is quite right, perhaps because of the above reason. Not 100% sure but something doesn't feel quite right. Also, if there is new version of this series, could you squash in these copyright/license tweaks please: https://git.yoctoproject.org/poky/commit/?h=master-next&id=45b396298c1dd638bb702f5251b4a663f07978ca Cheers, Richard ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [OE-core] [PATCH 2/5] meta/files: add layer setup JSON schema and example 2022-08-17 20:52 ` [OE-core] " Richard Purdie @ 2022-08-17 21:36 ` Alexander Kanavin 2022-08-17 22:27 ` Richard Purdie 2022-08-17 22:16 ` [OE-core] [PATCH 2/5] meta/files: add layer setup JSON schema and example Alexander Kanavin 1 sibling, 1 reply; 18+ messages in thread From: Alexander Kanavin @ 2022-08-17 21:36 UTC (permalink / raw) To: Richard Purdie; +Cc: OE-core, Joshua Watt, Alexander Kanavin On Wed, 17 Aug 2022 at 22:52, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > One thing that concerns me a little is that this mixes two things, > layer setup (as in repos) and configuration. I'm nervous about json > config which effectively has to duplicate the list of machines/distros > in layers. > > Where is the distro/machine data used? The use case I was thinking of is doing something useful with this information *before* doing actual checkout. I would even appreciate knowing in advance - maybe even by just looking at the json through gitweb ui - which machines, distros and config templates I would be getting, and from which layers. Also the proposed oe-setup tool could do useful things with this information, if it operates from the json definitions in the same format. You can discover and supply all needed information up front, and oe-setup will land you in a bitbake session. Without it, you have to first do the layer setup, then the configuration discovery, then the configuration setup, as separate commands - I'd say that would be an inferior experience for the users. > Are these the only config options people will want to add or will we > have others? init system? libc? Or feature information about which > configurations are expected to work? I worry this is a magnet for > future feature creep and duplication of information. These are not the same, as machines and distros are specifically defined in layers as .conf files in pre-determined locations. Other things are defined as variables in files, and aren't as easily discoverable by code. I only want to allow programmatically discoverable options (that are guaranteed to be in lockstep with the layer revision) in the json; no one should ever write or modify the json by hand. > I can see where the buildconfigs are used but again, does the json file > need to encode those or could you determine them by inspection of the > layers once setup? In addition to the above point (knowing them in advance is useful), there is no fixed location for these in layers. You could perhaps walk the layer trees, but I'd rather place the info in the json. > Also, if there is new version of this series, could you squash in these > copyright/license tweaks please: > > https://git.yoctoproject.org/poky/commit/?h=master-next&id=45b396298c1dd638bb702f5251b4a663f07978ca This is the easy part :-) Hope the above clarifies a bit what went on in my head when I was writing the code. Alex ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [OE-core] [PATCH 2/5] meta/files: add layer setup JSON schema and example 2022-08-17 21:36 ` Alexander Kanavin @ 2022-08-17 22:27 ` Richard Purdie 2022-08-18 8:24 ` Alexander Kanavin 0 siblings, 1 reply; 18+ messages in thread From: Richard Purdie @ 2022-08-17 22:27 UTC (permalink / raw) To: Alexander Kanavin; +Cc: OE-core, Joshua Watt, Alexander Kanavin On Wed, 2022-08-17 at 23:36 +0200, Alexander Kanavin wrote: > On Wed, 17 Aug 2022 at 22:52, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > One thing that concerns me a little is that this mixes two things, > > layer setup (as in repos) and configuration. I'm nervous about json > > config which effectively has to duplicate the list of machines/distros > > in layers. > > > > Where is the distro/machine data used? > > The use case I was thinking of is doing something useful with this > information *before* doing actual checkout. I would even appreciate > knowing in advance - maybe even by just looking at the json through > gitweb ui - which machines, distros and config templates I would be > getting, and from which layers. The trouble is that others already went a different direction with this which is why we have layer index and the concept of local layer indexes as well we the ability for bblayers to talk to such indexes if I'm remembering correctly. An advantage it has is that it does parse the layers so it can cache other information too. My worry is that you have two copies of this information, one in the json and one in the metadata and effectively by that fact, one of them will be wrong. You could report it to the user with a directory listing of the remote repo via git. > Also the proposed oe-setup tool could > do useful things with this information, if it operates from the json > definitions in the same format. You can discover and supply all needed > information up front, and oe-setup will land you in a bitbake session. > Without it, you have to first do the layer setup, then the > configuration discovery, then the configuration setup, as separate > commands - I'd say that would be an inferior experience for the users. I think you can get a lot of this information relatively easily and possibly without a clone, via a layer index or probably through a remote git command too. > > Are these the only config options people will want to add or will we > > have others? init system? libc? Or feature information about which > > configurations are expected to work? I worry this is a magnet for > > future feature creep and duplication of information. > > These are not the same, as machines and distros are specifically > defined in layers as .conf files in pre-determined locations. Other > things are defined as variables in files, and aren't as easily > discoverable by code. I only want to allow programmatically > discoverable options (that are guaranteed to be in lockstep with the > layer revision) in the json; no one should ever write or modify the > json by hand. I'm not sure that will scale for what users want then (which I suspect is why we ended up with the layer index instead). > > I can see where the buildconfigs are used but again, does the json file > > need to encode those or could you determine them by inspection of the > > layers once setup? > > In addition to the above point (knowing them in advance is useful), > there is no fixed location for these in layers. You could perhaps walk > the layer trees, but I'd rather place the info in the json. I have wondered if we should define one. I kind of have a plan in mind to turn the majority of the autobuilder configurations into config files somewhere in OE-Core to make it easier for people to reproduce autobuilder configs too, which may be an interesting example of that. > > Also, if there is new version of this series, could you squash in these > > copyright/license tweaks please: > > > > https://git.yoctoproject.org/poky/commit/?h=master-next&id=45b396298c1dd638bb702f5251b4a663f07978ca > > This is the easy part :-) Hope the above clarifies a bit what went on > in my head when I was writing the code. It does, thanks. I'm just not sure we're quite in the right place with things. I know in your other reply you say we have to start somewhere and we can change things. I also know the screaming if we try and remove something later... Cheers, Richard ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [OE-core] [PATCH 2/5] meta/files: add layer setup JSON schema and example 2022-08-17 22:27 ` Richard Purdie @ 2022-08-18 8:24 ` Alexander Kanavin 2022-08-18 9:27 ` Richard Purdie 0 siblings, 1 reply; 18+ messages in thread From: Alexander Kanavin @ 2022-08-18 8:24 UTC (permalink / raw) To: Richard Purdie; +Cc: OE-core, Joshua Watt, Alexander Kanavin On Thu, 18 Aug 2022 at 00:27, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > It does, thanks. I'm just not sure we're quite in the right place with > things. I know in your other reply you say we have to start somewhere > and we can change things. I also know the screaming if we try and > remove something later... Right, I'm going to make the following changes: - machine/distro/config-template stuff gets dropped from the schema and tools - I am convinced now that it can be discovered through other means :-) - config template location gets standardized (perhaps in conf/templates/), default template in oe-core moves to conf/templates/default/ - add a plugin abstraction for the layer setup writers, move the json writer to a plugin Anything else? Alex ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [OE-core] [PATCH 2/5] meta/files: add layer setup JSON schema and example 2022-08-18 8:24 ` Alexander Kanavin @ 2022-08-18 9:27 ` Richard Purdie 2022-09-01 15:29 ` Configuration fragments Alexander Kanavin 0 siblings, 1 reply; 18+ messages in thread From: Richard Purdie @ 2022-08-18 9:27 UTC (permalink / raw) To: Alexander Kanavin; +Cc: OE-core, Joshua Watt, Alexander Kanavin On Thu, 2022-08-18 at 10:24 +0200, Alexander Kanavin wrote: > On Thu, 18 Aug 2022 at 00:27, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > It does, thanks. I'm just not sure we're quite in the right place with > > things. I know in your other reply you say we have to start somewhere > > and we can change things. I also know the screaming if we try and > > remove something later... > > Right, I'm going to make the following changes: > - machine/distro/config-template stuff gets dropped from the schema > and tools - I am convinced now that it can be discovered through other > means :-) > - config template location gets standardized (perhaps in > conf/templates/), default template in oe-core moves to > conf/templates/default/ > - add a plugin abstraction for the layer setup writers, move the json > writer to a plugin > > Anything else? I think that is a good plan and lets us move forward. If it turns out we need to extend the json, we can. Incrementally extending if/as needed is the safer path forward. That would just leave the details of the template layout. I think what you have above is probably fine. I'm just wondering how it fits with something else I've been worrying about, e.g. how to add config fragments to be used on the autobuilder. The intent is the user could add something like: require conf/yocto-autobuilder/x32.inc or require conf/yocto-autobuilder/multilib-mipsn32.inc and similar to their local.conf and more easily replicate configurations. With those, I could add to OE-Core or meta-yocto. I'm leaning towards core, just so people can see the testing configs more easily but I know people don't like yocto mentions in core and whether or not it is too higher a level in conf, not sure. Anyway, I think it is tangential to what you're working on which is good and stops me worrying. I left this in here just to write it out somewhere. I would like to see if we can switch the eSDK to use the json format btw. I suspect that may be one of your next steps? :) Cheers, Richard ^ permalink raw reply [flat|nested] 18+ messages in thread
* Configuration fragments 2022-08-18 9:27 ` Richard Purdie @ 2022-09-01 15:29 ` Alexander Kanavin 2022-09-01 22:47 ` [Openembedded-architecture] " Mark Hatle 0 siblings, 1 reply; 18+ messages in thread From: Alexander Kanavin @ 2022-09-01 15:29 UTC (permalink / raw) To: Richard Purdie Cc: OE-core, Joshua Watt, Alexander Kanavin, openembedded-architecture On Thu, 18 Aug 2022 at 11:27, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > The intent is the user could add something like: > > require conf/yocto-autobuilder/x32.inc > > or > > require conf/yocto-autobuilder/multilib-mipsn32.inc > > and similar to their local.conf and more easily replicate > configurations. This could be a subset of a broader feature, one that allows adding and removing pre-canned 'config fragments' to local.conf. The proposal is to have such fragments live in meta-somelayer/conf/fragments/path/to/some/fragment (I think allowing multiple levels is beneficial here), and each fragment would start with a comment explaining what it does followed by the actual settings. Then adding fragments can be done by appending them to local.conf, and removing by looking for markers in local.conf and editing it out, which probably could use a helper tool/subcommand. Layers could also declare 'non-optional' fragments which get included automatically by the fact of including the layer into the build, which means layer.conf can go back to being short and sweet and to the point. Thoughts? > I would like to see if we can switch the eSDK to use the json format > btw. I suspect that may be one of your next steps? :) I don't think I understand this, can you explain? Alex ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [Openembedded-architecture] Configuration fragments 2022-09-01 15:29 ` Configuration fragments Alexander Kanavin @ 2022-09-01 22:47 ` Mark Hatle 0 siblings, 0 replies; 18+ messages in thread From: Mark Hatle @ 2022-09-01 22:47 UTC (permalink / raw) To: Alexander Kanavin, Richard Purdie Cc: OE-core, Joshua Watt, Alexander Kanavin, openembedded-architecture Wind River already has a mechanism to do something like this, called templates. https://github.com/WindRiver-Labs/wr-template/tree/WRLINUX_10_21_BASE Each template can have (optionally): # README - template README file # require - list of other templates required for this one # template.conf - template configuration fragment # image.inc - image fragment # bsp-pkgs.conf - BSP specific configuration fragment # bsp-pkgs.inc - BSP specific image fragment # The 'bsp-pkgs' files can only be in a template in a layer that provides a # specific conf/machine/${MACHINE}.conf file and layers it may contain, # otherwise they will be ignored I'm not saying what is implemented is a perfect solution, but it made a 'configuration fragment' approach to system configuration much easier when I worked with it. (For examples see: https://github.com/WindRiver-Labs/wrlinux/tree/WRLINUX_10_21_BASE/templates/feature) You could do things (in your local.conf, or machine.conf or distro.conf) like: WRTEMPLATE += "dpdk" This would add the dpdk recipes to your image and also configure the kernel for dpdk. https://github.com/WindRiver-Labs/wrlinux/tree/WRLINUX_10_21_BASE/templates/feature/dpdk --Mark On 9/1/22 10:29 AM, Alexander Kanavin wrote: > On Thu, 18 Aug 2022 at 11:27, Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: >> The intent is the user could add something like: >> >> require conf/yocto-autobuilder/x32.inc >> >> or >> >> require conf/yocto-autobuilder/multilib-mipsn32.inc >> >> and similar to their local.conf and more easily replicate >> configurations. > > This could be a subset of a broader feature, one that allows adding > and removing pre-canned 'config fragments' to local.conf. The proposal > is to have such fragments live in > meta-somelayer/conf/fragments/path/to/some/fragment (I think allowing > multiple levels is beneficial here), and each fragment would start > with a comment explaining what it does followed by the actual > settings. > > Then adding fragments can be done by appending them to local.conf, and > removing by looking for markers in local.conf and editing it out, > which probably could use a helper tool/subcommand. > > Layers could also declare 'non-optional' fragments which get included > automatically by the fact of including the layer into the build, which > means layer.conf can go back to being short and sweet and to the > point. > > Thoughts? > >> I would like to see if we can switch the eSDK to use the json format >> btw. I suspect that may be one of your next steps? :) > > I don't think I understand this, can you explain? > > Alex > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#1627): https://lists.openembedded.org/g/openembedded-architecture/message/1627 > Mute This Topic: https://lists.openembedded.org/mt/93398201/3616948 > Group Owner: openembedded-architecture+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [mark.hatle@kernel.crashing.org] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [OE-core] [PATCH 2/5] meta/files: add layer setup JSON schema and example 2022-08-17 20:52 ` [OE-core] " Richard Purdie 2022-08-17 21:36 ` Alexander Kanavin @ 2022-08-17 22:16 ` Alexander Kanavin 1 sibling, 0 replies; 18+ messages in thread From: Alexander Kanavin @ 2022-08-17 22:16 UTC (permalink / raw) To: Richard Purdie; +Cc: OE-core, Joshua Watt, Alexander Kanavin On Wed, 17 Aug 2022 at 22:52, Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > I'm still worried about bolting a format directly into one of our core > tools as it effectively means it is a definitive standard. We do > probably need one but I'm not convinced this is quite right, perhaps > because of the above reason. Not 100% sure but something doesn't feel > quite right. This part can perhaps be addressed by the venerable 'adding an abstraction layer', in the shape of config format generator plugins. I can supply the native json plugin, others can add kas, repo, or whatever else they find useful. Also, the json is versioned and almost nothing in it is required. If some ideas don't work out, or are found to cause problems, there are ways to transition out of problematic bits - perhaps even as simple as dropping the bad ideas from the generator, or the setup script, without modifying the schema. We do need a starting point though. Alex ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 3/5] bitbake-layers: add ability to save current layer repository configuration into a json file 2022-08-17 13:10 [PATCH 1/5] bitbake-layers: add a command to save the active build configuration as a template into a layer Alexander Kanavin 2022-08-17 13:10 ` [PATCH 2/5] meta/files: add layer setup JSON schema and example Alexander Kanavin @ 2022-08-17 13:10 ` Alexander Kanavin 2022-08-17 13:10 ` [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from " Alexander Kanavin 2022-08-17 13:10 ` [PATCH 5/5] selftest/bblayers: add a test for creating a layer setup and using it to restore the layers Alexander Kanavin 3 siblings, 0 replies; 18+ messages in thread From: Alexander Kanavin @ 2022-08-17 13:10 UTC (permalink / raw) To: openembedded-core; +Cc: Alexander Kanavin This addresses a long standing gap in the core offering: there is no tooling to capture the currently configured layers with their revisions, or restore the layers from a configuration file (without using external tools, some of which aren't particularly suitable for the task). This plugin addresses the 'capture' part. How to save a layer configuration: a) Running with default choices: NOTE: Starting bitbake server... NOTE: Created /srv/work/alex/meta-alex/setup-layers.json NOTE: Created /srv/work/alex/meta-alex/setup-layers b) Command line options: alex@Zen2:/srv/work/alex/poky/build-layersetup$ bitbake-layers create-layers-setup -h NOTE: Starting bitbake server... usage: bitbake-layers create-layers-setup [-h] [--output-prefix OUTPUT_PREFIX] [--json-only] destdir Writes out a python script and a json config that replicate the directory structure and revisions of the layers in a current build. positional arguments: destdir Directory where to write the output (if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script). optional arguments: -h, --help show this help message and exit --output-prefix OUTPUT_PREFIX, -o OUTPUT_PREFIX File name prefix for the output files, if the default (setup-layers) is undesirable. --json-only Write only the layer configuruation in json format. Otherwise, also a copy of poky/scripts/oe-setup-layers is provided, which is a self contained python script that fetches all the needed layers and sets them to correct revisions using the data from the json. Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- meta/lib/bblayers/makesetup.py | 139 +++++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 meta/lib/bblayers/makesetup.py diff --git a/meta/lib/bblayers/makesetup.py b/meta/lib/bblayers/makesetup.py new file mode 100644 index 0000000000..b7f62900f4 --- /dev/null +++ b/meta/lib/bblayers/makesetup.py @@ -0,0 +1,139 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import logging +import os +import stat +import sys +import shutil +import json + +import bb.utils +import bb.process + +from bblayers.common import LayerPlugin + +logger = logging.getLogger('bitbake-layers') + +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) + +import oe.buildcfg + +def plugin_init(plugins): + return MakeSetupPlugin() + +class MakeSetupPlugin(LayerPlugin): + + def _write_python(self, input, output): + with open(input) as f: + script = f.read() + with open(output, 'w') as f: + f.write(script) + st = os.stat(output) + os.chmod(output, st.st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) + + def _write_json(self, repos, output): + with open(output, 'w') as f: + json.dump(repos, f, sort_keys=True, indent=4) + + def _get_repo_path(self, layer_path): + repo_path, _ = bb.process.run('git rev-parse --show-toplevel', cwd=layer_path) + return repo_path.strip() + + def _get_remotes(self, repo_path): + remotes = {} + remotes_list,_ = bb.process.run('git remote', cwd=repo_path) + for r in remotes_list.split(): + uri,_ = bb.process.run('git remote get-url {r}'.format(r=r), cwd=repo_path) + remotes[r] = {'uri':uri.strip()} + return remotes + + def _get_describe(self, repo_path): + try: + describe,_ = bb.process.run('git describe --tags', cwd=repo_path) + except bb.process.ExecutionError: + return "" + return describe.strip() + + def _get_confs(self, conf_path): + try: + files = os.listdir(conf_path) + except: + return [] + return {f.replace(".conf",""):{} for f in files if f.endswith(".conf")} + + def _get_distros(self, layer_path): + return self._get_confs(os.path.join(layer_path, "conf/distro")) + + def _get_machines(self, layer_path): + return self._get_confs(os.path.join(layer_path, "conf/machine")) + + def _get_buildconfigs(self, layerpath): + return {os.path.relpath(dir, start=layerpath):{} for (dir, subdirs, files) in os.walk(layerpath) if 'local.conf.sample' in files and 'bblayers.conf.sample' in files} + + def _make_repo_config(self, destdir): + repos = {} + layers = oe.buildcfg.get_layer_revisions(self.tinfoil.config_data) + try: + destdir_repo = self._get_repo_path(destdir) + except bb.process.ExecutionError: + destdir_repo = None + + for (l_path, l_name, l_branch, l_rev, l_ismodified) in layers: + if l_name == 'workspace': + continue + if l_ismodified: + logger.error("Layer {name} in {path} has uncommitted modifications or is not in a git repository.".format(name=l_name,path=l_path)) + return + repo_path = self._get_repo_path(l_path) + if repo_path not in repos.keys(): + repos[repo_path] = {'path':os.path.basename(repo_path),'layers':{},'git-remote':{'rev':l_rev, 'branch':l_branch, 'remotes':self._get_remotes(repo_path), 'describe':self._get_describe(repo_path)}} + if repo_path == destdir_repo: + repos[repo_path]['contains_this_file'] = True + if not repos[repo_path]['git-remote']['remotes'] and not repos[repo_path]['contains_this_file']: + logger.error("Layer repository in {path} does not have any remotes configured. Please add at least one with 'git remote add'.".format(path=repo_path)) + return + repos[repo_path]['layers'][l_name] = {'subpath':l_path.replace(repo_path,'')[1:]} + distros = self._get_distros(l_path) + machines = self._get_machines(l_path) + buildconfigs = self._get_buildconfigs(l_path) + if distros: + repos[repo_path]['layers'][l_name]['distros'] = distros + if machines: + repos[repo_path]['layers'][l_name]['machines'] = machines + if buildconfigs: + repos[repo_path]['layers'][l_name]['buildconfigs'] = buildconfigs + + top_path = os.path.commonpath([os.path.dirname(r) for r in repos.keys()]) + + repos_nopaths = {} + for r in repos.keys(): + r_nopath = os.path.basename(r) + repos_nopaths[r_nopath] = repos[r] + r_relpath = os.path.relpath(r, top_path) + repos_nopaths[r_nopath]['path'] = r_relpath + return repos_nopaths + + def do_make_setup(self, args): + """ Writes out a python script and a json config that replicate the directory structure and revisions of the layers in a current build. """ + repos = self._make_repo_config(args.destdir) + json = {"version":"1.0","sources":repos} + if not repos: + raise Exception("Could not determine layer sources") + output = args.output_prefix or "setup-layers" + output = os.path.join(os.path.abspath(args.destdir),output) + self._write_json(json, output + ".json") + logger.info('Created {}.json'.format(output)) + if not args.json_only: + self._write_python(os.path.join(os.path.dirname(__file__),'../../../scripts/oe-setup-layers'), output) + logger.info('Created {}'.format(output)) + + def register_commands(self, sp): + parser_setup_layers = self.add_command(sp, 'create-layers-setup', self.do_make_setup, parserecipes=False) + parser_setup_layers.add_argument('destdir', + help='Directory where to write the output\n(if it is inside one of the layers, the layer becomes a bootstrap repository and thus will be excluded from fetching by the script).') + parser_setup_layers.add_argument('--output-prefix', '-o', + help='File name prefix for the output files, if the default (setup-layers) is undesirable.') + parser_setup_layers.add_argument('--json-only', action='store_true', + help='Write only the layer configuruation in json format. Otherwise, also a copy of poky/scripts/oe-setup-layers is provided, which is a self contained python script that fetches all the needed layers and sets them to correct revisions using the data from the json.') -- 2.30.2 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file 2022-08-17 13:10 [PATCH 1/5] bitbake-layers: add a command to save the active build configuration as a template into a layer Alexander Kanavin 2022-08-17 13:10 ` [PATCH 2/5] meta/files: add layer setup JSON schema and example Alexander Kanavin 2022-08-17 13:10 ` [PATCH 3/5] bitbake-layers: add ability to save current layer repository configuration into a json file Alexander Kanavin @ 2022-08-17 13:10 ` Alexander Kanavin 2022-08-21 22:04 ` [OE-core] " Philip Balister 2022-08-22 1:20 ` Chuck Wolber 2022-08-17 13:10 ` [PATCH 5/5] selftest/bblayers: add a test for creating a layer setup and using it to restore the layers Alexander Kanavin 3 siblings, 2 replies; 18+ messages in thread From: Alexander Kanavin @ 2022-08-17 13:10 UTC (permalink / raw) To: openembedded-core; +Cc: Alexander Kanavin This script can be used directly from poky, or can be copied directly into a layer or any other repository - it is self-suffucient and requires only python3 and git on the host where it will run. It is also copied by the bitbake-layers layers-setup plugin together with the json, unless requested otherwise. 1. How to restore the layers from the saved configuration: a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. (use 'bitbake-layers create-layer-setup' from the previous commit to create them) b) Running with default options: (note: this will work to update an existing checkout as well) alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override. Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master Running 'git init -q /srv/work/alex/my-build/meta-intel' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers Running 'git init -q /srv/work/alex/my-build/poky' Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky Available build configurations: /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gadget /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gizmo /srv/work/alex/my-build/poky/meta-poky/conf You can set up a build with: TEMPLATECONF=one/of/the/above . /srv/work/alex/my-build/poky/oe-init-build-env 2. Command line options: alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA] A self contained python script that fetches all the needed layers and sets them to correct revisions optional arguments: -h, --help show this help message and exit --force-bootstraplayer-checkout Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place). --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build). --jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json). Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- scripts/oe-setup-layers | 89 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100755 scripts/oe-setup-layers diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers new file mode 100755 index 0000000000..fe948898b0 --- /dev/null +++ b/scripts/oe-setup-layers @@ -0,0 +1,89 @@ +#!/usr/bin/env python3 +# +# This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running +# +# bitbake-layers create-layers-setup destdir +# +# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy. +# + +import argparse +import json +import os +import subprocess + +def _do_checkout(args, json): + layers = json['sources'] + buildconfs = [] + oecorepath = "" + for l_name in layers: + l_data = layers[l_name] + layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path'])) + + for ll_name in l_data['layers']: + if ll_name == 'meta': + oecorepath = layerdir + ll_data = l_data['layers'][ll_name] + if 'buildconfigs' in ll_data: + for c in ll_data['buildconfigs']: + buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c)) + + if 'contains_this_file' in l_data.keys(): + force_arg = 'force_bootstraplayer_checkout' + if not args[force_arg]: + print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout')) + continue + l_remote = l_data['git-remote'] + rev = l_remote['rev'] + desc = l_remote['describe'] + if not desc: + desc = rev[:10] + branch = l_remote['branch'] + remotes = l_remote['remotes'] + + print('\nSetting up source {}, revision {}, branch {}'.format(l_name, desc, branch)) + cmd = 'git init -q {}'.format(layerdir) + print("Running '{}'".format(cmd)) + subprocess.check_output(cmd, shell=True) + + for remote in remotes: + cmd = "git remote remove {} > /dev/null 2>&1; git remote add {} {}".format(remote, remote, remotes[remote]['uri']) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, shell=True, cwd=layerdir) + + cmd = "git fetch -q {} || true".format(remote) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, shell=True, cwd=layerdir) + + cmd = 'git checkout -q {}'.format(rev) + print("Running '{}' in {}".format(cmd, layerdir)) + subprocess.check_output(cmd, shell=True, cwd=layerdir) + + + print("\nAvailable build configurations:\n{}\n".format("\n".join(buildconfs))) + print("You can set up a build with:\nTEMPLATECONF=one/of/the/above . {}/oe-init-build-env".format(oecorepath)) + + +parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/") + +parser.add_argument('--force-bootstraplayer-checkout', action='store_true', + help='Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place).') + +try: + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', universal_newlines=True, shell=True, cwd=os.path.dirname(__file__))) +except subprocess.CalledProcessError as e: + defaultdest = os.path.abspath(".") + +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) +parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json")) + +args = parser.parse_args() + +with open(args.jsondata) as f: + json = json.load(f) + +supported_versions = ["1.0"] +if json["version"] not in supported_versions: + raise Exception("File {} has version {}, which is not in supported versions: {}".format(args.jsondata, json["version"], supported_versions)) + +_do_checkout(vars(args), json) -- 2.30.2 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file 2022-08-17 13:10 ` [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from " Alexander Kanavin @ 2022-08-21 22:04 ` Philip Balister 2022-08-22 7:45 ` Alexander Kanavin 2022-08-22 1:20 ` Chuck Wolber 1 sibling, 1 reply; 18+ messages in thread From: Philip Balister @ 2022-08-21 22:04 UTC (permalink / raw) To: Alexander Kanavin, openembedded-core; +Cc: Alexander Kanavin On 8/17/22 09:10, Alexander Kanavin wrote: > This script can be used directly from poky, or can be copied directly into a Does this work if you are not using poky? Philip > layer or any other repository - it is self-suffucient and requires only python3 > and git on the host where it will run. It is also copied by the bitbake-layers > layers-setup plugin together with the json, unless requested otherwise. > > 1. How to restore the layers from the saved configuration: > > a) Clone the bootstrap layer or some other repository to obtain the json config and the setup script that can use it. > (use 'bitbake-layers create-layer-setup' from the previous commit to create them) > > b) Running with default options: > (note: this will work to update an existing checkout as well) > > alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers > Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override. > > Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master > Running 'git init -q /srv/work/alex/my-build/meta-intel' > Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel > Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel > Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel > > Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers > Running 'git init -q /srv/work/alex/my-build/poky' > Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky > Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky > Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky > Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky > Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky > > Available build configurations: > /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gadget > /srv/work/alex/my-build/meta-alex/conf/templates/configuration-gizmo > /srv/work/alex/my-build/poky/meta-poky/conf > > You can set up a build with: > TEMPLATECONF=one/of/the/above . /srv/work/alex/my-build/poky/oe-init-build-env > > 2. Command line options: > > alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers -h > usage: setup-layers [-h] [--force-bootstraplayer-checkout] [--destdir DESTDIR] [--jsondata JSONDATA] > > A self contained python script that fetches all the needed layers and sets them to correct revisions > > optional arguments: > -h, --help show this help message and exit > --force-bootstraplayer-checkout > Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place). > --destdir DESTDIR Where to check out the layers (default is /srv/work/alex/my-build). > --jsondata JSONDATA File containing the layer data in json format (default is /srv/work/alex/my-build/meta-alex/setup-layers.json). > > Signed-off-by: Alexander Kanavin <alex@linutronix.de> > --- > scripts/oe-setup-layers | 89 +++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 89 insertions(+) > create mode 100755 scripts/oe-setup-layers > > diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers > new file mode 100755 > index 0000000000..fe948898b0 > --- /dev/null > +++ b/scripts/oe-setup-layers > @@ -0,0 +1,89 @@ > +#!/usr/bin/env python3 > +# > +# This file was copied from poky(or oe-core)/scripts/oe-setup-layers by running > +# > +# bitbake-layers create-layers-setup destdir > +# > +# It is recommended that you do not modify this file directly, but rather re-run the above command to get the freshest upstream copy. > +# > + > +import argparse > +import json > +import os > +import subprocess > + > +def _do_checkout(args, json): > + layers = json['sources'] > + buildconfs = [] > + oecorepath = "" > + for l_name in layers: > + l_data = layers[l_name] > + layerdir = os.path.abspath(os.path.join(args['destdir'], l_data['path'])) > + > + for ll_name in l_data['layers']: > + if ll_name == 'meta': > + oecorepath = layerdir > + ll_data = l_data['layers'][ll_name] > + if 'buildconfigs' in ll_data: > + for c in ll_data['buildconfigs']: > + buildconfs.append(os.path.join(layerdir, ll_data['subpath'], c)) > + > + if 'contains_this_file' in l_data.keys(): > + force_arg = 'force_bootstraplayer_checkout' > + if not args[force_arg]: > + print('Note: not checking out source {layer}, use {layerflag} to override.'.format(layer=l_name, layerflag='--force-bootstraplayer-checkout')) > + continue > + l_remote = l_data['git-remote'] > + rev = l_remote['rev'] > + desc = l_remote['describe'] > + if not desc: > + desc = rev[:10] > + branch = l_remote['branch'] > + remotes = l_remote['remotes'] > + > + print('\nSetting up source {}, revision {}, branch {}'.format(l_name, desc, branch)) > + cmd = 'git init -q {}'.format(layerdir) > + print("Running '{}'".format(cmd)) > + subprocess.check_output(cmd, shell=True) > + > + for remote in remotes: > + cmd = "git remote remove {} > /dev/null 2>&1; git remote add {} {}".format(remote, remote, remotes[remote]['uri']) > + print("Running '{}' in {}".format(cmd, layerdir)) > + subprocess.check_output(cmd, shell=True, cwd=layerdir) > + > + cmd = "git fetch -q {} || true".format(remote) > + print("Running '{}' in {}".format(cmd, layerdir)) > + subprocess.check_output(cmd, shell=True, cwd=layerdir) > + > + cmd = 'git checkout -q {}'.format(rev) > + print("Running '{}' in {}".format(cmd, layerdir)) > + subprocess.check_output(cmd, shell=True, cwd=layerdir) > + > + > + print("\nAvailable build configurations:\n{}\n".format("\n".join(buildconfs))) > + print("You can set up a build with:\nTEMPLATECONF=one/of/the/above . {}/oe-init-build-env".format(oecorepath)) > + > + > +parser = argparse.ArgumentParser(description="A self contained python script that fetches all the needed layers and sets them to correct revisions using data in a json format from a separate file. The json data can be created from an active build directory with 'bitbake-layers create-layers-setup destdir' and there's a sample file and a schema in meta/files/") > + > +parser.add_argument('--force-bootstraplayer-checkout', action='store_true', > + help='Force the checkout of the layer containing this file (by default it is presumed that as this script is in it, the layer is already in place).') > + > +try: > + defaultdest = os.path.dirname(subprocess.check_output('git rev-parse --show-toplevel', universal_newlines=True, shell=True, cwd=os.path.dirname(__file__))) > +except subprocess.CalledProcessError as e: > + defaultdest = os.path.abspath(".") > + > +parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest)) > +parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json")) > + > +args = parser.parse_args() > + > +with open(args.jsondata) as f: > + json = json.load(f) > + > +supported_versions = ["1.0"] > +if json["version"] not in supported_versions: > + raise Exception("File {} has version {}, which is not in supported versions: {}".format(args.jsondata, json["version"], supported_versions)) > + > +_do_checkout(vars(args), json) > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#169481): https://lists.openembedded.org/g/openembedded-core/message/169481 > Mute This Topic: https://lists.openembedded.org/mt/93080238/384425 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [philip@balister.org] > -=-=-=-=-=-=-=-=-=-=-=- > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file 2022-08-21 22:04 ` [OE-core] " Philip Balister @ 2022-08-22 7:45 ` Alexander Kanavin 0 siblings, 0 replies; 18+ messages in thread From: Alexander Kanavin @ 2022-08-22 7:45 UTC (permalink / raw) To: Philip Balister; +Cc: OE-core, Alexander Kanavin On Mon, 22 Aug 2022 at 00:04, Philip Balister <philip@balister.org> wrote: > > On 8/17/22 09:10, Alexander Kanavin wrote: > > This script can be used directly from poky, or can be copied directly into a > > Does this work if you are not using poky? Yes. It's a case of 'poky chauvinism' on my part. I'll edit the commit messages to reassure everyone :) https://en.wikipedia.org/wiki/Carbon_chauvinism Alex ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file 2022-08-17 13:10 ` [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from " Alexander Kanavin 2022-08-21 22:04 ` [OE-core] " Philip Balister @ 2022-08-22 1:20 ` Chuck Wolber 2022-08-22 7:52 ` Alexander Kanavin 1 sibling, 1 reply; 18+ messages in thread From: Chuck Wolber @ 2022-08-22 1:20 UTC (permalink / raw) To: Alexander Kanavin; +Cc: OE-core, Alexander Kanavin [-- Attachment #1: Type: text/plain, Size: 1239 bytes --] On Wed, Aug 17, 2022 at 6:10 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote: > This script can be used directly from poky, or can be copied directly into > a > layer or any other repository - it is self-suffucient and requires only > python3 > and git on the host where it will run. It is also copied by the > bitbake-layers > layers-setup plugin together with the json, unless requested otherwise. > I need to review the code a bit closer, but on the face of it, this appears to solve the repository pinning problem quite elegantly. This matters a great deal in tightly configuration controlled environments, such as where regulatory oversight is involved. One feedback item I might suggest is to check if the layer is already pulled and to avoid pulling again. Same goes for the commit hash - no reason to checkout the hash if HEAD is already pointing at it. Adding those checks would enable your script to be added as a pre-build step in a developer workflow wrapper script. Our environment takes a different approach to repository pinning, but if setup-layers had those checks, we could probably switch to this approach. ..Ch:W.. -- *"Perfection must be reached by degrees; she requires the slow hand of time." - Voltaire* [-- Attachment #2: Type: text/html, Size: 1821 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [OE-core] [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from a json file 2022-08-22 1:20 ` Chuck Wolber @ 2022-08-22 7:52 ` Alexander Kanavin 0 siblings, 0 replies; 18+ messages in thread From: Alexander Kanavin @ 2022-08-22 7:52 UTC (permalink / raw) To: Chuck Wolber; +Cc: OE-core, Alexander Kanavin On Mon, 22 Aug 2022 at 03:20, Chuck Wolber <chuckwolber@gmail.com> wrote: > One feedback item I might suggest is to check if the layer is already pulled and to avoid pulling > again. Same goes for the commit hash - no reason to checkout the hash if HEAD is already pointing > at it. The script does 'git fetch' which is an incremental operation, and becomes cheap and quick after the initial fetch. Same for the checkout, which is trivial. I'd rather not complicate the code with additional conditions, unless there's some scenario I am not aware of. The script needs to be robust on a bulletproof level. > Adding those checks would enable your script to be added as a pre-build step in a developer > workflow wrapper script. Our environment takes a different approach to repository pinning, but if > setup-layers had those checks, we could probably switch to this approach. Please explain the details, I'd like to understand where the problem is specifically. Alex ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 5/5] selftest/bblayers: add a test for creating a layer setup and using it to restore the layers 2022-08-17 13:10 [PATCH 1/5] bitbake-layers: add a command to save the active build configuration as a template into a layer Alexander Kanavin ` (2 preceding siblings ...) 2022-08-17 13:10 ` [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from " Alexander Kanavin @ 2022-08-17 13:10 ` Alexander Kanavin 3 siblings, 0 replies; 18+ messages in thread From: Alexander Kanavin @ 2022-08-17 13:10 UTC (permalink / raw) To: openembedded-core; +Cc: Alexander Kanavin This does a basic run-through of the bitbake-layers plugin, and the resulting json layer config and the layer setup script that uses it. Only poky is actually fetched by the script. Signed-off-by: Alexander Kanavin <alex@linutronix.de> --- meta/lib/oeqa/selftest/cases/bblayers.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/bblayers.py b/meta/lib/oeqa/selftest/cases/bblayers.py index c753a7b795..c6f3000c90 100644 --- a/meta/lib/oeqa/selftest/cases/bblayers.py +++ b/meta/lib/oeqa/selftest/cases/bblayers.py @@ -142,3 +142,25 @@ class BitbakeLayers(OESelftestTestCase): def test_validate_examplelayersjson(self): json = os.path.join(get_bb_var('COREBASE'), "meta/files/layers.example.json") self.validate_layersjson(json) + + def test_bitbakelayers_setup(self): + result = runCmd('bitbake-layers create-layers-setup {}'.format(self.testlayer_path)) + jsonfile = os.path.join(self.testlayer_path, "setup-layers.json") + self.validate_layersjson(jsonfile) + + # The revision-under-test may not necessarily be available on the remote server, + # so replace it with a stable release tag. + import json + with open(jsonfile) as f: + data = json.load(f) + for s in data['sources']: + data['sources'][s]['git-remote']['rev'] = 'yocto-4.0' + with open(jsonfile, 'w') as f: + json.dump(data, f) + + testcheckoutdir = os.path.join(self.builddir, 'test-layer-checkout') + result = runCmd('{}/setup-layers --destdir {}'.format(self.testlayer_path, testcheckoutdir)) + # May not necessarily be named 'poky' + pokydir = os.listdir(testcheckoutdir)[0] + testcheckoutfile = os.path.join(testcheckoutdir, pokydir, "oe-init-build-env") + self.assertTrue(os.path.exists(testcheckoutfile), "File {} not found in test layer checkout".format(testcheckoutfile)) -- 2.30.2 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* configuration fragments @ 2022-03-14 19:57 Monsees, Steven C (US) 0 siblings, 0 replies; 18+ messages in thread From: Monsees, Steven C (US) @ 2022-03-14 19:57 UTC (permalink / raw) To: yocto [-- Attachment #1: Type: text/plain, Size: 799 bytes --] Under my aarm64 platform ../recipes-core/images I have a file "aiox-swdebugfs.inc" Which basically contains: EXTRAPATHS_prepend := "$(THISDIR}/files:" LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" SRC_URI += file://aiox-swdebugfs.cfg I include this in my aiox-swdebugfs.bb like so: "require aiox-swdebugfs.inc" It does not appear to apply my configuration fragments... (CONFIG_SCHED_TRACER, CONFIG_TRACER, etc. for additional perf support) I have the exact same implementation working under my intel platform. Can someone tell me what I might be doing wrong, or how I can get my configuration fragments to be applied properly ? Any ideas why it might work properly under one architecture and not on the other ? Thanks, Steve [-- Attachment #2: Type: text/html, Size: 3273 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2022-09-01 22:50 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-08-17 13:10 [PATCH 1/5] bitbake-layers: add a command to save the active build configuration as a template into a layer Alexander Kanavin 2022-08-17 13:10 ` [PATCH 2/5] meta/files: add layer setup JSON schema and example Alexander Kanavin 2022-08-17 20:52 ` [OE-core] " Richard Purdie 2022-08-17 21:36 ` Alexander Kanavin 2022-08-17 22:27 ` Richard Purdie 2022-08-18 8:24 ` Alexander Kanavin 2022-08-18 9:27 ` Richard Purdie 2022-09-01 15:29 ` Configuration fragments Alexander Kanavin 2022-09-01 22:47 ` [Openembedded-architecture] " Mark Hatle 2022-08-17 22:16 ` [OE-core] [PATCH 2/5] meta/files: add layer setup JSON schema and example Alexander Kanavin 2022-08-17 13:10 ` [PATCH 3/5] bitbake-layers: add ability to save current layer repository configuration into a json file Alexander Kanavin 2022-08-17 13:10 ` [PATCH 4/5] scripts/oe-setup-layers: add a script that restores the layer configuration from " Alexander Kanavin 2022-08-21 22:04 ` [OE-core] " Philip Balister 2022-08-22 7:45 ` Alexander Kanavin 2022-08-22 1:20 ` Chuck Wolber 2022-08-22 7:52 ` Alexander Kanavin 2022-08-17 13:10 ` [PATCH 5/5] selftest/bblayers: add a test for creating a layer setup and using it to restore the layers Alexander Kanavin -- strict thread matches above, loose matches on Subject: below -- 2022-03-14 19:57 configuration fragments Monsees, Steven C (US)
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.