Re: [OE-core] [PATCH 1/7] scripts/oe-setup-builddir: add a check that TEMPLATECONF is valid

[Alexander Kanavin <alex.kanavin@gmail.com>]

On Thu, 15 Sept 2022 at 03:07, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> I know you have suggested this to me too as we use a similar setup to
> generate the bblayers.conf.sample file based on the layers that repo
> has checked out in our environment, and now I realized why this is not
> as simple as it seems. The problem (at least in our case) is that we
> run as a wrapper around oe-init-build-env. This means that before the
> wrapper sources oe-init-build-env, the build directory and thus the
> conf directory inside it (where the bblayers.conf file is supposed to
> go) may not exist. And after it has sourced oe-init-build-env it is
> too late as the bblayers.conf file needs to exists before. So to create
> the bblayers.conf file, the wrapper would have to duplicate all the
> non-trivial code in oe-setup-builddir that determines what the build
> directory should be called. And while this of course is possible to
> do, it means duplicating code that is internal to bitbake, and risking
> missing important changes to the upstream code in the future. Thus it
> is much easier and safer to generate the template files and let
> oe-init-build-env do its job.

Wait, I have to ask for details here. What is the non-trivial code
specifically? Isn't it simply

mkdir -p builddir/conf
write_out bblayers.conf

where builddir is the same argument that is passed to
oe-init-build-env? Something doesn't compute here for me.

> Where I (and apparently WindRiver) are coming from, the decision on what
> layers are part of the configuration is not something a specific layer
> is responsible for. I.e., in our setup we fetch the layers we want for
> a build, write the list of layers to bblayers.conf.sample and source
> oe-init-build-env. With your setup, if I want a project with the two
> layers meta-a and meta-b, I would also have to create a layer meta-x
> where I create a static bblayers.conf.sample file that lists these
> three layers. Then, if another project wants to use meta-a and meta-c,
> they would have to add a second static bblayers.conf.sample file in
> meta-x that lists those three layers. Basically every time there is a
> new combination of layers fetched, there would also have to be a
> duplicate of the information in a static bblayers.conf.sample file
> somewhere. I am probably missing something here, but to me this seems
> suboptimal. If the list of layers in the static file had been used to
> fetch the layers, I would better understand it, but then the problem
> becomes that the list is now in one of the arbitrary layers that are
> supposed to be fetched.

The idea is that yes, every possible combination of layers needs its
own template, and it goes under version control into a layer.
Typically this would be a layer that is common to all of your build
configurations (which is meta-a in the above example - *not* meta-x)
and that is where the templates go. Do you have such a layer?

I also have to ask, what prevents you from simply listing all of the
layers in a single unified bblayers.conf.sample. Do they step on each
other? How?

I also need to remind you that bblayers.conf(.sample) does support
includes, so you can implement a structured hierarchy of layer config
fragments, where the common bits are only written once.

Basically, there are alternatives, and I think all of them are better
than writing out the list of layers just before setting up a build
with a custom script.

> in our manifest repository. This simple manifest is all the project
> team needs to create to build with their project specific layer. It
> also means that the only input to our Jenkins jobs to build for this
> project (instead of the main platform) is the name of the project's
> repo manifest. It also means that if we make any changes to the main
> platform, e.g., add a new layer (which is a one line change to the
> cfp.xml manifest), all projects will automatically get and use that
> new layer without having to make any changes to all the project
> manifests.

This is somewhat tangential, as it is about extending the default
layer setup config, not the combinations of layers in the build
configuration. So far, the json is not extensible, but we can think
about how to extend and overlay it too. As far as development in
meta-foo goes, the templates (including layer lists) for it can be
hosted there, until the layer 'graduates' into the common structure.

Alex