[PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[Alexander Kanavin]

This is done only for default oe-core/poky templates; for anything
else the locations themselves need to be migrated first, and there
is no way to tell where they would be.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 scripts/oe-setup-builddir | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
index bf832ee0ca..32bc6580eb 100755
--- a/scripts/oe-setup-builddir
+++ b/scripts/oe-setup-builddir
@@ -40,6 +40,12 @@ cd "$BUILDDIR"
 
 if [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then
     TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
+    # The following two are no longer valid; unsetting them will automatically get them replaced
+    # with correct ones.
+    if [ $TEMPLATECONF = "meta/conf" -o $TEMPLATECONF = "meta-poky/conf" ]; then
+        unset TEMPLATECONF
+        rm $BUILDDIR/conf/templateconf.cfg
+    fi
 fi
 
 . "$OEROOT"/.templateconf
-- 
2.30.2

RE: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Alexander Kanavin
> Sent: den 1 september 2022 16:23
> To: openembedded-core@lists.openembedded.org
> Cc: Alexander Kanavin <alex@linutronix.de>
> Subject: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations
> 
> This is done only for default oe-core/poky templates; for anything
> else the locations themselves need to be migrated first, and there
> is no way to tell where they would be.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  scripts/oe-setup-builddir | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
> index bf832ee0ca..32bc6580eb 100755
> --- a/scripts/oe-setup-builddir
> +++ b/scripts/oe-setup-builddir
> @@ -40,6 +40,12 @@ cd "$BUILDDIR"
> 
>  if [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then
>      TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
> +    # The following two are no longer valid; unsetting them will automatically get them replaced
> +    # with correct ones.
> +    if [ $TEMPLATECONF = "meta/conf" -o $TEMPLATECONF = "meta-poky/conf" ]; then
> +        unset TEMPLATECONF
> +        rm $BUILDDIR/conf/templateconf.cfg
> +    fi

Since the test you added previously now requires that the path contains 
"/templates/", we might as well do this for any faulty configuration. I.e.:

    # The path in $TEMPLATECONF must contain "/templates/". If it doesn't, 
    # unset it to have it replaced by a (hopefully) correct path.
    case $TEMPLATECONF in
        */templates/*)
            ;;
        *)
            unset TEMPLATECONF
            rm "$BUILDDIR/conf/templateconf.cfg"
            ;;
    esac

If .templateconf hasn't been updated to comply with the new rules, the 
test later will point this out. But if it has been updated, the case 
statement above will avoid the problem of an old, non-complying 
configuration regardless of which layer it comes from.

>  fi
> 
>  . "$OEROOT"/.templateconf
> --
> 2.30.2

//Peter

Re: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[Alexander Kanavin]

On Fri, 2 Sept 2022 at 02:12, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> Since the test you added previously now requires that the path contains
> "/templates/", we might as well do this for any faulty configuration. I.e.:
>
>     # The path in $TEMPLATECONF must contain "/templates/". If it doesn't,
>     # unset it to have it replaced by a (hopefully) correct path.
>     case $TEMPLATECONF in
>         */templates/*)
>             ;;
>         *)
>             unset TEMPLATECONF
>             rm "$BUILDDIR/conf/templateconf.cfg"
>             ;;
>     esac
>
> If .templateconf hasn't been updated to comply with the new rules, the
> test later will point this out. But if it has been updated, the case
> statement above will avoid the problem of an old, non-complying
> configuration regardless of which layer it comes from.

.templateconf is not a universal file that can be placed into any
layer, it is specific to poky/oe-core, as it is read from $OEROOT,
which in turn is defined as location of oe-init-build-env.

So if the fixing up is made generic as suggested, it will only replace
incorrect but at least helpful value of the original custom template
location with the generic default template location in poky/core.

We can however add a hint to check what is in
build/conf/templateconf.cfg if the test fails, as it is not obvious
that TEMPLATECONF obtained from there takes priority over other
sources (which are, in order, TEMPLATECONF passed via environment,
then .templateconf next to oe-init-build-env).

Alex

RE: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[Peter Kjellerstedt]

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 2 september 2022 06:32
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: openembedded-core@lists.openembedded.org; Alexander Kanavin
> <alex@linutronix.de>
> Subject: Re: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate
> build/conf/templateconf.cfg to new template locations
> 
> On Fri, 2 Sept 2022 at 02:12, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> > Since the test you added previously now requires that the path contains
> > "/templates/", we might as well do this for any faulty configuration.
> I.e.:
> >
> >     # The path in $TEMPLATECONF must contain "/templates/". If it doesn't,
> >     # unset it to have it replaced by a (hopefully) correct path.
> >     case $TEMPLATECONF in
> >         */templates/*)
> >             ;;
> >         *)
> >             unset TEMPLATECONF
> >             rm "$BUILDDIR/conf/templateconf.cfg"
> >             ;;
> >     esac
> >
> > If .templateconf hasn't been updated to comply with the new rules, the
> > test later will point this out. But if it has been updated, the case
> > statement above will avoid the problem of an old, non-complying
> > configuration regardless of which layer it comes from.
> 
> .templateconf is not a universal file that can be placed into any
> layer, it is specific to poky/oe-core, as it is read from $OEROOT,
> which in turn is defined as location of oe-init-build-env.

That assumption doesn't hold true. Our environment is set up to mimic 
Poky, but with extra layers and our own version of ".templateconf" 
(installed via a <copyfile> element in the repo manifest). So 
everything works and looks like Poky, except there are more layers 
available.

With my suggested change above the code just works, without any 
artificial limits such as only working for plain OE/Poky.

> So if the fixing up is made generic as suggested, it will only replace
> incorrect but at least helpful value of the original custom template
> location with the generic default template location in poky/core.

The expectation is of course that if one has one's own .templateconf 
file, then it will of course have to be updated to match the changes 
in OE. Nothing unusual with that. But that is something I do as 
maintainer of our environment. For our developers it will then just 
work.

> 
> We can however add a hint to check what is in
> build/conf/templateconf.cfg if the test fails, as it is not obvious
> that TEMPLATECONF obtained from there takes priority over other
> sources (which are, in order, TEMPLATECONF passed via environment,
> then .templateconf next to oe-init-build-env).
> 
> Alex

//Peter

Re: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[Alexander Kanavin]

On Fri, 2 Sept 2022 at 16:47, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:

> The expectation is of course that if one has one's own .templateconf
> file, then it will of course have to be updated to match the changes
> in OE. Nothing unusual with that. But that is something I do as
> maintainer of our environment. For our developers it will then just
> work.

Please consider that there are three different ways to obtain the
location of the template, and they must all work. For those who have
created a build/conf by supplying TEMPLATECONF on the command line or
templateconf.cfg (which I would say is everyone, except you) the
suggested code will only make things worse, not better. It would take
wrong but helpful value in templateconf.cfg that points to a custom
template, and replace it with meta/conf/templates/default, because
that is what they have in .templateconf. Then all trace of where the
template came from is lost.

And as discussed previously, you don't even need to jump through the
extra hoops with generated template configurations to begin with, and
can write the dynamically generated configs directly into build/conf.

Alex

RE: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[Peter Kjellerstedt]

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 2 september 2022 17:09
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: openembedded-core@lists.openembedded.org; Alexander Kanavin
> <alex@linutronix.de>
> Subject: Re: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate
> build/conf/templateconf.cfg to new template locations
> 
> On Fri, 2 Sept 2022 at 16:47, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> 
> > The expectation is of course that if one has one's own .templateconf
> > file, then it will of course have to be updated to match the changes
> > in OE. Nothing unusual with that. But that is something I do as
> > maintainer of our environment. For our developers it will then just
> > work.
> 
> Please consider that there are three different ways to obtain the
> location of the template, and they must all work. For those who have
> created a build/conf by supplying TEMPLATECONF on the command line or
> templateconf.cfg (which I would say is everyone, except you) the
> suggested code will only make things worse, not better. It would take
> wrong but helpful value in templateconf.cfg that points to a custom
> template, and replace it with meta/conf/templates/default, because
> that is what they have in .templateconf. Then all trace of where the
> template came from is lost.

Actually, that does not correspond to what the documentation says. 
The way we do things (i.e., modify .templateconf to define a default 
TEMPLATECONF) _is_ what the documentation says one should do. See:
https://docs.yoctoproject.org/dev/singleindex.html#creating-a-custom-template-configuration-directory

Also note that templateconf.cfg is not documented at all. My guess 
(based on the current code) is that it is only supposed to be used 
internally as a way to store the value of a previous build where 
TEMPLATECONF was explicitly set.

So based on the documentation, the officially supported ways to 
configure the template directory is to either specify TEMPLATECONF 
explicitly or by modifying .templateconf to set a default value.

That said, I do agree with your suggested change to prefer an 
explicitly set TEMPLATECONF before a value from templateconf.cfg, 
which also agrees with the idea that templateconf.cfg is only an 
internal way of storing the last TEMPLATECONF used.

> 
> And as discussed previously, you don't even need to jump through the
> extra hoops with generated template configurations to begin with, and
> can write the dynamically generated configs directly into build/conf.
> 
> Alex

//Peter

Re: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[Alexander Kanavin]

On Fri, 2 Sept 2022 at 17:41, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> So based on the documentation, the officially supported ways to
> configure the template directory is to either specify TEMPLATECONF
> explicitly or by modifying .templateconf to set a default value.

Okay, I read that section, and I still think the requirement to have
.templateconf next to oe-init-build-env makes it altogether
impractical as a way to change the default from what poky/oe-core
prescribes, unless you want to maintain yet another custom script
doing non-standard magic before a build can start. I'm going to modify
the section so that TEMPLATECONF passed in explicitly is emphasized as
the standard method to set the template, and .templateconf is
described as primarily a fallback to the standard core/poky template.

Maybe we can decouple .templateconf from oe-init-build-env but I don't know how.

Alex

Re: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[Alexander Kanavin]

On Fri, 2 Sept 2022 at 06:32, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
> We can however add a hint to check what is in
> build/conf/templateconf.cfg if the test fails, as it is not obvious
> that TEMPLATECONF obtained from there takes priority over other
> sources (which are, in order, TEMPLATECONF passed via environment,
> then .templateconf next to oe-init-build-env).

Ok, the more I think about this, the more I'm convinced that
oe-setup-builddir needs a complete refactor. Specifically:

1. Let's rewrite it in python; shell just isn't expressive or readable
enough for what it needs to do.
2. Let's have a clear, logical priority order: TEMPLATECONF env var,
build/conf/templateconf.cfg, dirname('oe-init-build-env') +
'/.templateconf'
3. build/conf/templateconf.cfg becomes read-only; the final selected
value for TEMPLATECONF is written into a separate file (for having a
reference to where the configs came from).
4. oe-setup-builddir can also later be extended to support
'configuration fragments' management, see my email to oe-archtecture
from yesterday.

It's just too late to do this in this release cycle, but worth looking
into afterwards.

Alex

Re: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[Richard Purdie]

On Fri, 2022-09-02 at 09:16 +0200, Alexander Kanavin wrote:
> On Fri, 2 Sept 2022 at 06:32, Alexander Kanavin via
> lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
> wrote:
> > We can however add a hint to check what is in
> > build/conf/templateconf.cfg if the test fails, as it is not obvious
> > that TEMPLATECONF obtained from there takes priority over other
> > sources (which are, in order, TEMPLATECONF passed via environment,
> > then .templateconf next to oe-init-build-env).
> 
> Ok, the more I think about this, the more I'm convinced that
> oe-setup-builddir needs a complete refactor.

That script is meant to be really really simple and push any heavy
lifting off to bitbake or other tools. Ideally we were never even meant
to need such a script and the tools would "just work". I'd prefer it
went away, not rewrite and grow it.

Cheers,

Richard

Re: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[Alexander Kanavin]

On Fri, 2 Sept 2022 at 09:37, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > > We can however add a hint to check what is in
> > > build/conf/templateconf.cfg if the test fails, as it is not obvious
> > > that TEMPLATECONF obtained from there takes priority over other
> > > sources (which are, in order, TEMPLATECONF passed via environment,
> > > then .templateconf next to oe-init-build-env).
> >
> > Ok, the more I think about this, the more I'm convinced that
> > oe-setup-builddir needs a complete refactor.
>
> That script is meant to be really really simple and push any heavy
> lifting off to bitbake or other tools. Ideally we were never even meant
> to need such a script and the tools would "just work". I'd prefer it
> went away, not rewrite and grow it.

Rewriting it in python will make it twice (or more) shorter, and the
order of templateconfs does need clarity and logic. We can leave the
'config fragments' or anything more advanced out.

Alex

Re: [OE-core] [PATCH] scripts/oe-setup-builddir: migrate build/conf/templateconf.cfg to new template locations

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1695 bytes --]

Ok, I think I overstated things :) a couple of small targeted fixes
incoming, no python rewrite.

Alex


On Fri 2. Sep 2022 at 9.41, Alexander Kanavin via lists.openembedded.org
<alex.kanavin=gmail.com@lists.openembedded.org> wrote:

> On Fri, 2 Sept 2022 at 09:37, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > > > We can however add a hint to check what is in
> > > > build/conf/templateconf.cfg if the test fails, as it is not obvious
> > > > that TEMPLATECONF obtained from there takes priority over other
> > > > sources (which are, in order, TEMPLATECONF passed via environment,
> > > > then .templateconf next to oe-init-build-env).
> > >
> > > Ok, the more I think about this, the more I'm convinced that
> > > oe-setup-builddir needs a complete refactor.
> >
> > That script is meant to be really really simple and push any heavy
> > lifting off to bitbake or other tools. Ideally we were never even meant
> > to need such a script and the tools would "just work". I'd prefer it
> > went away, not rewrite and grow it.
>
> Rewriting it in python will make it twice (or more) shorter, and the
> order of templateconfs does need clarity and logic. We can leave the
> 'config fragments' or anything more advanced out.
>
> Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#170238):
> https://lists.openembedded.org/g/openembedded-core/message/170238
> Mute This Topic: https://lists.openembedded.org/mt/93396426/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 2912 bytes --]