[PATCH 1/2] .templateconf: Mark "meta/conf" as an invalid path for templates

[PATCH 1/2] .templateconf: Mark "meta/conf" as an invalid path for templates

[Peter Kjellerstedt]

Previously this was hardcoded in oe-setup-builddir.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 .templateconf | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.templateconf b/.templateconf
index e399e06acd..8e4a9ebdc6 100644
--- a/.templateconf
+++ b/.templateconf
@@ -1,2 +1,3 @@
 # Template settings
 TEMPLATECONF=${TEMPLATECONF:-meta/conf/templates/default}
+INVALID_TEMPLATECONFS=meta/conf

[PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Peter Kjellerstedt]

Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded
as invalid paths for templates. However, this is suboptimal for other
distros that are setup similarly to Poky. Instead, add support for a
new variable INVALID_TEMPLATECONFS, which takes a list of invalid
paths.  It is expected that this variable is set in the .templateconf
file together with the default value for TEMPLATECONF, typically at
the same time that TEMPLATECONF is updated to match the new
requirements.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 scripts/oe-setup-builddir | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
index 225919be92..c1148daf4f 100755
--- a/scripts/oe-setup-builddir
+++ b/scripts/oe-setup-builddir
@@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD
 
 cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!"
 
-if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then
-    TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
-    # The following two are no longer valid; unsetting them will automatically get them replaced
-    # with correct ones.
-    if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then
-        unset TEMPLATECONF
-        rm "$BUILDDIR/conf/templateconf.cfg"
-    fi
+TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg"
+if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then
+    TEMPLATECONF=$(cat "$TEMPLATECONF_CFG")
+    # Unset TEMPLATECONF if it is set to a known invalid value to have it
+    # automatically replaced with a correct one.
+    for dir in $INVALID_TEMPLATECONFS; do
+        if [ "$TEMPLATECONF" = "$dir" ]; then
+           echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'"
+           unset TEMPLATECONF
+           rm "$TEMPLATECONF_CFG"
+	   break;
+        fi
+    done
 fi
 
 . "$OEROOT/.templateconf"

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Richard Purdie]

On Thu, 2022-09-22 at 14:47 +0200, Peter Kjellerstedt wrote:
> Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded
> as invalid paths for templates. However, this is suboptimal for other
> distros that are setup similarly to Poky. Instead, add support for a
> new variable INVALID_TEMPLATECONFS, which takes a list of invalid
> paths.  It is expected that this variable is set in the .templateconf
> file together with the default value for TEMPLATECONF, typically at
> the same time that TEMPLATECONF is updated to match the new
> requirements.
> 
> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> ---
>  scripts/oe-setup-builddir | 21 +++++++++++++--------
>  1 file changed, 13 insertions(+), 8 deletions(-)
> 
> diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
> index 225919be92..c1148daf4f 100755
> --- a/scripts/oe-setup-builddir
> +++ b/scripts/oe-setup-builddir
> @@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD
>  
>  cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!"
>  
> -if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then
> -    TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
> -    # The following two are no longer valid; unsetting them will automatically get them replaced
> -    # with correct ones.
> -    if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then
> -        unset TEMPLATECONF
> -        rm "$BUILDDIR/conf/templateconf.cfg"
> -    fi
> +TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg"
> +if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then
> +    TEMPLATECONF=$(cat "$TEMPLATECONF_CFG")
> +    # Unset TEMPLATECONF if it is set to a known invalid value to have it
> +    # automatically replaced with a correct one.
> +    for dir in $INVALID_TEMPLATECONFS; do
> +        if [ "$TEMPLATECONF" = "$dir" ]; then
> +           echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'"
> +           unset TEMPLATECONF
> +           rm "$TEMPLATECONF_CFG"
> +	   break;
> +        fi
> +    done
>  fi

I've been trying to keep out this discussion but I really don't like
piling in more complexity and more things we have to support because of
a legacy code path we're trying to obsolete.

I also suspect this is going to be used to "subvert" the move to try
and standardise and if that happens, we may as well not have bothered
making changes at all :(.

As such I'm worried about this direction.

Cheers,

Richard

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Peter Kjellerstedt]

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: den 22 september 2022 15:14
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates
> 
> On Thu, 2022-09-22 at 14:47 +0200, Peter Kjellerstedt wrote:
> > Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded
> > as invalid paths for templates. However, this is suboptimal for other
> > distros that are setup similarly to Poky. Instead, add support for a
> > new variable INVALID_TEMPLATECONFS, which takes a list of invalid
> > paths.  It is expected that this variable is set in the .templateconf
> > file together with the default value for TEMPLATECONF, typically at
> > the same time that TEMPLATECONF is updated to match the new
> > requirements.
> >
> > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> > ---
> >  scripts/oe-setup-builddir | 21 +++++++++++++--------
> >  1 file changed, 13 insertions(+), 8 deletions(-)
> >
> > diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
> > index 225919be92..c1148daf4f 100755
> > --- a/scripts/oe-setup-builddir
> > +++ b/scripts/oe-setup-builddir
> > @@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD
> >
> >  cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!"
> >
> > -if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then
> > -    TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
> > -    # The following two are no longer valid; unsetting them will automatically get them replaced
> > -    # with correct ones.
> > -    if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then
> > -        unset TEMPLATECONF
> > -        rm "$BUILDDIR/conf/templateconf.cfg"
> > -    fi
> > +TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg"
> > +if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then
> > +    TEMPLATECONF=$(cat "$TEMPLATECONF_CFG")
> > +    # Unset TEMPLATECONF if it is set to a known invalid value to have it
> > +    # automatically replaced with a correct one.
> > +    for dir in $INVALID_TEMPLATECONFS; do
> > +        if [ "$TEMPLATECONF" = "$dir" ]; then
> > +           echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'"
> > +           unset TEMPLATECONF
> > +           rm "$TEMPLATECONF_CFG"
> > +	   break;
> > +        fi
> > +    done
> >  fi
> 
> I've been trying to keep out this discussion but I really don't like
> piling in more complexity and more things we have to support because of
> a legacy code path we're trying to obsolete.
> 
> I also suspect this is going to be used to "subvert" the move to try
> and standardise and if that happens, we may as well not have bothered
> making changes at all :(.
> 
> As such I'm worried about this direction.
> 
> Cheers,
> 
> Richard

All I want is to remove the hardcoded superficial limitations and 
allow us that provide distros setup just as Poky is setup to continue 
to do so.

//Peter

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

.templateconf is not the way to specify the location of custom
templates, and never was. It got added to documentation by
misunderstanding, this has now been removed, and so you need to update
your tooling so that templates are set with TEMPLATECONF going
forward.

Alex

On Thu, 22 Sept 2022 at 15:26, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
>
> > -----Original Message-----
> > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> > Sent: den 22 september 2022 15:14
> > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates
> >
> > On Thu, 2022-09-22 at 14:47 +0200, Peter Kjellerstedt wrote:
> > > Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded
> > > as invalid paths for templates. However, this is suboptimal for other
> > > distros that are setup similarly to Poky. Instead, add support for a
> > > new variable INVALID_TEMPLATECONFS, which takes a list of invalid
> > > paths.  It is expected that this variable is set in the .templateconf
> > > file together with the default value for TEMPLATECONF, typically at
> > > the same time that TEMPLATECONF is updated to match the new
> > > requirements.
> > >
> > > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> > > ---
> > >  scripts/oe-setup-builddir | 21 +++++++++++++--------
> > >  1 file changed, 13 insertions(+), 8 deletions(-)
> > >
> > > diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
> > > index 225919be92..c1148daf4f 100755
> > > --- a/scripts/oe-setup-builddir
> > > +++ b/scripts/oe-setup-builddir
> > > @@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD
> > >
> > >  cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!"
> > >
> > > -if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then
> > > -    TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
> > > -    # The following two are no longer valid; unsetting them will automatically get them replaced
> > > -    # with correct ones.
> > > -    if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then
> > > -        unset TEMPLATECONF
> > > -        rm "$BUILDDIR/conf/templateconf.cfg"
> > > -    fi
> > > +TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg"
> > > +if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then
> > > +    TEMPLATECONF=$(cat "$TEMPLATECONF_CFG")
> > > +    # Unset TEMPLATECONF if it is set to a known invalid value to have it
> > > +    # automatically replaced with a correct one.
> > > +    for dir in $INVALID_TEMPLATECONFS; do
> > > +        if [ "$TEMPLATECONF" = "$dir" ]; then
> > > +           echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'"
> > > +           unset TEMPLATECONF
> > > +           rm "$TEMPLATECONF_CFG"
> > > +      break;
> > > +        fi
> > > +    done
> > >  fi
> >
> > I've been trying to keep out this discussion but I really don't like
> > piling in more complexity and more things we have to support because of
> > a legacy code path we're trying to obsolete.
> >
> > I also suspect this is going to be used to "subvert" the move to try
> > and standardise and if that happens, we may as well not have bothered
> > making changes at all :(.
> >
> > As such I'm worried about this direction.
> >
> > Cheers,
> >
> > Richard
>
> All I want is to remove the hardcoded superficial limitations and
> allow us that provide distros setup just as Poky is setup to continue
> to do so.
>
> //Peter
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#170981): https://lists.openembedded.org/g/openembedded-core/message/170981
> Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Peter Kjellerstedt]

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 22 september 2022 17:53
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; openembedded-
> core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode
> invalid paths for templates
> 
> .templateconf is not the way to specify the location of custom
> templates, and never was. It got added to documentation by
> misunderstanding, this has now been removed, and so you need to update
> your tooling so that templates are set with TEMPLATECONF going
> forward.
> 
> Alex

[Irritated rant below, and I apologize in advance. Sorry. I know 
you want to improve things, but I just have to state my opinion.]

Ok, this is silly. .templateconf _is_ the documented way to specify a 
default TEMPLATECONF and has been so for a long time. Just because 
you don't use it or like it does not mean that you can just rip it 
out and say we now have to use some (for us) inferior method. We have 
500+ developers and a huge number of CI/CD pipelines configured and 
built thousand times a day. They all expect the current setup, one 
they have used for ten years. The _exact same setup_ you get if you 
only fetch Poky and start a build.

All I want to do is remove hardcoded values in favor of remaining 
useful for all users of Yocto. Why is that a problem? I simply cannot 
understand it. To me, hardcoded values should never be used when a 
generic solution is possible. Why is it better to hardcode knowledge 
about Poky in OE-Core, when there is no reason to do so?

I know (or rather hope) you have some plan going forward for how to 
setup the build environment, but it is not there in Langdale. Also, 
I want to be able to continue with fetching the layers ourselves 
the way we always have done it and then continue to tell bitbake 
where to find them as we always have done without having to go 
through hoops or having to duplicate a lot of bitbake's internals.
And I believe this was actually said during all discussions about 
how to add support for a common way of setting up the build 
environment, that it should be possible for us that already have 
ways to do it (be they repo, submodules, kas, whisk or whatever) 
to continue to do so. So please, please be mindful of simple changes 
that allow us to continue to use the processes we already have in 
place when it really doesn't cost anything.

//Peter

> On Thu, 22 Sept 2022 at 15:26, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> >
> > > -----Original Message-----
> > > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > Sent: den 22 september 2022 15:14
> > > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-
> core@lists.openembedded.org
> > > Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode
> invalid paths for templates
> > >
> > > On Thu, 2022-09-22 at 14:47 +0200, Peter Kjellerstedt wrote:
> > > > Previously, the paths "meta/conf" and "meta-poky/conf" were
> hardcoded
> > > > as invalid paths for templates. However, this is suboptimal for
> other
> > > > distros that are setup similarly to Poky. Instead, add support for a
> > > > new variable INVALID_TEMPLATECONFS, which takes a list of invalid
> > > > paths.  It is expected that this variable is set in the
> .templateconf
> > > > file together with the default value for TEMPLATECONF, typically at
> > > > the same time that TEMPLATECONF is updated to match the new
> > > > requirements.
> > > >
> > > > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> > > > ---
> > > >  scripts/oe-setup-builddir | 21 +++++++++++++--------
> > > >  1 file changed, 13 insertions(+), 8 deletions(-)
> > > >
> > > > diff --git a/scripts/oe-setup-builddir b/scripts/oe-setup-builddir
> > > > index 225919be92..c1148daf4f 100755
> > > > --- a/scripts/oe-setup-builddir
> > > > +++ b/scripts/oe-setup-builddir
> > > > @@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo
> "WARNING: unable to chmod $BUILDD
> > > >
> > > >  cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!"
> > > >
> > > > -if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg"
> ]; then
> > > > -    TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
> > > > -    # The following two are no longer valid; unsetting them will
> automatically get them replaced
> > > > -    # with correct ones.
> > > > -    if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-
> poky/conf ]; then
> > > > -        unset TEMPLATECONF
> > > > -        rm "$BUILDDIR/conf/templateconf.cfg"
> > > > -    fi
> > > > +TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg"
> > > > +if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then
> > > > +    TEMPLATECONF=$(cat "$TEMPLATECONF_CFG")
> > > > +    # Unset TEMPLATECONF if it is set to a known invalid value to
> have it
> > > > +    # automatically replaced with a correct one.
> > > > +    for dir in $INVALID_TEMPLATECONFS; do
> > > > +        if [ "$TEMPLATECONF" = "$dir" ]; then
> > > > +           echo "WARNING: Removing $TEMPLATECONF_CFG as it
> contained the invalid value '$TEMPLATECONF'"
> > > > +           unset TEMPLATECONF
> > > > +           rm "$TEMPLATECONF_CFG"
> > > > +      break;
> > > > +        fi
> > > > +    done
> > > >  fi
> > >
> > > I've been trying to keep out this discussion but I really don't like
> > > piling in more complexity and more things we have to support because
> of
> > > a legacy code path we're trying to obsolete.
> > >
> > > I also suspect this is going to be used to "subvert" the move to try
> > > and standardise and if that happens, we may as well not have bothered
> > > making changes at all :(.
> > >
> > > As such I'm worried about this direction.
> > >
> > > Cheers,
> > >
> > > Richard
> >
> > All I want is to remove the hardcoded superficial limitations and
> > allow us that provide distros setup just as Poky is setup to continue
> > to do so.
> >
> > //Peter
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#170981):
> https://lists.openembedded.org/g/openembedded-core/message/170981
> > Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
> [alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Richard Purdie]

On Thu, 2022-09-22 at 17:52 +0200, Alexander Kanavin wrote:
> .templateconf is not the way to specify the location of custom
> templates, and never was.

That much isn't true. It was a way this has been done, it was suggested
as a way people do it and it was even documented as such. The original
commit does make the intent clear:

https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9

I think the key point is that it has been of limited use as it requires
a custom repo due to the use of OEROOT. Where you had a custom repo, it
was fine to use. Poky was one example.

I haven't read back through the discussions but I suspect Peter must
have some kind of custom repo setup like that to be able to alter files
in OEROOT.

> It got added to documentation by misunderstanding,

Again, I'm not sure about that. I think at the time, when layers were
much newer things and technology was more in flux, we did think this
was a valid customisation. At the time we were suggesting people might
want to use tools like combo-layer to mess with repos.

What the documentation doesn't make clear is that this trick only works
for repo alterations as far as I can see.

Since we don't recommend such repo alterations now and combo-layer is
pretty dead, the code is of limited use and I do think it probably
should be removed. I therefore don't like adding complexity to
something which really shouldn't be there at all. I appreciate this
does looks like it creates a challenge for Peter's situation though.

I have put off touching this particular area of code since I know there
will be a ton of problems exactly like this. The success or failure of
such changes does depend on how the migration is handled. A large part
of this is communication and to some extend negotiation too. The fact
I'm having to start trying to understand all the details is the
situation I didn't really want to be in. 

I suspect to get anywhere I now need to go and read all the previous
threads to then try and unravel this :/.

Peter: is there a pointer you can give to the setup you're using?

Cheers,

Richard

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

First of all, I would like to understand why writing to .templateconf
cannot be replaced with specifying TEMPLATECONF= in your setup.

Then we can maybe proceed to the similar question asked earlier about
replacing writing to bblayer.conf with using bitbake-layers to adjust
the layer list.
There was also a question about removing the step of creating a custom
template altogether, and just writing a build configuration directly.

Where are the answers to those, Peter?

Alex

On Thu, 22 Sept 2022 at 22:32, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Thu, 2022-09-22 at 17:52 +0200, Alexander Kanavin wrote:
> > .templateconf is not the way to specify the location of custom
> > templates, and never was.
>
> That much isn't true. It was a way this has been done, it was suggested
> as a way people do it and it was even documented as such. The original
> commit does make the intent clear:
>
> https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9
>
> I think the key point is that it has been of limited use as it requires
> a custom repo due to the use of OEROOT. Where you had a custom repo, it
> was fine to use. Poky was one example.
>
> I haven't read back through the discussions but I suspect Peter must
> have some kind of custom repo setup like that to be able to alter files
> in OEROOT.
>
> > It got added to documentation by misunderstanding,
>
> Again, I'm not sure about that. I think at the time, when layers were
> much newer things and technology was more in flux, we did think this
> was a valid customisation. At the time we were suggesting people might
> want to use tools like combo-layer to mess with repos.
>
> What the documentation doesn't make clear is that this trick only works
> for repo alterations as far as I can see.
>
> Since we don't recommend such repo alterations now and combo-layer is
> pretty dead, the code is of limited use and I do think it probably
> should be removed. I therefore don't like adding complexity to
> something which really shouldn't be there at all. I appreciate this
> does looks like it creates a challenge for Peter's situation though.
>
> I have put off touching this particular area of code since I know there
> will be a ton of problems exactly like this. The success or failure of
> such changes does depend on how the migration is handled. A large part
> of this is communication and to some extend negotiation too. The fact
> I'm having to start trying to understand all the details is the
> situation I didn't really want to be in.
>
> I suspect to get anywhere I now need to go and read all the previous
> threads to then try and unravel this :/.
>
> Peter: is there a pointer you can give to the setup you're using?
>
> Cheers,
>
> Richard
>
>
>
>
>

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

I've just written and sent a sort of 'introduction' to the official
layer setup tooling:

https://lists.openembedded.org/g/openembedded-core/message/171070

Build configuration management is a separate concern, I have an
introduction to that too, but not sending it yet, as particularly
oe-setup-build needs to be reviewed and discussed and (hopefully)
merged.

Alex

On Thu, 22 Sept 2022 at 23:54, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
>
> First of all, I would like to understand why writing to .templateconf
> cannot be replaced with specifying TEMPLATECONF= in your setup.
>
> Then we can maybe proceed to the similar question asked earlier about
> replacing writing to bblayer.conf with using bitbake-layers to adjust
> the layer list.
> There was also a question about removing the step of creating a custom
> template altogether, and just writing a build configuration directly.
>
> Where are the answers to those, Peter?
>
> Alex
>
> On Thu, 22 Sept 2022 at 22:32, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Thu, 2022-09-22 at 17:52 +0200, Alexander Kanavin wrote:
> > > .templateconf is not the way to specify the location of custom
> > > templates, and never was.
> >
> > That much isn't true. It was a way this has been done, it was suggested
> > as a way people do it and it was even documented as such. The original
> > commit does make the intent clear:
> >
> > https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9
> >
> > I think the key point is that it has been of limited use as it requires
> > a custom repo due to the use of OEROOT. Where you had a custom repo, it
> > was fine to use. Poky was one example.
> >
> > I haven't read back through the discussions but I suspect Peter must
> > have some kind of custom repo setup like that to be able to alter files
> > in OEROOT.
> >
> > > It got added to documentation by misunderstanding,
> >
> > Again, I'm not sure about that. I think at the time, when layers were
> > much newer things and technology was more in flux, we did think this
> > was a valid customisation. At the time we were suggesting people might
> > want to use tools like combo-layer to mess with repos.
> >
> > What the documentation doesn't make clear is that this trick only works
> > for repo alterations as far as I can see.
> >
> > Since we don't recommend such repo alterations now and combo-layer is
> > pretty dead, the code is of limited use and I do think it probably
> > should be removed. I therefore don't like adding complexity to
> > something which really shouldn't be there at all. I appreciate this
> > does looks like it creates a challenge for Peter's situation though.
> >
> > I have put off touching this particular area of code since I know there
> > will be a ton of problems exactly like this. The success or failure of
> > such changes does depend on how the migration is handled. A large part
> > of this is communication and to some extend negotiation too. The fact
> > I'm having to start trying to understand all the details is the
> > situation I didn't really want to be in.
> >
> > I suspect to get anywhere I now need to go and read all the previous
> > threads to then try and unravel this :/.
> >
> > Peter: is there a pointer you can give to the setup you're using?
> >
> > Cheers,
> >
> > Richard
> >
> >
> >
> >
> >
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#170995): https://lists.openembedded.org/g/openembedded-core/message/170995
> Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Peter Kjellerstedt]

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: den 22 september 2022 22:33
> To: Alexander Kanavin <alex.kanavin@gmail.com>; Peter Kjellerstedt
> <peter.kjellerstedt@axis.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode
> invalid paths for templates
> 
> On Thu, 2022-09-22 at 17:52 +0200, Alexander Kanavin wrote:
> > .templateconf is not the way to specify the location of custom
> > templates, and never was.
> 
> That much isn't true. It was a way this has been done, it was suggested
> as a way people do it and it was even documented as such. The original
> commit does make the intent clear:
> 
> https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9
> 
> I think the key point is that it has been of limited use as it requires
> a custom repo due to the use of OEROOT. Where you had a custom repo, it
> was fine to use. Poky was one example.
> 
> I haven't read back through the discussions but I suspect Peter must
> have some kind of custom repo setup like that to be able to alter files
> in OEROOT.

Yes, and it is very simple to do when using `repo` like we do. Just add 
a <copyfile> line in the manifest that tells `repo` to copy a 
.templateconf file from a suitable repository and place it in the root.

> > It got added to documentation by misunderstanding,
> 
> Again, I'm not sure about that. I think at the time, when layers were
> much newer things and technology was more in flux, we did think this
> was a valid customisation. At the time we were suggesting people might
> want to use tools like combo-layer to mess with repos.
> 
> What the documentation doesn't make clear is that this trick only works
> for repo alterations as far as I can see.

See above. At least in the case of using `repo` to fetch the layers, 
this way of specifying the default template makes perfect sense.

> Since we don't recommend such repo alterations now and combo-layer is
> pretty dead, the code is of limited use and I do think it probably
> should be removed. I therefore don't like adding complexity to
> something which really shouldn't be there at all. I appreciate this
> does looks like it creates a challenge for Peter's situation though.

The expectation I have on Yocto is that whatever is fine and 
works for Poky should also be fine and work for anyone else 
setting up the environment in the same way.

I.e., for Poky I expect the following:

* Fetch the distribution (poky). In this case a simple git clone with 
  the URL to the poky repository and the wanted version.
* Change to the newly fetched directory.
* Run `. ./ oe-init-build-env`.

Which in our case corresponds to:

* Create and change to a new directory.
* Run `repo init` with the URL to the manifest repository, its version 
  and the manifest to use, followed by `repo sync`.
* Run `. ./ oe-init-build-env`.

So, as long as the layers with their correct versions have been 
fetched, all I should have to do is run `. ./ oe-init-build-env`.
That is what we have always done, and I am not going to change 
that. Changing that would be a huge undertaking for no gain to us.

Now, the above works since I have modified our wrapper to create 
the template files in "templateconf/templates/axis" rather than 
the old "templateconf" and to create a dummy layer.conf file so 
the new requirements are satisfied.

The problem is that because how the templateconf.cfg file is read, 
its value overrides the default in the .templateconf file and 
thus anyone updating an existing build tree to Langdale faces an 
error because the old templates path is unlikely to satisfy the 
new requirements. This is why the code in scripts/oe-setup-builddir 
was changed to remove the old templateconf.cfg file if it contains 
a known bad path.

Since I of course do not want for my developers to face this 
error when they update, I have two options:

1) I can remove the old templateconf.cfg file in our wrapper. 
   This means I have to duplicate the code in 
   scripts/oe-buildenv-internal that calculates the BUILDDIR 
   variable. It also means I will have to keep and maintain this 
   code for the foreseeable future.
2) Change the code in scripts/oe-setup-builddir so that it is 
   not hardcoded to only support OE and Poky, but instead is 
   generic and works the same for anyone.

From my point of view, option 2 should be the natural choice. 

> I have put off touching this particular area of code since I know there
> will be a ton of problems exactly like this. The success or failure of
> such changes does depend on how the migration is handled. A large part
> of this is communication and to some extend negotiation too. The fact
> I'm having to start trying to understand all the details is the
> situation I didn't really want to be in.
> 
> I suspect to get anywhere I now need to go and read all the previous
> threads to then try and unravel this :/.
> 
> Peter: is there a pointer you can give to the setup you're using?

I gave an example in 
https://lists.openembedded.org/g/openembedded-core/message/170677. 
Is that what you were looking for?

> 
> Cheers,
> 
> Richard

//Peter

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

On Tue, 27 Sept 2022 at 16:56, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> Since I of course do not want for my developers to face this
> error when they update, I have two options:
>
> 1) I can remove the old templateconf.cfg file in our wrapper.
>    This means I have to duplicate the code in
>    scripts/oe-buildenv-internal that calculates the BUILDDIR
>    variable. It also means I will have to keep and maintain this
>    code for the foreseeable future.
> 2) Change the code in scripts/oe-setup-builddir so that it is
>    not hardcoded to only support OE and Poky, but instead is
>    generic and works the same for anyone.
>
> From my point of view, option 2 should be the natural choice.

Three options. You can also instruct your developers to simply remove
their build directories. Is that such a catastrophe? We do not promise
that build directories stay backwards compatible and never explode on
distro upgrades, we try out best, but sometimes they just don't.

Alex

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

Now it's my time to rant at you, Peter. Brace yourself.

Let me just say this, you don't get to have expectations and
entitlements, if none of your '500 developers' are working fulltime to
help the core. I think you've been in the room last week when Nico and
Philip presented the numbers and the downward trend of who is doing
the work in core; what are you doing to address that situation, given
the resources your organization has?

We have a small handful of exhausted maintainers, I in particular have
to squeeze my maintenance work between fixing various customer BSPs,
and so it is on you to help us first. Otherwise, yes, I will say this:
fix your scripts, and stop complaining. I have a large amount of
manual recipe updates I need to do, because no one else wants to do
it, and so I will not be arguing with you anymore.

Alex

On Tue, 27 Sept 2022 at 17:29, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
>
> On Tue, 27 Sept 2022 at 16:56, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> > Since I of course do not want for my developers to face this
> > error when they update, I have two options:
> >
> > 1) I can remove the old templateconf.cfg file in our wrapper.
> >    This means I have to duplicate the code in
> >    scripts/oe-buildenv-internal that calculates the BUILDDIR
> >    variable. It also means I will have to keep and maintain this
> >    code for the foreseeable future.
> > 2) Change the code in scripts/oe-setup-builddir so that it is
> >    not hardcoded to only support OE and Poky, but instead is
> >    generic and works the same for anyone.
> >
> > From my point of view, option 2 should be the natural choice.
>
> Three options. You can also instruct your developers to simply remove
> their build directories. Is that such a catastrophe? We do not promise
> that build directories stay backwards compatible and never explode on
> distro upgrades, we try out best, but sometimes they just don't.
>
> Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#171107): https://lists.openembedded.org/g/openembedded-core/message/171107
> Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Peter Kjellerstedt]

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 27 september 2022 17:42
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core
> <openembedded-core@lists.openembedded.org>
> Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode
> invalid paths for templates
> 
> Now it's my time to rant at you, Peter. Brace yourself.
> 
> Let me just say this, you don't get to have expectations and
> entitlements, if none of your '500 developers' are working fulltime to
> help the core. I think you've been in the room last week when Nico and
> Philip presented the numbers and the downward trend of who is doing
> the work in core; what are you doing to address that situation, given
> the resources your organization has?

I (and the small team I am part of) is the proxy between them and Yocto. 
Thus I try to address their needs by making sure new versions of Yocto 
continues to work for them without having 500+ developers facing the 
same problems. I.e., it is my job to work with upstream, which is what 
I am trying to do here.

> We have a small handful of exhausted maintainers, I in particular have
> to squeeze my maintenance work between fixing various customer BSPs,
> and so it is on you to help us first.

Which is exactly what I am trying to do. I am trying to make sure that 
what has worked, continues to work, when it is not in opposition to the 
actual changes that you are making. You made this exact change (that 
my patch addresses) for OE and Poky because you realized that there was 
an error that could be avoided in a simple way. And I just want to 
extend that to be possible to do for all distributions besides Poky. 
This is typically something that affect those of us who have existing 
distributions and not users of the new way to setup the environment. 

> Otherwise, yes, I will say this:
> fix your scripts, and stop complaining. 

I can (and will) of course do that, but since I believe my fix is of 
interest not only to us, but others who setup their environment 
using the .templateconf file, my primary goal is to try to improve 
upstream first.

> I have a large amount of
> manual recipe updates I need to do, because no one else wants to do
> it, and so I will not be arguing with you anymore.
> 
> Alex

//Peter

> On Tue, 27 Sept 2022 at 17:29, Alexander Kanavin via
> lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
> wrote:
> >
> > On Tue, 27 Sept 2022 at 16:56, Peter Kjellerstedt
> > <peter.kjellerstedt@axis.com> wrote:
> > > Since I of course do not want for my developers to face this
> > > error when they update, I have two options:
> > >
> > > 1) I can remove the old templateconf.cfg file in our wrapper.
> > >    This means I have to duplicate the code in
> > >    scripts/oe-buildenv-internal that calculates the BUILDDIR
> > >    variable. It also means I will have to keep and maintain this
> > >    code for the foreseeable future.
> > > 2) Change the code in scripts/oe-setup-builddir so that it is
> > >    not hardcoded to only support OE and Poky, but instead is
> > >    generic and works the same for anyone.
> > >
> > > From my point of view, option 2 should be the natural choice.
> >
> > Three options. You can also instruct your developers to simply remove
> > their build directories. Is that such a catastrophe? We do not promise
> > that build directories stay backwards compatible and never explode on
> > distro upgrades, we try out best, but sometimes they just don't.
> >
> > Alex
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#171107):
> https://lists.openembedded.org/g/openembedded-core/message/171107
> > Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
> [alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

On Tue, 27 Sept 2022 at 18:06, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> > Otherwise, yes, I will say this:
> > fix your scripts, and stop complaining.
>
> I can (and will) of course do that, but since I believe my fix is of
> interest not only to us, but others who setup their environment
> using the .templateconf file, my primary goal is to try to improve
> upstream first.

Yes, but:

1. There is no evidence anyone else except Axis is actually using .templateconf.

2. .templateconf is now officially retired as a way to set the
template location. The patch you've sent is completely contrary to
that: rather it makes the incorrect impression that it is still
supported, and the implication is that the core upstream would still
have to accommodate and think of those who are using it going forward.
Yes it's mildly annoying to delete the build directory, but can you
just do it and move on? And yes, make a plan for replacing it with
TEMPLATECONF, which is what I've been trying to squeeze out of you for
I don't know how many times.

Alex

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Peter Kjellerstedt]

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 27 september 2022 18:17
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core <openembedded-core@lists.openembedded.org>
> Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates
> 
> On Tue, 27 Sept 2022 at 18:06, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote:
> > > Otherwise, yes, I will say this:
> > > fix your scripts, and stop complaining.
> >
> > I can (and will) of course do that, but since I believe my fix is of
> > interest not only to us, but others who setup their environment
> > using the .templateconf file, my primary goal is to try to improve
> > upstream first.
> 
> Yes, but:
> 
> 1. There is no evidence anyone else except Axis is actually using
> .templateconf.

Since we obviously use our own distribution, I am limited in 
experience with other distributions. The one example I do know about, 
which I have mentioned before, is WindRiver. See line 831-835 in 
https://github.com/WindRiver-Labs/wrlinux-x/blob/WRLINUX_10_21_BASE/bin/setup.py
for an example of where they do basically the same as what we do.

> 
> 2. .templateconf is now officially retired as a way to set the
> template location. The patch you've sent is completely contrary to
> that: rather it makes the incorrect impression that it is still
> supported, and the implication is that the core upstream would still
> have to accommodate and think of those who are using it going forward.

Which is not entirely true, is it? Support for .templateconf I still 
there. I am only extending what is already there to work without errors 
for everyone, not just OE and Poky.

> Yes it's mildly annoying to delete the build directory, but can you
> just do it and move on? And yes, make a plan for replacing it with
> TEMPLATECONF, which is what I've been trying to squeeze out of you for
> I don't know how many times.
> 
> Alex

I do not understand what the problem with .templateconf is? There is 
a use case for being able to specify a default TEMPLATECONF, and that 
is what .templateconf satisfies.

There are currently three ways to specify where the templates files 
are (in order of preference):

1) TEMPLATECONF=
2) templateconf.cfg
3) .templateconf

In our case, the templates are always in the same location, thus we 
want to specify this as a default to give the developers a better 
user experience.

With the above three methods available, this is easy as we just set 
the default in the .templateconf file.

Now, if we remove support for .templateconf, how are we going to do it
instead? I can of course change our wrapper around oe-build-init-env 
to do TEMPLATECONF=${TEMPLATECONF:-path/to/our/templates} and this 
might seem fine at first. However, this means that the wrapper will 
always set TEMPLATECONF and thus templateconf.cfg looses its meaning, 
which to the user means that he must now always specify TEMPLATECONF= 
if he wants to set it to something other than the default where before 
he only had to specify it the first time he initializes a new build 
environment.

Or I can duplicate the calculations done in the OE scripts to figure 
out where BUILDDIR will end up to see if there is a templateconf.cfg 
file available, and then use that to set TEMPLATECONF if the user 
hasn't specified it. All of this duplicating upstream code, which 
may change with any future release. (Did I mention that I hate 
duplicating code?)

Now I understand that you have some idea that the user should always 
specify TEMPLATECONF=, in which case a default TEMPLATECONF has no 
value. But since we have a method for fetching exactly the layers 
that are to be used in a given build, to me it does not make sense 
to also have to specify a TEMPLATECONF to match what has already 
been fetched. It is just duplicating information, which IMHO is bad. 

What _I_ would prefer to see, is a way to specify that the build 
shall use the available layers. I.e., rather than a huge number of 
static templates configurations, I would like to see the 
initialization script take the layers as arguments, with the default 
being anything that matches meta* (or maybe more correctly, anything 
that has a conf/layers.conf file). As for the local.conf file, in 
my world, this would be made up from fragment files in the layers. 
So for any layer that ends up in the bblayers.conf file, the layer 
would be searched for any local.conf.XX files that would be 
concatenated together in the order of XX. This is (obviously) very 
similar to what we do today, so I am of course biased, but I think 
this model is a lot more appealing than static templates files. 
YMMV.

//Peter

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

On Tue, 27 Sept 2022 at 20:53, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> Since we obviously use our own distribution, I am limited in
> experience with other distributions. The one example I do know about,
> which I have mentioned before, is WindRiver. See line 831-835 in
> https://github.com/WindRiver-Labs/wrlinux-x/blob/WRLINUX_10_21_BASE/bin/setup.py
> for an example of where they do basically the same as what we do.

Yes, they have to fix this too. And not by replacing .templateconf
with TEMPLATECONF, but by writing a config directly, particularly
using available tooling to compose bblayers.conf. Templates are
intended for layers and version control, not as an extra intermediate
step when setting up builds - the goal is to standardize build
configuration management, and static templates is a starting point.

> I do not understand what the problem with .templateconf is? There is
> a use case for being able to specify a default TEMPLATECONF, and that
> is what .templateconf satisfies.

I'd say it does not satisfy the use case, as it relies on being next
to oe-init-build-env, which in turn requires custom scripting. Show
how to do this using standard tools in core/poky, write a patch for
the official documentation, and then you have a point.

> There are currently three ways to specify where the templates files
> are (in order of preference):
>
> 1) TEMPLATECONF=
> 2) templateconf.cfg
> 3) .templateconf
>
> In our case, the templates are always in the same location, thus we
> want to specify this as a default to give the developers a better
> user experience.

I get this, it is useful to make a pointer to a 'default' template
when you have a set of layers, but the tooling for managing a modified
.templateconf is absent. As things are, it is useful only as a
fallback to the default core/poky template.

And if you propose such tooling, it somehow has to avoid modifying any
of the layers after they have been fetched. 'repo' allows you to do
this, this doesn't mean it would be an acceptable thing to do in
general. I did think about it, and I don't have good or even workable
ideas.

> With the above three methods available, this is easy as we just set
> the default in the .templateconf file.
>
> Now, if we remove support for .templateconf, how are we going to do it
> instead? I can of course change our wrapper around oe-build-init-env
> to do TEMPLATECONF=${TEMPLATECONF:-path/to/our/templates} and this
> might seem fine at first. However, this means that the wrapper will
> always set TEMPLATECONF and thus templateconf.cfg looses its meaning,
> which to the user means that he must now always specify TEMPLATECONF=
> if he wants to set it to something other than the default where before
> he only had to specify it the first time he initializes a new build
> environment.

I'm not sure I understand this. If the build environment has been
initialized before from templates, values of TEMPLATECONF or
templateconf.cfg become irrelevant, as the files in build/conf/ are
already in place, and scripts will skip copying them over from
templates. So you can continue to *not* have to specify TEMPLATECONF
when you want to reuse an existing build directory without changing
anything in it.

First time:
TEMPLATECONF=path/to/template . oe-init-build-env (will copy from template)

Second+ time:
. oe-init-build-env (will detect that no copying is needed, regardless
of where the template is set to)

Can you describe the sequence of commands where the problem arises? If
there is a problem here I absolutely want to see it.

> What _I_ would prefer to see, is a way to specify that the build
> shall use the available layers. I.e., rather than a huge number of
> static templates configurations, I would like to see the
> initialization script take the layers as arguments, with the default
> being anything that matches meta* (or maybe more correctly, anything
> that has a conf/layers.conf file). As for the local.conf file, in
> my world, this would be made up from fragment files in the layers.
> So for any layer that ends up in the bblayers.conf file, the layer
> would be searched for any local.conf.XX files that would be
> concatenated together in the order of XX. This is (obviously) very
> similar to what we do today, so I am of course biased, but I think
> this model is a lot more appealing than static templates files.
> YMMV.

Yes, config fragments are the next step. Whether they be automatic
like you prefer, or manual ('oe-setup-build add systemd') or both is
to be determined.

Alex

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Peter Kjellerstedt]

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 27 september 2022 21:58
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core
> <openembedded-core@lists.openembedded.org>
> Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode
> invalid paths for templates
> 
> On Tue, 27 Sept 2022 at 20:53, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> > Since we obviously use our own distribution, I am limited in
> > experience with other distributions. The one example I do know about,
> > which I have mentioned before, is WindRiver. See line 831-835 in
> > https://github.com/WindRiver-Labs/wrlinux-
> x/blob/WRLINUX_10_21_BASE/bin/setup.py
> > for an example of where they do basically the same as what we do.
> 
> Yes, they have to fix this too. And not by replacing .templateconf
> with TEMPLATECONF, but by writing a config directly, particularly
> using available tooling to compose bblayers.conf.

I am not too fond of the "writing the config directly" idea. The reason 
for that is that things change upstream (as you are well aware of, that's 
after all the whole reason for the discussion). This is why I prefer to 
go through the templates and basically only provide the list of layers 
to add. That way when things change upstream in the template files, we 
automatically follow. (And if it has not been obvious before, our wrapper 
uses the bblayers.conf.sample file from meta-poky and just replaces the 
list of layers with our list of layers. Which of course sounds like then 
we could just write the final config file directly, but we also want to 
continue to rely on oe-init-build-env for determining _where_ to write 
the final files.)

> Templates are
> intended for layers and version control, not as an extra intermediate
> step when setting up builds - the goal is to standardize build
> configuration management, and static templates is a starting point.
> 
> > I do not understand what the problem with .templateconf is? There is
> > a use case for being able to specify a default TEMPLATECONF, and that
> > is what .templateconf satisfies.
> 
> I'd say it does not satisfy the use case, as it relies on being next
> to oe-init-build-env, which in turn requires custom scripting. Show
> how to do this using standard tools in core/poky, write a patch for
> the official documentation, and then you have a point.

Well, up until now there have been no tools in OE/poky to do anything 
related to setting up the layers, so I will go by the tools that I 
know are actually being used.

With repo and git submodules it is simple to setup the .templateconf 
file to match the distribution. I cannot speak for kas, as I have never 
used it, but I believe it uses its own way of doing things. I have 
tried whisk, though it was a while ago, but if I remember correctly, 
it also wrote the files itself. And, as of yet, I have not tried your 
new tools, but I plan on doing that as soon as I have the time, though 
I am pretty sure they do not rely on .templateconf. ;)

> 
> > There are currently three ways to specify where the templates files
> > are (in order of preference):
> >
> > 1) TEMPLATECONF=
> > 2) templateconf.cfg
> > 3) .templateconf
> >
> > In our case, the templates are always in the same location, thus we
> > want to specify this as a default to give the developers a better
> > user experience.
> 
> I get this, it is useful to make a pointer to a 'default' template
> when you have a set of layers, but the tooling for managing a modified
> .templateconf is absent. As things are, it is useful only as a
> fallback to the default core/poky template.

Or anyone who sets up their distribution to mimic OE/Poky...

> And if you propose such tooling, it somehow has to avoid modifying any
> of the layers after they have been fetched. 'repo' allows you to do
> this, this doesn't mean it would be an acceptable thing to do in
> general. I did think about it, and I don't have good or even workable
> ideas.
> 
> > With the above three methods available, this is easy as we just set
> > the default in the .templateconf file.
> >
> > Now, if we remove support for .templateconf, how are we going to do it
> > instead? I can of course change our wrapper around oe-build-init-env
> > to do TEMPLATECONF=${TEMPLATECONF:-path/to/our/templates} and this
> > might seem fine at first. However, this means that the wrapper will
> > always set TEMPLATECONF and thus templateconf.cfg looses its meaning,
> > which to the user means that he must now always specify TEMPLATECONF=
> > if he wants to set it to something other than the default where before
> > he only had to specify it the first time he initializes a new build
> > environment.
> 
> I'm not sure I understand this. If the build environment has been
> initialized before from templates, values of TEMPLATECONF or
> templateconf.cfg become irrelevant, as the files in build/conf/ are
> already in place, and scripts will skip copying them over from
> templates. So you can continue to *not* have to specify TEMPLATECONF
> when you want to reuse an existing build directory without changing
> anything in it.
> 
> First time:
> TEMPLATECONF=path/to/template . oe-init-build-env (will copy from
> template)
> 
> Second+ time:
> . oe-init-build-env (will detect that no copying is needed, regardless
> of where the template is set to)
> 
> Can you describe the sequence of commands where the problem arises? If
> there is a problem here I absolutely want to see it.

Well, the real problem is that since the path _is_ stored in 
templateconf.cfg and this path takes precedence over the path in 
.templateconf, it causes problems if it is not updated, due to the 
new requirements on the stored path, whether it is actually used 
or not. :( This is what my patch tries to address.

However, having the path stored in templateconf.cfg is only really 
useful for anything if you delete one or more of the generated files 
so that they need to be regenerated. I assume that is why it was 
done, but I really do not know.

We use this feature in a help script we have, where you can have it 
update the config files by giving it an --update option, in which case 
it just deletes the old files in the config directory and then relies 
on oe-init-build-env to regenerate them. We could of course have solved 
this some other way, but since the feature is there, why not make use 
of it?

That said, having a simple way like that to update the config files 
is also something I would not mind seeing upstream. What we do in the 
local.conf is that we add a marker line at the end. Then when the 
user uses the --update option, we keep anything after that line but 
regenerate anything before it. So typically you add, e.g., MACHINE= 
at the end and then it is simple to update the file after an update 
to a new version of OE. This may not be the way local.conf is 
intended to be used, but I find it hard to use it if you actually 
uncomment any of the existing lines and then want to update the file 
to match a new version of OE. Thus I only use the existing lines as 
examples and do any modifications at the end.

But as I said, this is a convenience script that some of our users 
use and not part of the wrapper we have for oe-init-build-env.

> > What _I_ would prefer to see, is a way to specify that the build
> > shall use the available layers. I.e., rather than a huge number of
> > static templates configurations, I would like to see the
> > initialization script take the layers as arguments, with the default
> > being anything that matches meta* (or maybe more correctly, anything
> > that has a conf/layers.conf file). As for the local.conf file, in
> > my world, this would be made up from fragment files in the layers.
> > So for any layer that ends up in the bblayers.conf file, the layer
> > would be searched for any local.conf.XX files that would be
> > concatenated together in the order of XX. This is (obviously) very
> > similar to what we do today, so I am of course biased, but I think
> > this model is a lot more appealing than static templates files.
> > YMMV.
> 
> Yes, config fragments are the next step. Whether they be automatic
> like you prefer, or manual ('oe-setup-build add systemd') or both is
> to be determined.
> 
> Alex

//Peter

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

On Tue, 27 Sept 2022 at 23:31, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> I am not too fond of the "writing the config directly" idea. The reason
> for that is that things change upstream (as you are well aware of, that's
> after all the whole reason for the discussion). This is why I prefer to
> go through the templates and basically only provide the list of layers
> to add. That way when things change upstream in the template files, we
> automatically follow. (And if it has not been obvious before, our wrapper
> uses the bblayers.conf.sample file from meta-poky and just replaces the
> list of layers with our list of layers. Which of course sounds like then
> we could just write the final config file directly, but we also want to
> continue to rely on oe-init-build-env for determining _where_ to write
> the final files.)

Wait, wait, wait. If so, you can initialize using the template from
poky, then you get BBPATH or BUILDDIR placed in the environment (by
oe-init-build-env) telling where the config files are. So you can
write to them directly (e.g. local.conf), or use 'bitbake-layers
add-layer' (or 'remove-layer') to tweak the list of layers using
supported tooling -  after the initialization, but before any builds.
No?

Alex

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Peter Kjellerstedt]

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 27 september 2022 23:50
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core
> <openembedded-core@lists.openembedded.org>
> Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode
> invalid paths for templates
> 
> On Tue, 27 Sept 2022 at 23:31, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> > I am not too fond of the "writing the config directly" idea. The reason
> > for that is that things change upstream (as you are well aware of, that's
> > after all the whole reason for the discussion). This is why I prefer to
> > go through the templates and basically only provide the list of layers
> > to add. That way when things change upstream in the template files, we
> > automatically follow. (And if it has not been obvious before, our wrapper
> > uses the bblayers.conf.sample file from meta-poky and just replaces the
> > list of layers with our list of layers. Which of course sounds like then
> > we could just write the final config file directly, but we also want to
> > continue to rely on oe-init-build-env for determining _where_ to write
> > the final files.)
> 
> Wait, wait, wait. If so, you can initialize using the template from
> poky, then you get BBPATH or BUILDDIR placed in the environment (by
> oe-init-build-env) telling where the config files are. So you can
> write to them directly (e.g. local.conf), or use 'bitbake-layers
> add-layer' (or 'remove-layer') to tweak the list of layers using
> supported tooling -  after the initialization, but before any builds.
> No?
> 
> Alex

I am not sure if you are suggesting that the user shall run `bitbake-layers
add-layer` after sourcing oe-init-build-env, or if you mean that our wrapper 
should do it. 

If it is the former, then that is obviously not how we want things to work. 
The layers that are fetched shall by default be part of the build. That is 
an absolute requirement. There is no expectancy (or want) that the user 
shall have to add/remove the layers manually. 

If it is the latter, it wouldn't work (unless I yet again duplicate the 
code to figure out the build directory before sourcing the real 
oe-init-build-env) since before sourcing the real oe-init-build-env I 
cannot tell if there already is a bblayers.conf file in which case the 
wrapper should not add/remove any layers, and after sourcing the real 
oe-init-build-env it is too late as I then cannot distinguish between 
this being the first time oe-init-build-env is sourced or just a 
reinitialization of an already existing environment.

Our philosophy when we originally set this up was to keep it simple and 
as close to how OE/Poky works as possible. That way, if the developers 
have any questions and look at the Yocto documentation, the commands 
suggested there should work the same for us. And that is what we get 
from using the templates and letting the init scripts provided by OE do 
the work. The fact that we use the bblayers.conf.sample file from 
meta-poky rather than providing our own shouldn't change that. It is 
just a way to avoid having to manually keep our sample file in sync 
with what happens in OE Core.

//Peter

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

On Wed, 28 Sept 2022 at 11:54, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> If it is the latter, it wouldn't work (unless I yet again duplicate the
> code to figure out the build directory before sourcing the real
> oe-init-build-env) since before sourcing the real oe-init-build-env I
> cannot tell if there already is a bblayers.conf file in which case the
> wrapper should not add/remove any layers, and after sourcing the real
> oe-init-build-env it is too late as I then cannot distinguish between
> this being the first time oe-init-build-env is sourced or just a
> reinitialization of an already existing environment.

Yes this is the scenario I was thinking of: what should happen inside
the wrapper:

. oe-init-build-env (this takes the original poky template)
bitbake-layers add-layer meta-a
bitbake-layers add-layer meta-b
...

Why would you want to distinguish between whether this is against a
newly made dir
or an older one? I just checked: adding a layer that is already in
bblayers.conf does nothing.

I'm also unsure why are you so adamantly against checking if the build
dir already exists before the fact. It's not that difficult (either
it's relative to `pwd` or an absolute path), and not likely to change
upstream.

Alex

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Peter Kjellerstedt]

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 28 september 2022 13:00
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core
> <openembedded-core@lists.openembedded.org>
> Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode
> invalid paths for templates
> 
> On Wed, 28 Sept 2022 at 11:54, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> > If it is the latter, it wouldn't work (unless I yet again duplicate the
> > code to figure out the build directory before sourcing the real
> > oe-init-build-env) since before sourcing the real oe-init-build-env I
> > cannot tell if there already is a bblayers.conf file in which case the
> > wrapper should not add/remove any layers, and after sourcing the real
> > oe-init-build-env it is too late as I then cannot distinguish between
> > this being the first time oe-init-build-env is sourced or just a
> > reinitialization of an already existing environment.
> 
> Yes this is the scenario I was thinking of: what should happen inside
> the wrapper:
> 
> . oe-init-build-env (this takes the original poky template)
> bitbake-layers add-layer meta-a
> bitbake-layers add-layer meta-b
> ...
> 
> Why would you want to distinguish between whether this is against a
> newly made dir or an older one? I just checked: adding a layer that 
> is already in bblayers.conf does nothing.

Because, while we want the layers fetched to be present in bblayers.conf 
by default, it is still expected that developers during development can 
add and remove layers. And if a developer has, e.g., removed a layer, it 
is not expected that sourcing oe-init-build-env again should restore the 
removed layer.

> 
> I'm also unsure why are you so adamantly against checking if the build
> dir already exists before the fact. It's not that difficult (either
> it's relative to `pwd` or an absolute path), and not likely to change
> upstream.

Since our wrapper is expected to behave as oe-init-build-env it must 
also calculate the build directory as oe-init-build-env would. And no, 
it is not hard to do, but it means code duplication and that we are 
susceptible to upstream changes. It is not a major problem and it is 
the way I will solve this if I cannot get a change accepted upstream, 
but every such small addition adds to the maintenance burden.

> Alex

//Peter

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

On Wed, 28 Sept 2022 at 14:45, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> Because, while we want the layers fetched to be present in bblayers.conf
> by default, it is still expected that developers during development can
> add and remove layers. And if a developer has, e.g., removed a layer, it
> is not expected that sourcing oe-init-build-env again should restore the
> removed layer.

I wonder if we can somehow make oe-init-build-env indicate (in a way
that is programmatically robust, e.g. perhaps not a marker in stdout)
that it has initialized a new build directory. Ideas?

> Since our wrapper is expected to behave as oe-init-build-env it must
> also calculate the build directory as oe-init-build-env would. And no,
> it is not hard to do, but it means code duplication and that we are
> susceptible to upstream changes. It is not a major problem and it is
> the way I will solve this if I cannot get a change accepted upstream,
> but every such small addition adds to the maintenance burden.

All you need is:
os.path.exists(os.path.abspath(build_dir))
(probably abspath is unneeded either, it's just there to make clear
what the full path is in case it does not exist)
And no, that logic is not going to change upstream - it's far too
basic and pervasive.

Tweaking the list of layers by directly manipulating the config file
content when there are tools that do this for you is, I think, a worse
example of duplication.

Alex

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

On Wed, 28 Sept 2022 at 13:00, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:

> I'm also unsure why are you so adamantly against checking if the build
> dir already exists before the fact. It's not that difficult (either
> it's relative to `pwd` or an absolute path), and not likely to change
> upstream.

Which is a trivial one-liner in python:
os.path.exists(os.path.abspath(build_dir))

Alex

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

On Tue, 27 Sept 2022 at 18:06, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> I (and the small team I am part of) is the proxy between them and Yocto.
> Thus I try to address their needs by making sure new versions of Yocto
> continues to work for them without having 500+ developers facing the
> same problems. I.e., it is my job to work with upstream, which is what
> I am trying to do here.

Excuse me, but your job is not to 'work with upstream'. Your job is to
make damn sure 'upstream' continues to exist. Which means: go to your
management, describe the situation we're in (you have the numbers),
and ask that they allocate at least one developer to work in upstream
full time. Not drive-by patches to fix your specific product problems,
but upstream proper: something that makes the project sustainable,
e.g. fixing medium+/high bugs, working on new features, upgrading
recipes, sending pending patches upstream. This works never ends, and
no one wants to do it.

If you can do this, I promise to be more sympathetic to your
particular issues, but as things stand, it's rather difficult.

Alex

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Peter Kjellerstedt]

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 27 september 2022 19:05
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>; OE-core
> <openembedded-core@lists.openembedded.org>
> Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode
> invalid paths for templates
> 
> On Tue, 27 Sept 2022 at 18:06, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> > I (and the small team I am part of) is the proxy between them and Yocto.
> > Thus I try to address their needs by making sure new versions of Yocto
> > continues to work for them without having 500+ developers facing the
> > same problems. I.e., it is my job to work with upstream, which is what
> > I am trying to do here.
> 
> Excuse me, but your job is not to 'work with upstream'. Your job is to
> make damn sure 'upstream' continues to exist. Which means: go to your
> management, describe the situation we're in (you have the numbers),

Which is exactly why I have worked on and finally convinced my 
management that it is important for us to become a Yocto member, 
which we now are.

> and ask that they allocate at least one developer to work in upstream
> full time. 

That is harder. We have a hard enough time to recruit people to 
our tools team (which includes the build team that I am part of) 
as it is. For some reason, working with tools and build systems 
apparently is not as sexy as working with web development and AI.

> Not drive-by patches to fix your specific product problems,

I'm sorry to hear that you think all I do is drive-by patches. 
I thought my contributions were a little bit more appreciated 
than that. :(

> but upstream proper: something that makes the project sustainable,
> e.g. fixing medium+/high bugs, working on new features, upgrading
> recipes, sending pending patches upstream. This works never ends, and
> no one wants to do it.

And I can assure you that everyone appreciate the ungrateful 
work you and others do to keep things running. We really do.
 
> If you can do this, I promise to be more sympathetic to your
> particular issues, but as things stand, it's rather difficult.
> 
> Alex

//Peter

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

[Alexander Kanavin]

On Tue, 27 Sept 2022 at 21:22, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:

> > Not drive-by patches to fix your specific product problems,
>
> I'm sorry to hear that you think all I do is drive-by patches.
> I thought my contributions were a little bit more appreciated
> than that. :(

They are appreciated like any contribution is, but let me clarify:
anything that is not a part of a long term commitment to maintaining
some piece is a drive-by patch. If you're not sure what needs a
maintainer, let me remind you of that as well:
https://git.yoctoproject.org/poky/tree/MAINTAINERS.md

> > but upstream proper: something that makes the project sustainable,
> > e.g. fixing medium+/high bugs, working on new features, upgrading
> > recipes, sending pending patches upstream. This works never ends, and
> > no one wants to do it.
>
> And I can assure you that everyone appreciate the ungrateful
> work you and others do to keep things running. We really do.

You should understand the gravity of my situation as an independent
consultant, Peter. I constantly have to explain to my clients why
fixing their custom internal crap (that benefits only them) is not
always my top priority. Last week, I had to essentially blackmail a
client that pays real money, saying that I will not do any further
work for them, until they allow me to spend time upstream, and allow
me to bill them for it. I am still waiting for an answer. The timing
was poor as well, on the same day my friends and relatives in my home
country started getting draft notices to what is likely to be useless
deaths.

Alex