[PATCH 0/2] Extensible SDK improvements

[PATCH 0/2] Extensible SDK improvements

[Andrej Valek]

"add option to append lines into local.conf":
 - What about dropping "sdk_extraconf"?

Andrej Valek (2):
  populate_sdk_ext.bbclass: enable custom templateconf.cfg
  populate_sdk_ext.bbclass: add option to append lines into local.conf

 meta/classes/buildhistory.bbclass     |  4 ++--
 meta/classes/populate_sdk_ext.bbclass | 22 ++++++++++++++++++----
 2 files changed, 20 insertions(+), 6 deletions(-)

-- 
2.11.0

[PATCH 1/2] populate_sdk_ext: enable custom templateconf.cfg

[Andrej Valek]

Do not always override templateconf.cfg content. Add option to use
already existing file.

Signed-off-by: Andrej Valek <andrej.valek@siemens.com>
---
 meta/classes/populate_sdk_ext.bbclass | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 57fd29b1bb..9f26cfc131 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -388,9 +388,13 @@ python copy_buildsystem () {
         bb.utils.mkdirhier(os.path.join(baseoutpath, 'cache'))
         shutil.copyfile(builddir + '/cache/bb_unihashes.dat', baseoutpath + '/cache/bb_unihashes.dat')
 
-    # Write a templateconf.cfg
-    with open(baseoutpath + '/conf/templateconf.cfg', 'w') as f:
-        f.write('meta/conf\n')
+    # Use templateconf.cfg file from builddir if exists
+    if os.path.exists(builddir + '/conf/templateconf.cfg'):
+        shutil.copyfile(builddir + '/conf/templateconf.cfg', baseoutpath + '/conf/templateconf.cfg')
+    else:
+        # Write a templateconf.cfg
+        with open(baseoutpath + '/conf/templateconf.cfg', 'w') as f:
+            f.write('meta/conf\n')
 
     # Ensure any variables set from the external environment (by way of
     # BB_ENV_EXTRAWHITE) are set in the SDK's configuration
-- 
2.11.0

[PATCH 2/2] populate_sdk_ext: add option to append lines into local.conf

[Andrej Valek]

Add option to add new lines into local.conf content in ext_sdk case
via SDK_LOCAL_CONF_EXTRALIST variable.

Signed-off-by: Andrej Valek <andrej.valek@siemens.com>
---
 meta/classes/buildhistory.bbclass     |  4 ++--
 meta/classes/populate_sdk_ext.bbclass | 12 +++++++++++-
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index eb7295570d..a347811edc 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -757,8 +757,8 @@ def buildhistory_get_sdkvars(d):
     sdkvars = "DISTRO DISTRO_VERSION SDK_NAME SDK_VERSION SDKMACHINE SDKIMAGE_FEATURES BAD_RECOMMENDATIONS NO_RECOMMENDATIONS PACKAGE_EXCLUDE"
     if d.getVar('BB_CURRENTTASK') == 'populate_sdk_ext':
         # Extensible SDK uses some additional variables
-        sdkvars += " SDK_LOCAL_CONF_WHITELIST SDK_LOCAL_CONF_BLACKLIST SDK_INHERIT_BLACKLIST SDK_UPDATE_URL SDK_EXT_TYPE SDK_RECRDEP_TASKS SDK_INCLUDE_PKGDATA SDK_INCLUDE_TOOLCHAIN"
-    listvars = "SDKIMAGE_FEATURES BAD_RECOMMENDATIONS PACKAGE_EXCLUDE SDK_LOCAL_CONF_WHITELIST SDK_LOCAL_CONF_BLACKLIST SDK_INHERIT_BLACKLIST"
+        sdkvars += " SDK_LOCAL_CONF_EXTRALIST SDK_LOCAL_CONF_WHITELIST SDK_LOCAL_CONF_BLACKLIST SDK_INHERIT_BLACKLIST SDK_UPDATE_URL SDK_EXT_TYPE SDK_RECRDEP_TASKS SDK_INCLUDE_PKGDATA SDK_INCLUDE_TOOLCHAIN"
+    listvars = "SDKIMAGE_FEATURES BAD_RECOMMENDATIONS PACKAGE_EXCLUDE SDK_LOCAL_CONF_EXTRALIST SDK_LOCAL_CONF_WHITELIST SDK_LOCAL_CONF_BLACKLIST SDK_INHERIT_BLACKLIST"
     return outputvars(sdkvars, listvars, d)
 
 
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 9f26cfc131..2eccf52df1 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -25,6 +25,7 @@ SDK_INCLUDE_BUILDTOOLS ?= '1'
 
 SDK_RECRDEP_TASKS ?= ""
 
+SDK_LOCAL_CONF_EXTRALIST ?= ""
 SDK_LOCAL_CONF_WHITELIST ?= ""
 SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION \
                              BB_NUMBER_THREADS \
@@ -372,7 +373,16 @@ python copy_buildsystem () {
                     for line in xf:
                         f.write(line)
 
-            # If you define a sdk_extraconf() function then it can contain additional config
+            # Allow additional config through SDK_LOCAL_CONF_EXTRALIST
+            local_conf_extralist = (d.getVar('SDK_LOCAL_CONF_EXTRALIST') or '').split()
+            if local_conf_extralist:
+                for var in local_conf_extralist:
+                    value = d.getVar(var) or ""
+                    if value:
+                        f.write('%s = "%s"\n' % (var, value))
+                f.write('\n')
+
+            # If you define a sdk_extraconf variable then it can contain additional config
             # (Though this is awkward; sdk-extra.conf should probably be used instead)
             extraconf = (d.getVar('sdk_extraconf') or '').strip()
             if extraconf:
-- 
2.11.0

Re: [PATCH 0/2] Extensible SDK improvements

[Richard Purdie]

On Fri, 2020-03-06 at 16:32 +0100, Andrej Valek wrote:
> "add option to append lines into local.conf":
>  - What about dropping "sdk_extraconf"?

I'm curious what you didn't find useful about the other format and why
this version is better?

Cheers,

Richard

Re: [PATCH 0/2] Extensible SDK improvements

[Andrej Valek]

Hello Richard,

I can try to explain it in some examples.

I would like to add some extra (dynamic) variables into local.conf in
eSDK mode. I have found, that there are 2 possibilities (in
populate_sdk_ext.bbclass) sdk-extra.conf and sdk_extraconf.

- sdk-extra.conf
 - You have to have handle own function to write into sdk-extra.conf. I
think, this is not the right way for me.
- sdk_extraconf
 - This variable is fine, but there are some restrictions. There is no
easy way to add multiple variables including values.

I would like to have:
aaa = "valueA"
bbb = "valueB"
in local.conf file.

1. sdk_extraconf = 'aaa = "valueA"\nbbb = "valueB"'
 - this will write aaa = "valueA"\nbbb = "valueB"
2. python copy_buildsystem_prepend() {
  d.setVar('sdk_extraconf','aaa = "valueA"\nbbb = "valueB"')
}
 - This will write exactly, what you want, but you have to have an
copy_buildsystem_prepend and mentioned all variables with values defined
in your class/recipe.
3. sdk_extraconf = "# My notes about local.conf"
 - Yes, you can write some string into local.conf with this way.

So I have founded an way, how to add multiple variables without
explicitly specifying their values.

# format variables for sdk_extraconf
def write_sdk_vars(d):
  return '\n' + '\n'.join([var + ' = "' + d.getVar(var) + '"' for var in
d.getVar('SDK_EXTRACONF_VARS').split()])

SDK_EXTRACONF_VARS = "aaa bbb"
sdk_extraconf_append = " ${@write_sdk_vars(d)}"

It will do exactly what I want, but this way is little bit nasty. So I
have created a more complex solution for community usage.
I think, this patch adds more functionality also for other projects.
They can add only variables into list and the class will handle all the
rest.

Is it enough for the explanation?

Regards,
Andrej

On 2020-03-06 18:16, Richard Purdie wrote:
> On Fri, 2020-03-06 at 16:32 +0100, Andrej Valek wrote:
>> "add option to append lines into local.conf":
>>  - What about dropping "sdk_extraconf"?
> 
> I'm curious what you didn't find useful about the other format and why
> this version is better?
> 
> Cheers,
> 
> Richard
> 

Re: [PATCH 0/2] Extensible SDK improvements

[Andrej Valek]

Hello Richard,

Did you have some time to take a look on it?

It would be really helpful to have this feature in.
As I said I am trying to use this functionality for write some extra
variables into local.conf only in eSDK case. I am using a custom
oe-init-buildenv script, means that not all cases are covered by the
general one.

Thank you,
Andrej

On 2020-03-09 09:05, Andrej Valek wrote:
> Hello Richard,
> 
> I can try to explain it in some examples.
> 
> I would like to add some extra (dynamic) variables into local.conf in
> eSDK mode. I have found, that there are 2 possibilities (in
> populate_sdk_ext.bbclass) sdk-extra.conf and sdk_extraconf.
> 
> - sdk-extra.conf
>  - You have to have handle own function to write into sdk-extra.conf. I
> think, this is not the right way for me.
> - sdk_extraconf
>  - This variable is fine, but there are some restrictions. There is no
> easy way to add multiple variables including values.
> 
> I would like to have:
> aaa = "valueA"
> bbb = "valueB"
> in local.conf file.
> 
> 1. sdk_extraconf = 'aaa = "valueA"\nbbb = "valueB"'
>  - this will write aaa = "valueA"\nbbb = "valueB"
> 2. python copy_buildsystem_prepend() {
>   d.setVar('sdk_extraconf','aaa = "valueA"\nbbb = "valueB"')
> }
>  - This will write exactly, what you want, but you have to have an
> copy_buildsystem_prepend and mentioned all variables with values defined
> in your class/recipe.
> 3. sdk_extraconf = "# My notes about local.conf"
>  - Yes, you can write some string into local.conf with this way.
> 
> So I have founded an way, how to add multiple variables without
> explicitly specifying their values.
> 
> # format variables for sdk_extraconf
> def write_sdk_vars(d):
>   return '\n' + '\n'.join([var + ' = "' + d.getVar(var) + '"' for var in
> d.getVar('SDK_EXTRACONF_VARS').split()])
> 
> SDK_EXTRACONF_VARS = "aaa bbb"
> sdk_extraconf_append = " ${@write_sdk_vars(d)}"
> 
> It will do exactly what I want, but this way is little bit nasty. So I
> have created a more complex solution for community usage.
> I think, this patch adds more functionality also for other projects.
> They can add only variables into list and the class will handle all the
> rest.
> 
> Is it enough for the explanation?
> 
> Regards,
> Andrej
> 
> On 2020-03-06 18:16, Richard Purdie wrote:
>> On Fri, 2020-03-06 at 16:32 +0100, Andrej Valek wrote:
>>> "add option to append lines into local.conf":
>>>  - What about dropping "sdk_extraconf"?
>>
>> I'm curious what you didn't find useful about the other format and why
>> this version is better?
>>
>> Cheers,
>>
>> Richard
>>

Re: [PATCH 0/2] Extensible SDK improvements

[Richard Purdie]

Hi Andrej,

On Mon, 2020-03-09 at 09:05 +0100, Andrej Valek wrote:
> I can try to explain it in some examples.
> 
> I would like to add some extra (dynamic) variables into local.conf in
> eSDK mode. I have found, that there are 2 possibilities (in
> populate_sdk_ext.bbclass) sdk-extra.conf and sdk_extraconf.
> 
> - sdk-extra.conf
>  - You have to have handle own function to write into sdk-extra.conf. 
> I
> think, this is not the right way for me.
> - sdk_extraconf
>  - This variable is fine, but there are some restrictions. There is
> no
> easy way to add multiple variables including values.
> 
> I would like to have:
> aaa = "valueA"
> bbb = "valueB"
> in local.conf file.
> 
> 1. sdk_extraconf = 'aaa = "valueA"\nbbb = "valueB"'
>  - this will write aaa = "valueA"\nbbb = "valueB"
> 2. python copy_buildsystem_prepend() {
>   d.setVar('sdk_extraconf','aaa = "valueA"\nbbb = "valueB"')
> }
>  - This will write exactly, what you want, but you have to have an
> copy_buildsystem_prepend and mentioned all variables with values
> defined in your class/recipe.

Why do you have to set this in a copy_buildsystem_prepend ? Shouldn't
setting the variable in your distro config or class or somewhere else
work?

> 3. sdk_extraconf = "# My notes about local.conf"
>  - Yes, you can write some string into local.conf with this way.

I suspect you can also do:

sdk_extraconf () {
# My notes about local.conf
aaa = "valueA"
bbb = "valueB"
}

or something like that, I admit I've not tested it though.

> So I have founded an way, how to add multiple variables without
> explicitly specifying their values.
> 
> # format variables for sdk_extraconf
> def write_sdk_vars(d):
>   return '\n' + '\n'.join([var + ' = "' + d.getVar(var) + '"' for var
> in
> d.getVar('SDK_EXTRACONF_VARS').split()])
> 
> SDK_EXTRACONF_VARS = "aaa bbb"
> sdk_extraconf_append = " ${@write_sdk_vars(d)}"
> 
> It will do exactly what I want, but this way is little bit nasty. So
> I have created a more complex solution for community usage.
> I think, this patch adds more functionality also for other projects.
> They can add only variables into list and the class will handle all
> the
> rest.

I do see the problem with expansion. I suspect the code in
populate_sdk_ext should be:

extraconf = (d.getVar('sdk_extraconf', False) or '').strip()
instead of:
extraconf = (d.getVar('sdk_extraconf') or '').strip()

so that variables can be used as part of the expressions. It may be our
change to the defaults for getVar messed this up, or it may be we just
didn't correctly predict this problem.

> Is it enough for the explanation?

It helps, yes, thanks. What I don't want to do is add multiple
confusing mechanisms which do approximately the same thing slightly
differently. I'd much prefer one mechanism most people can use.

As such the original proposal still concerns me. I'd prefer to tweak
one of the existing ones (even merge them!) rather than add a third
slightly different one.

I do also still think some (but not all) of what you're describing
should be in your distro/classes rather than local.conf.

Is there a way we can improve the current code rather than adding more
options. Would the expansion change be enough?

Cheers,

Richard

Re: [PATCH 0/2] Extensible SDK improvements

[Andrej Valek]

Hello Richard,

Thank you for taking care of it. I didn't know, that is possible to use
"functions" in d.getVar case. Yes, I have tested it. And yes, what you
have mentioned is working, as we were expecting.

I will rather use my mechanism to just fill a list with variables. Then
I don't need to take a care, if there are fully expanded it or not.

So we can keep like it is.

But, yes, a small improvement for someone else could be, that we will
put a "False" there, but not for me.

Regards,
Andrej

On 2020-03-16 14:07, Richard Purdie wrote:
> Hi Andrej,
> 
> On Mon, 2020-03-09 at 09:05 +0100, Andrej Valek wrote:
>> I can try to explain it in some examples.
>>
>> I would like to add some extra (dynamic) variables into local.conf in
>> eSDK mode. I have found, that there are 2 possibilities (in
>> populate_sdk_ext.bbclass) sdk-extra.conf and sdk_extraconf.
>>
>> - sdk-extra.conf
>>  - You have to have handle own function to write into sdk-extra.conf. 
>> I
>> think, this is not the right way for me.
>> - sdk_extraconf
>>  - This variable is fine, but there are some restrictions. There is
>> no
>> easy way to add multiple variables including values.
>>
>> I would like to have:
>> aaa = "valueA"
>> bbb = "valueB"
>> in local.conf file.
>>
>> 1. sdk_extraconf = 'aaa = "valueA"\nbbb = "valueB"'
>>  - this will write aaa = "valueA"\nbbb = "valueB"
>> 2. python copy_buildsystem_prepend() {
>>   d.setVar('sdk_extraconf','aaa = "valueA"\nbbb = "valueB"')
>> }
>>  - This will write exactly, what you want, but you have to have an
>> copy_buildsystem_prepend and mentioned all variables with values
>> defined in your class/recipe.
> 
> Why do you have to set this in a copy_buildsystem_prepend ? Shouldn't
> setting the variable in your distro config or class or somewhere else
> work?
> 
>> 3. sdk_extraconf = "# My notes about local.conf"
>>  - Yes, you can write some string into local.conf with this way.
> 
> I suspect you can also do:
> 
> sdk_extraconf () {
> # My notes about local.conf
> aaa = "valueA"
> bbb = "valueB"
> }
> 
> or something like that, I admit I've not tested it though.
> 
>> So I have founded an way, how to add multiple variables without
>> explicitly specifying their values.
>>
>> # format variables for sdk_extraconf
>> def write_sdk_vars(d):
>>   return '\n' + '\n'.join([var + ' = "' + d.getVar(var) + '"' for var
>> in
>> d.getVar('SDK_EXTRACONF_VARS').split()])
>>
>> SDK_EXTRACONF_VARS = "aaa bbb"
>> sdk_extraconf_append = " ${@write_sdk_vars(d)}"
>>
>> It will do exactly what I want, but this way is little bit nasty. So
>> I have created a more complex solution for community usage.
>> I think, this patch adds more functionality also for other projects.
>> They can add only variables into list and the class will handle all
>> the
>> rest.
> 
> I do see the problem with expansion. I suspect the code in
> populate_sdk_ext should be:
> 
> extraconf = (d.getVar('sdk_extraconf', False) or '').strip()
> instead of:
> extraconf = (d.getVar('sdk_extraconf') or '').strip()
> 
> so that variables can be used as part of the expressions. It may be our
> change to the defaults for getVar messed this up, or it may be we just
> didn't correctly predict this problem.
> 
>> Is it enough for the explanation?
> 
> It helps, yes, thanks. What I don't want to do is add multiple
> confusing mechanisms which do approximately the same thing slightly
> differently. I'd much prefer one mechanism most people can use.
> 
> As such the original proposal still concerns me. I'd prefer to tweak
> one of the existing ones (even merge them!) rather than add a third
> slightly different one.
> 
> I do also still think some (but not all) of what you're describing
> should be in your distro/classes rather than local.conf.
> 
> Is there a way we can improve the current code rather than adding more
> options. Would the expansion change be enough?
> 
> Cheers,
> 
> Richard
> 
> 

Re: [PATCH 0/2] Extensible SDK improvements

[Adrian Freihofer]

Hi Richard,

We are going to handle that in our own layer. That's fine. Thank you
for the timely clarification anyway.

Regards,
Adrian 

On Sat, 2020-03-07 at 16:20 +0000, Richard Purdie wrote:
> On Sat, 2020-03-07 at 15:55 +0100, Adrian Freihofer wrote:
> > On Sat, 2020-03-07 at 12:40 +0000, Richard Purdie wrote:
> > > On Sat, 2020-03-07 at 12:54 +0100, Adrian Freihofer wrote:
> > > > Hi Richard,
> > > > 
> > > > We have found two already supported ways to copy variables from
> > > > the
> > > > bitbake environment local.conf to the eSDK local.conf
> > > > 
> > > > If a variable is defined in the local.conf bitbake environment,
> > > > SDK_LOCAL_CONF_WHITELIST and SDK_LOCAL_CONF_BLACKLIST can be used
> > > > to
> > > > add it to the local.conf eSDK file.
> > > > 
> > > > If a variable should be statically defined for the eSDK but not
> > > > for
> > > > the
> > > > bitbake environment, sdk-extra.conf is useful.
> > > > 
> > > > Now we would like to add a third way to add variables which are
> > > > dynamically calculated by bitbake but need to be statically added
> > > > to
> > > > the eSDK local.conf. For example we would like to support
> > > > something
> > > > like that:
> > > > 
> > > > def get_version_from_git(d):
> > > >     version = d.getVar("GIT_VERSION", True)
> > > >     if version:
> > > >         return version                    # runs in eSDK
> > > >     else:
> > > >         return bb.process.run("git...     # runs in bitbake
> > > > 
> > > > GIT_VERSION := "${@get_version_from_git(d)}"
> > > > 
> > > > SDK_LOCAL_CONF_EXTRALIST_append = " GIT_VERSION"
> > > 
> > > This worries me a bit since it means the eSDK and the "real" build
> > > can
> > > behave differently. I appreciate that can happen even with the
> > > other
> > > variables and means of setting them but this takes it to a new
> > > level.
> > > 
> > That I understand. The usage of the SDK_LOCAL_CONF_EXTRALIST would be
> > very specific. Wrong usage would lead to a broken sstate in the eSDK.
> > 
> > > Ultimately I think we're aiming to have normal builds convert into
> > > an
> > > eSDK and vice versa more easily. This seems to pull us further away
> > > from that :/.
> > > 
> > > What is the reasoning for having them behaving differently?
> > 
> > Our goal is to equate the eSDK behavior with the behavior of the real
> > build, also for the example with the GIT_VERSION, which bitbake and
> > git
> > will dynamically evaluate at eSDK build time.
> > 
> > Suppose we want to compile the GIT_VERSION (last tag) from poky
> > without
> > any manual steps into the firmware. bitbake can simply call
> > 
> > $ describe git --tags --dirty
> > uninative-2,8-74-g56446f4570-dirty
> > 
> > With Bitbake the variable can change. But in eSDK the GIT_VERSION
> > must
> > be a constant. The above function behaves like a constant if
> > GIT_VERSION is defined in the local.conf for example. But it has a
> > dynamic behavior if GIT_VERSION is undefined. The only missing part
> > in
> > the current Poky, is a way to automatically write the value to the
> > local.conf of the eSDK. I don't think this would be much different
> > than
> > the already existing sdk-extra.conf file.
> 
> I've been thinking about this further. Why is any of this in local.conf
> in the first place?
> 
> I'd suggest the bulk of it should be in your distro or class files?
> 
> I understand we do need some simple changes to local.conf in many cases
> but any complex logic really should not be there. The above does look
> like more complex logic to me...
> 
> I'm therefore leaning towards saying no to this patch, its just going
> to cause us problems in the future.
> 
> Cheers,
> 
> Richard
> 
> 
> 

Re: [PATCH 0/2] Extensible SDK improvements

[Richard Purdie]

On Sat, 2020-03-07 at 15:55 +0100, Adrian Freihofer wrote:
> On Sat, 2020-03-07 at 12:40 +0000, Richard Purdie wrote:
> > On Sat, 2020-03-07 at 12:54 +0100, Adrian Freihofer wrote:
> > > Hi Richard,
> > > 
> > > We have found two already supported ways to copy variables from
> > > the
> > > bitbake environment local.conf to the eSDK local.conf
> > > 
> > > If a variable is defined in the local.conf bitbake environment,
> > > SDK_LOCAL_CONF_WHITELIST and SDK_LOCAL_CONF_BLACKLIST can be used
> > > to
> > > add it to the local.conf eSDK file.
> > > 
> > > If a variable should be statically defined for the eSDK but not
> > > for
> > > the
> > > bitbake environment, sdk-extra.conf is useful.
> > > 
> > > Now we would like to add a third way to add variables which are
> > > dynamically calculated by bitbake but need to be statically added
> > > to
> > > the eSDK local.conf. For example we would like to support
> > > something
> > > like that:
> > > 
> > > def get_version_from_git(d):
> > >     version = d.getVar("GIT_VERSION", True)
> > >     if version:
> > >         return version                    # runs in eSDK
> > >     else:
> > >         return bb.process.run("git...     # runs in bitbake
> > > 
> > > GIT_VERSION := "${@get_version_from_git(d)}"
> > > 
> > > SDK_LOCAL_CONF_EXTRALIST_append = " GIT_VERSION"
> > 
> > This worries me a bit since it means the eSDK and the "real" build
> > can
> > behave differently. I appreciate that can happen even with the
> > other
> > variables and means of setting them but this takes it to a new
> > level.
> > 
> That I understand. The usage of the SDK_LOCAL_CONF_EXTRALIST would be
> very specific. Wrong usage would lead to a broken sstate in the eSDK.
> 
> > Ultimately I think we're aiming to have normal builds convert into
> > an
> > eSDK and vice versa more easily. This seems to pull us further away
> > from that :/.
> > 
> > What is the reasoning for having them behaving differently?
> 
> Our goal is to equate the eSDK behavior with the behavior of the real
> build, also for the example with the GIT_VERSION, which bitbake and
> git
> will dynamically evaluate at eSDK build time.
> 
> Suppose we want to compile the GIT_VERSION (last tag) from poky
> without
> any manual steps into the firmware. bitbake can simply call
> 
> $ describe git --tags --dirty
> uninative-2,8-74-g56446f4570-dirty
> 
> With Bitbake the variable can change. But in eSDK the GIT_VERSION
> must
> be a constant. The above function behaves like a constant if
> GIT_VERSION is defined in the local.conf for example. But it has a
> dynamic behavior if GIT_VERSION is undefined. The only missing part
> in
> the current Poky, is a way to automatically write the value to the
> local.conf of the eSDK. I don't think this would be much different
> than
> the already existing sdk-extra.conf file.

I've been thinking about this further. Why is any of this in local.conf
in the first place?

I'd suggest the bulk of it should be in your distro or class files?

I understand we do need some simple changes to local.conf in many cases
but any complex logic really should not be there. The above does look
like more complex logic to me...

I'm therefore leaning towards saying no to this patch, its just going
to cause us problems in the future.

Cheers,

Richard

Re: [PATCH 0/2] Extensible SDK improvements

[Adrian Freihofer]

On Sat, 2020-03-07 at 12:40 +0000, Richard Purdie wrote:
> On Sat, 2020-03-07 at 12:54 +0100, Adrian Freihofer wrote:
> > Hi Richard,
> > 
> > We have found two already supported ways to copy variables from the
> > bitbake environment local.conf to the eSDK local.conf
> > 
> > If a variable is defined in the local.conf bitbake environment,
> > SDK_LOCAL_CONF_WHITELIST and SDK_LOCAL_CONF_BLACKLIST can be used to
> > add it to the local.conf eSDK file.
> > 
> > If a variable should be statically defined for the eSDK but not for
> > the
> > bitbake environment, sdk-extra.conf is useful.
> > 
> > Now we would like to add a third way to add variables which are
> > dynamically calculated by bitbake but need to be statically added to
> > the eSDK local.conf. For example we would like to support something
> > like that:
> > 
> > def get_version_from_git(d):
> >     version = d.getVar("GIT_VERSION", True)
> >     if version:
> >         return version                    # runs in eSDK
> >     else:
> >         return bb.process.run("git...     # runs in bitbake
> > 
> > GIT_VERSION := "${@get_version_from_git(d)}"
> > 
> > SDK_LOCAL_CONF_EXTRALIST_append = " GIT_VERSION"
> 
> This worries me a bit since it means the eSDK and the "real" build can
> behave differently. I appreciate that can happen even with the other
> variables and means of setting them but this takes it to a new level.
> 
That I understand. The usage of the SDK_LOCAL_CONF_EXTRALIST would be
very specific. Wrong usage would lead to a broken sstate in the eSDK.

> Ultimately I think we're aiming to have normal builds convert into an
> eSDK and vice versa more easily. This seems to pull us further away
> from that :/.
> 
> What is the reasoning for having them behaving differently?

Our goal is to equate the eSDK behavior with the behavior of the real
build, also for the example with the GIT_VERSION, which bitbake and git
will dynamically evaluate at eSDK build time.

Suppose we want to compile the GIT_VERSION (last tag) from poky without
any manual steps into the firmware. bitbake can simply call

$ describe git --tags --dirty
uninative-2,8-74-g56446f4570-dirty

With Bitbake the variable can change. But in eSDK the GIT_VERSION must
be a constant. The above function behaves like a constant if
GIT_VERSION is defined in the local.conf for example. But it has a
dynamic behavior if GIT_VERSION is undefined. The only missing part in
the current Poky, is a way to automatically write the value to the
local.conf of the eSDK. I don't think this would be much different than
the already existing sdk-extra.conf file.

Regards,
Adrian

> Cheers,
> 
> Richard
> 
> 
> 

Re: [PATCH 0/2] Extensible SDK improvements

[Richard Purdie]

On Sat, 2020-03-07 at 12:54 +0100, Adrian Freihofer wrote:
> Hi Richard,
> 
> We have found two already supported ways to copy variables from the
> bitbake environment local.conf to the eSDK local.conf
> 
> If a variable is defined in the local.conf bitbake environment,
> SDK_LOCAL_CONF_WHITELIST and SDK_LOCAL_CONF_BLACKLIST can be used to
> add it to the local.conf eSDK file.
> 
> If a variable should be statically defined for the eSDK but not for
> the
> bitbake environment, sdk-extra.conf is useful.
> 
> Now we would like to add a third way to add variables which are
> dynamically calculated by bitbake but need to be statically added to
> the eSDK local.conf. For example we would like to support something
> like that:
> 
> def get_version_from_git(d):
>     version = d.getVar("GIT_VERSION", True)
>     if version:
>         return version                    # runs in eSDK
>     else:
>         return bb.process.run("git...     # runs in bitbake
> 
> GIT_VERSION := "${@get_version_from_git(d)}"
> 
> SDK_LOCAL_CONF_EXTRALIST_append = " GIT_VERSION"

This worries me a bit since it means the eSDK and the "real" build can
behave differently. I appreciate that can happen even with the other
variables and means of setting them but this takes it to a new level.

Ultimately I think we're aiming to have normal builds convert into an
eSDK and vice versa more easily. This seems to pull us further away
from that :/.

What is the reasoning for having them behaving differently?

Cheers,

Richard

Re: [PATCH 0/2] Extensible SDK improvements

[Adrian Freihofer]

Hi Richard,

We have found two already supported ways to copy variables from the
bitbake environment local.conf to the eSDK local.conf

If a variable is defined in the local.conf bitbake environment,
SDK_LOCAL_CONF_WHITELIST and SDK_LOCAL_CONF_BLACKLIST can be used to
add it to the local.conf eSDK file.

If a variable should be statically defined for the eSDK but not for the
bitbake environment, sdk-extra.conf is useful.

Now we would like to add a third way to add variables which are
dynamically calculated by bitbake but need to be statically added to
the eSDK local.conf. For example we would like to support something
like that:

def get_version_from_git(d):
    version = d.getVar("GIT_VERSION", True)
    if version:
        return version                    # runs in eSDK
    else:
        return bb.process.run("git...     # runs in bitbake

GIT_VERSION := "${@get_version_from_git(d)}"

SDK_LOCAL_CONF_EXTRALIST_append = " GIT_VERSION"

Regards,
Adrian