[PATCH] glib-2.0: replace THISDIR instead of COREBASE in find_meson_cross_files --cross-file paths

[PATCH] glib-2.0: replace THISDIR instead of COREBASE in find_meson_cross_files --cross-file paths

[Martin Jansa]

* COREBASE works correctly only as long as this recipe is inside of
  COREBASE directory and leaves the full absolute paths if you copy
  this recipe into your own layer (e.g. to backport newer version)

* use THISDIR as all FILESPATH should be relative to that, unless
  there is a .bbappend not only extending FILESPATH, but also adding
  meson.cross.d files there (at which point it also didn't work with
  COREBASE replacements as well already) - show an ERROR in such case,
  I don't think it's common scenario to spend more time to support that.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
index 15e9dbaf6f..882a89da7a 100644
--- a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
+++ b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
@@ -31,7 +31,7 @@ def find_meson_cross_files(d):
     if bb.data.inherits_class('native', d):
         return ""
 
-    corebase = d.getVar("COREBASE")
+    thisdir = d.getVar("THISDIR")
     import collections
     sitedata = siteinfo_data(d)
     # filename -> found
@@ -39,7 +39,11 @@ def find_meson_cross_files(d):
     for path in d.getVar("FILESPATH").split(":"):
         for element in sitedata:
             filename = os.path.normpath(os.path.join(path, "meson.cross.d", element))
-            files[filename.replace(corebase, "${COREBASE}")] = os.path.exists(filename)
+            sanitized_path = filename.replace(thisdir, "${THISDIR}")
+            if sanitized_path == filename:
+                bb.error("Cannot add '%s' to --cross-file, because it's not relative to THISDIR '%s' and sstate signature would contain this full path" % (filename, thisdir))
+                continue
+            files[filename.replace(thisdir, "${THISDIR}")] = os.path.exists(filename)
 
     items = ["--cross-file=" + k for k,v in files.items() if v]
     d.appendVar("EXTRA_OEMESON", " " + " ".join(items))
-- 
2.27.0

Re: [OE-core] [PATCH] glib-2.0: replace THISDIR instead of COREBASE in find_meson_cross_files --cross-file paths

[Anibal Limon]

[-- Attachment #1: Type: text/plain, Size: 2859 bytes --]

This breaks our builds,

...
ERROR: Cannot add
'/home/builds/oe-rpb-master/layers/openembedded-core/meta/recipes-core/glib-2.0/files/meson.cross.d/common'
to --cross-file, because it's not relative to THISDIR
'/home/builds/oe-rpb-master/build-410c-2/conf/../../layers/openembedded-core/meta/recipes-core/glib-2.0'
and sstate signature would contain this full path
...

In,

https://ci.linaro.org/job/lt-qcom-openembedded-rpb-master/DISTRO=rpb,MACHINE=dragonboard-410c,label=docker-buster-amd64/470/console

Regards,
Anibal

On Thu, 25 Feb 2021 at 11:45, Martin Jansa <Martin.Jansa@gmail.com> wrote:

> * COREBASE works correctly only as long as this recipe is inside of
>   COREBASE directory and leaves the full absolute paths if you copy
>   this recipe into your own layer (e.g. to backport newer version)
>
> * use THISDIR as all FILESPATH should be relative to that, unless
>   there is a .bbappend not only extending FILESPATH, but also adding
>   meson.cross.d files there (at which point it also didn't work with
>   COREBASE replacements as well already) - show an ERROR in such case,
>   I don't think it's common scenario to spend more time to support that.
>
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
> b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
> index 15e9dbaf6f..882a89da7a 100644
> --- a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
> +++ b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
> @@ -31,7 +31,7 @@ def find_meson_cross_files(d):
>      if bb.data.inherits_class('native', d):
>          return ""
>
> -    corebase = d.getVar("COREBASE")
> +    thisdir = d.getVar("THISDIR")
>      import collections
>      sitedata = siteinfo_data(d)
>      # filename -> found
> @@ -39,7 +39,11 @@ def find_meson_cross_files(d):
>      for path in d.getVar("FILESPATH").split(":"):
>          for element in sitedata:
>              filename = os.path.normpath(os.path.join(path,
> "meson.cross.d", element))
> -            files[filename.replace(corebase, "${COREBASE}")] =
> os.path.exists(filename)
> +            sanitized_path = filename.replace(thisdir, "${THISDIR}")
> +            if sanitized_path == filename:
> +                bb.error("Cannot add '%s' to --cross-file, because it's
> not relative to THISDIR '%s' and sstate signature would contain this full
> path" % (filename, thisdir))
> +                continue
> +            files[filename.replace(thisdir, "${THISDIR}")] =
> os.path.exists(filename)
>
>      items = ["--cross-file=" + k for k,v in files.items() if v]
>      d.appendVar("EXTRA_OEMESON", " " + " ".join(items))
> --
> 2.27.0
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 4208 bytes --]

Re: [OE-core] [PATCH] glib-2.0: replace THISDIR instead of COREBASE in find_meson_cross_files --cross-file paths

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3436 bytes --]

what's the value of COREBASE in that build? Wasn't full path already in
your sstate signatures - in that case this change didn't break your builds,
just highlighted an issue in your builds which was already there.

We can fix that by calling os.path.normpath on THISDIR, but I would like to
know the reason why THISDIR is set this way for you (also to reproduce it
locally to verify the fix).




On Tue, Mar 2, 2021 at 6:14 PM Anibal Limon <anibal.limon@linaro.org> wrote:

> This breaks our builds,
>
> ...
> ERROR: Cannot add
> '/home/builds/oe-rpb-master/layers/openembedded-core/meta/recipes-core/glib-2.0/files/meson.cross.d/common'
> to --cross-file, because it's not relative to THISDIR
> '/home/builds/oe-rpb-master/build-410c-2/conf/../../layers/openembedded-core/meta/recipes-core/glib-2.0'
> and sstate signature would contain this full path
> ...
>
> In,
>
>
> https://ci.linaro.org/job/lt-qcom-openembedded-rpb-master/DISTRO=rpb,MACHINE=dragonboard-410c,label=docker-buster-amd64/470/console
>
> Regards,
> Anibal
>
> On Thu, 25 Feb 2021 at 11:45, Martin Jansa <Martin.Jansa@gmail.com> wrote:
>
>> * COREBASE works correctly only as long as this recipe is inside of
>>   COREBASE directory and leaves the full absolute paths if you copy
>>   this recipe into your own layer (e.g. to backport newer version)
>>
>> * use THISDIR as all FILESPATH should be relative to that, unless
>>   there is a .bbappend not only extending FILESPATH, but also adding
>>   meson.cross.d files there (at which point it also didn't work with
>>   COREBASE replacements as well already) - show an ERROR in such case,
>>   I don't think it's common scenario to spend more time to support that.
>>
>> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
>> ---
>>  meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 8 ++++++--
>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>> b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>> index 15e9dbaf6f..882a89da7a 100644
>> --- a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>> +++ b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>> @@ -31,7 +31,7 @@ def find_meson_cross_files(d):
>>      if bb.data.inherits_class('native', d):
>>          return ""
>>
>> -    corebase = d.getVar("COREBASE")
>> +    thisdir = d.getVar("THISDIR")
>>      import collections
>>      sitedata = siteinfo_data(d)
>>      # filename -> found
>> @@ -39,7 +39,11 @@ def find_meson_cross_files(d):
>>      for path in d.getVar("FILESPATH").split(":"):
>>          for element in sitedata:
>>              filename = os.path.normpath(os.path.join(path,
>> "meson.cross.d", element))
>> -            files[filename.replace(corebase, "${COREBASE}")] =
>> os.path.exists(filename)
>> +            sanitized_path = filename.replace(thisdir, "${THISDIR}")
>> +            if sanitized_path == filename:
>> +                bb.error("Cannot add '%s' to --cross-file, because it's
>> not relative to THISDIR '%s' and sstate signature would contain this full
>> path" % (filename, thisdir))
>> +                continue
>> +            files[filename.replace(thisdir, "${THISDIR}")] =
>> os.path.exists(filename)
>>
>>      items = ["--cross-file=" + k for k,v in files.items() if v]
>>      d.appendVar("EXTRA_OEMESON", " " + " ".join(items))
>> --
>> 2.27.0
>>
>>
>> 
>>
>>

[-- Attachment #2: Type: text/html, Size: 5077 bytes --]

Re: [OE-core] [PATCH] glib-2.0: replace THISDIR instead of COREBASE in find_meson_cross_files --cross-file paths

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4026 bytes --]

I guess this relative path comes from OEROOT setting in:
https://github.com/96boards/oe-rpb-manifest/blob/master/conf/bblayers.conf#L4

maybe you can call os.path.normpath there as COREBASE is doing in:
meta/conf/layer.conf:COREBASE = '${@os.path.normpath("${LAYERDIR}/../")}'

That should help with both sstate reuse as well as this error.

On Tue, Mar 2, 2021 at 6:33 PM Martin Jansa via lists.openembedded.org
<Martin.Jansa=gmail.com@lists.openembedded.org> wrote:

> what's the value of COREBASE in that build? Wasn't full path already in
> your sstate signatures - in that case this change didn't break your builds,
> just highlighted an issue in your builds which was already there.
>
> We can fix that by calling os.path.normpath on THISDIR, but I would like
> to know the reason why THISDIR is set this way for you (also to reproduce
> it locally to verify the fix).
>
>
>
>
> On Tue, Mar 2, 2021 at 6:14 PM Anibal Limon <anibal.limon@linaro.org>
> wrote:
>
>> This breaks our builds,
>>
>> ...
>> ERROR: Cannot add
>> '/home/builds/oe-rpb-master/layers/openembedded-core/meta/recipes-core/glib-2.0/files/meson.cross.d/common'
>> to --cross-file, because it's not relative to THISDIR
>> '/home/builds/oe-rpb-master/build-410c-2/conf/../../layers/openembedded-core/meta/recipes-core/glib-2.0'
>> and sstate signature would contain this full path
>> ...
>>
>> In,
>>
>>
>> https://ci.linaro.org/job/lt-qcom-openembedded-rpb-master/DISTRO=rpb,MACHINE=dragonboard-410c,label=docker-buster-amd64/470/console
>>
>> Regards,
>> Anibal
>>
>> On Thu, 25 Feb 2021 at 11:45, Martin Jansa <Martin.Jansa@gmail.com>
>> wrote:
>>
>>> * COREBASE works correctly only as long as this recipe is inside of
>>>   COREBASE directory and leaves the full absolute paths if you copy
>>>   this recipe into your own layer (e.g. to backport newer version)
>>>
>>> * use THISDIR as all FILESPATH should be relative to that, unless
>>>   there is a .bbappend not only extending FILESPATH, but also adding
>>>   meson.cross.d files there (at which point it also didn't work with
>>>   COREBASE replacements as well already) - show an ERROR in such case,
>>>   I don't think it's common scenario to spend more time to support that.
>>>
>>> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
>>> ---
>>>  meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 8 ++++++--
>>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> index 15e9dbaf6f..882a89da7a 100644
>>> --- a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> +++ b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> @@ -31,7 +31,7 @@ def find_meson_cross_files(d):
>>>      if bb.data.inherits_class('native', d):
>>>          return ""
>>>
>>> -    corebase = d.getVar("COREBASE")
>>> +    thisdir = d.getVar("THISDIR")
>>>      import collections
>>>      sitedata = siteinfo_data(d)
>>>      # filename -> found
>>> @@ -39,7 +39,11 @@ def find_meson_cross_files(d):
>>>      for path in d.getVar("FILESPATH").split(":"):
>>>          for element in sitedata:
>>>              filename = os.path.normpath(os.path.join(path,
>>> "meson.cross.d", element))
>>> -            files[filename.replace(corebase, "${COREBASE}")] =
>>> os.path.exists(filename)
>>> +            sanitized_path = filename.replace(thisdir, "${THISDIR}")
>>> +            if sanitized_path == filename:
>>> +                bb.error("Cannot add '%s' to --cross-file, because it's
>>> not relative to THISDIR '%s' and sstate signature would contain this full
>>> path" % (filename, thisdir))
>>> +                continue
>>> +            files[filename.replace(thisdir, "${THISDIR}")] =
>>> os.path.exists(filename)
>>>
>>>      items = ["--cross-file=" + k for k,v in files.items() if v]
>>>      d.appendVar("EXTRA_OEMESON", " " + " ".join(items))
>>> --
>>> 2.27.0
>>>
>>>
>>>
>>>
>>>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 6102 bytes --]

Re: [OE-core] [PATCH] glib-2.0: replace THISDIR instead of COREBASE in find_meson_cross_files --cross-file paths

[Geoff Parker]

[-- Attachment #1: Type: text/plain, Size: 3589 bytes --]

Same here. Our build artifacts get created but, I get a few thousand of
these errors:

ERROR: Cannot add
'/home/geoff/yocto-tegra/poky/meta/recipes-core/glib-2.0/glib-2.0/meson.cross.d/arm-64'
to --cross-file, because it's not relative to THISDIR '/
home/geoff/yocto-tegra/poky/build-ccu3/../meta/recipes-core/glib-2.0' and
sstate signature would contain this full path

 Since the bitbake return code is 1 our CI pipeline does not deploy the
build artifacts.

Would bb.warn() be more appropriate to use for this change than bb.error()?

On Tue, Mar 2, 2021 at 9:14 AM Anibal Limon <anibal.limon@linaro.org> wrote:

> This breaks our builds,
>
> ...
> ERROR: Cannot add
> '/home/builds/oe-rpb-master/layers/openembedded-core/meta/recipes-core/glib-2.0/files/meson.cross.d/common'
> to --cross-file, because it's not relative to THISDIR
> '/home/builds/oe-rpb-master/build-410c-2/conf/../../layers/openembedded-core/meta/recipes-core/glib-2.0'
> and sstate signature would contain this full path
> ...
>
> In,
>
>
> https://ci.linaro.org/job/lt-qcom-openembedded-rpb-master/DISTRO=rpb,MACHINE=dragonboard-410c,label=docker-buster-amd64/470/console
>
> Regards,
> Anibal
>
> On Thu, 25 Feb 2021 at 11:45, Martin Jansa <Martin.Jansa@gmail.com> wrote:
>
>> * COREBASE works correctly only as long as this recipe is inside of
>>   COREBASE directory and leaves the full absolute paths if you copy
>>   this recipe into your own layer (e.g. to backport newer version)
>>
>> * use THISDIR as all FILESPATH should be relative to that, unless
>>   there is a .bbappend not only extending FILESPATH, but also adding
>>   meson.cross.d files there (at which point it also didn't work with
>>   COREBASE replacements as well already) - show an ERROR in such case,
>>   I don't think it's common scenario to spend more time to support that.
>>
>> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
>> ---
>>  meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 8 ++++++--
>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>> b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>> index 15e9dbaf6f..882a89da7a 100644
>> --- a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>> +++ b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>> @@ -31,7 +31,7 @@ def find_meson_cross_files(d):
>>      if bb.data.inherits_class('native', d):
>>          return ""
>>
>> -    corebase = d.getVar("COREBASE")
>> +    thisdir = d.getVar("THISDIR")
>>      import collections
>>      sitedata = siteinfo_data(d)
>>      # filename -> found
>> @@ -39,7 +39,11 @@ def find_meson_cross_files(d):
>>      for path in d.getVar("FILESPATH").split(":"):
>>          for element in sitedata:
>>              filename = os.path.normpath(os.path.join(path,
>> "meson.cross.d", element))
>> -            files[filename.replace(corebase, "${COREBASE}")] =
>> os.path.exists(filename)
>> +            sanitized_path = filename.replace(thisdir, "${THISDIR}")
>> +            if sanitized_path == filename:
>> +                bb.error("Cannot add '%s' to --cross-file, because it's
>> not relative to THISDIR '%s' and sstate signature would contain this full
>> path" % (filename, thisdir))
>> +                continue
>> +            files[filename.replace(thisdir, "${THISDIR}")] =
>> os.path.exists(filename)
>>
>>      items = ["--cross-file=" + k for k,v in files.items() if v]
>>      d.appendVar("EXTRA_OEMESON", " " + " ".join(items))
>> --
>> 2.27.0
>>
>>
>>
>>
>>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 5399 bytes --]

Re: [OE-core] [PATCH] glib-2.0: replace THISDIR instead of COREBASE in find_meson_cross_files --cross-file paths

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4206 bytes --]

Geoff,

the same question for you, what's the value of COREBASE in your build and
what's the value of --cross-file in your glib-2.0 do_configure sstate
signature?

If both of these are valid issues in sstate signatures then fixing your
build configuration is IMHO better than ignoring a warning from bb.warn().
Or at least for this specific case we would need to find some more robust
solution than this string replace.

On Tue, Mar 2, 2021 at 6:43 PM Geoff Parker <geoffhp@gmail.com> wrote:

> Same here. Our build artifacts get created but, I get a few thousand of
> these errors:
>
> ERROR: Cannot add
> '/home/geoff/yocto-tegra/poky/meta/recipes-core/glib-2.0/glib-2.0/meson.cross.d/arm-64'
> to --cross-file, because it's not relative to THISDIR '/
> home/geoff/yocto-tegra/poky/build-ccu3/../meta/recipes-core/glib-2.0' and
> sstate signature would contain this full path
>
>  Since the bitbake return code is 1 our CI pipeline does not deploy the
> build artifacts.
>
> Would bb.warn() be more appropriate to use for this change than bb.error()?
>
> On Tue, Mar 2, 2021 at 9:14 AM Anibal Limon <anibal.limon@linaro.org>
> wrote:
>
>> This breaks our builds,
>>
>> ...
>> ERROR: Cannot add
>> '/home/builds/oe-rpb-master/layers/openembedded-core/meta/recipes-core/glib-2.0/files/meson.cross.d/common'
>> to --cross-file, because it's not relative to THISDIR
>> '/home/builds/oe-rpb-master/build-410c-2/conf/../../layers/openembedded-core/meta/recipes-core/glib-2.0'
>> and sstate signature would contain this full path
>> ...
>>
>> In,
>>
>>
>> https://ci.linaro.org/job/lt-qcom-openembedded-rpb-master/DISTRO=rpb,MACHINE=dragonboard-410c,label=docker-buster-amd64/470/console
>>
>> Regards,
>> Anibal
>>
>> On Thu, 25 Feb 2021 at 11:45, Martin Jansa <Martin.Jansa@gmail.com>
>> wrote:
>>
>>> * COREBASE works correctly only as long as this recipe is inside of
>>>   COREBASE directory and leaves the full absolute paths if you copy
>>>   this recipe into your own layer (e.g. to backport newer version)
>>>
>>> * use THISDIR as all FILESPATH should be relative to that, unless
>>>   there is a .bbappend not only extending FILESPATH, but also adding
>>>   meson.cross.d files there (at which point it also didn't work with
>>>   COREBASE replacements as well already) - show an ERROR in such case,
>>>   I don't think it's common scenario to spend more time to support that.
>>>
>>> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
>>> ---
>>>  meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 8 ++++++--
>>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> index 15e9dbaf6f..882a89da7a 100644
>>> --- a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> +++ b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> @@ -31,7 +31,7 @@ def find_meson_cross_files(d):
>>>      if bb.data.inherits_class('native', d):
>>>          return ""
>>>
>>> -    corebase = d.getVar("COREBASE")
>>> +    thisdir = d.getVar("THISDIR")
>>>      import collections
>>>      sitedata = siteinfo_data(d)
>>>      # filename -> found
>>> @@ -39,7 +39,11 @@ def find_meson_cross_files(d):
>>>      for path in d.getVar("FILESPATH").split(":"):
>>>          for element in sitedata:
>>>              filename = os.path.normpath(os.path.join(path,
>>> "meson.cross.d", element))
>>> -            files[filename.replace(corebase, "${COREBASE}")] =
>>> os.path.exists(filename)
>>> +            sanitized_path = filename.replace(thisdir, "${THISDIR}")
>>> +            if sanitized_path == filename:
>>> +                bb.error("Cannot add '%s' to --cross-file, because it's
>>> not relative to THISDIR '%s' and sstate signature would contain this full
>>> path" % (filename, thisdir))
>>> +                continue
>>> +            files[filename.replace(thisdir, "${THISDIR}")] =
>>> os.path.exists(filename)
>>>
>>>      items = ["--cross-file=" + k for k,v in files.items() if v]
>>>      d.appendVar("EXTRA_OEMESON", " " + " ".join(items))
>>> --
>>> 2.27.0
>>>
>>>
>>>
>>>
>>>
>> 
>>
>>

[-- Attachment #2: Type: text/html, Size: 6241 bytes --]

Re: [PATCH] glib-2.0: replace THISDIR instead of COREBASE in find_meson_cross_files --cross-file paths

[Geoff Parker]

[-- Attachment #1: Type: text/plain, Size: 137 bytes --]

In my directory layout,  COREBASE is the poky directory. The layers are under the poky dir also.  The TOPDIR is poky/build-${MACHINE}

[-- Attachment #2: Type: text/html, Size: 151 bytes --]

Re: [OE-core] [PATCH] glib-2.0: replace THISDIR instead of COREBASE in find_meson_cross_files --cross-file paths

[Anibal Limon]

[-- Attachment #1: Type: text/plain, Size: 4780 bytes --]

On Tue, 2 Mar 2021 at 11:32, Martin Jansa <martin.jansa@gmail.com> wrote:

> what's the value of COREBASE in that build? Wasn't full path already in
> your sstate signatures - in that case this change didn't break your builds,
> just highlighted an issue in your builds which was already there.
>
> Our COREBASE is,

  COREBASE="/home/builds/oe-rpb-master/layers/openembedded-core"

Expands as follows,

...
# $COREBASE [3 operations]
#   set
/home/builds/oe-rpb-master/build-845c/conf/../../layers/openembedded-core/meta/conf/layer.conf:20
#     "${@os.path.normpath("${LAYERDIR}/../")}"
#   set data_smart.py:931 [expandVarref]
#
"${@os.path.normpath("/home/builds/oe-rpb-master/build-845c/conf/../../layers/openembedded-core/meta/../")}"
#   set
/home/builds/oe-rpb-master/build-845c/conf/../../layers/openembedded-core/meta/conf/documentation.conf:124
#     [doc] "Specifies the parent directory of the OpenEmbedded Core
Metadata layer (i.e. meta)."
# pre-expansion value:
#
"${@os.path.normpath("/home/builds/oe-rpb-master/build-845c/conf/../../layers/openembedded-core/meta/../")}"
COREBASE="/home/builds/oe-rpb-master/layers/openembedded-core"
...


> We can fix that by calling os.path.normpath on THISDIR, but I would like
> to know the reason why THISDIR is set this way for you (also to reproduce
> it locally to verify the fix).
>

I tried it and fixes the issue,

-    thisdir = d.getVar("THISDIR")
+    thisdir = os.path.normpath(d.getVar("THISDIR"))

We use the next config files for bblayers and local.conf,

https://github.com/96boards/oe-rpb-manifest/tree/qcom/master/conf

Regards,
Anibal


>
>
>
> On Tue, Mar 2, 2021 at 6:14 PM Anibal Limon <anibal.limon@linaro.org>
> wrote:
>
>> This breaks our builds,
>>
>> ...
>> ERROR: Cannot add
>> '/home/builds/oe-rpb-master/layers/openembedded-core/meta/recipes-core/glib-2.0/files/meson.cross.d/common'
>> to --cross-file, because it's not relative to THISDIR
>> '/home/builds/oe-rpb-master/build-410c-2/conf/../../layers/openembedded-core/meta/recipes-core/glib-2.0'
>> and sstate signature would contain this full path
>> ...
>>
>> In,
>>
>>
>> https://ci.linaro.org/job/lt-qcom-openembedded-rpb-master/DISTRO=rpb,MACHINE=dragonboard-410c,label=docker-buster-amd64/470/console
>>
>> Regards,
>> Anibal
>>
>> On Thu, 25 Feb 2021 at 11:45, Martin Jansa <Martin.Jansa@gmail.com>
>> wrote:
>>
>>> * COREBASE works correctly only as long as this recipe is inside of
>>>   COREBASE directory and leaves the full absolute paths if you copy
>>>   this recipe into your own layer (e.g. to backport newer version)
>>>
>>> * use THISDIR as all FILESPATH should be relative to that, unless
>>>   there is a .bbappend not only extending FILESPATH, but also adding
>>>   meson.cross.d files there (at which point it also didn't work with
>>>   COREBASE replacements as well already) - show an ERROR in such case,
>>>   I don't think it's common scenario to spend more time to support that.
>>>
>>> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
>>> ---
>>>  meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 8 ++++++--
>>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> index 15e9dbaf6f..882a89da7a 100644
>>> --- a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> +++ b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>> @@ -31,7 +31,7 @@ def find_meson_cross_files(d):
>>>      if bb.data.inherits_class('native', d):
>>>          return ""
>>>
>>> -    corebase = d.getVar("COREBASE")
>>> +    thisdir = d.getVar("THISDIR")
>>>      import collections
>>>      sitedata = siteinfo_data(d)
>>>      # filename -> found
>>> @@ -39,7 +39,11 @@ def find_meson_cross_files(d):
>>>      for path in d.getVar("FILESPATH").split(":"):
>>>          for element in sitedata:
>>>              filename = os.path.normpath(os.path.join(path,
>>> "meson.cross.d", element))
>>> -            files[filename.replace(corebase, "${COREBASE}")] =
>>> os.path.exists(filename)
>>> +            sanitized_path = filename.replace(thisdir, "${THISDIR}")
>>> +            if sanitized_path == filename:
>>> +                bb.error("Cannot add '%s' to --cross-file, because it's
>>> not relative to THISDIR '%s' and sstate signature would contain this full
>>> path" % (filename, thisdir))
>>> +                continue
>>> +            files[filename.replace(thisdir, "${THISDIR}")] =
>>> os.path.exists(filename)
>>>
>>>      items = ["--cross-file=" + k for k,v in files.items() if v]
>>>      d.appendVar("EXTRA_OEMESON", " " + " ".join(items))
>>> --
>>> 2.27.0
>>>
>>>
>>> 
>>>
>>>

[-- Attachment #2: Type: text/html, Size: 7434 bytes --]

Re: [OE-core] [PATCH] glib-2.0: replace THISDIR instead of COREBASE in find_meson_cross_files --cross-file paths

[Geoff Parker]

[-- Attachment #1: Type: text/plain, Size: 5765 bytes --]

Hi Martin,

The normpath change worked for me also:
-    thisdir = d.getVar("THISDIR")
+    thisdir = os.path.normpath(d.getVar("THISDIR"))

Another work-around was to replace the relative paths in our bblayers.conf
file which looked like:

    ${TOPDIR}/../meta-openembedded/meta-oe \
    ...

With
  YOCTOROOT = "${@os.path.abspath(os.path.join(d.getVar('TOPDIR'),
os.pardir))}"

   ${YOCTOROOT}/meta-poky \
   ...

So it looks like the root cause is related to having our layers and build
dir under the poky directory and relative paths in the bblayers.conf file
(this should be supported, no?).
I plan to change that at some point, but this layout evolved from previous
generations of this project.


On Tue, Mar 2, 2021 at 11:01 AM Anibal Limon <anibal.limon@linaro.org>
wrote:

>
>
> On Tue, 2 Mar 2021 at 11:32, Martin Jansa <martin.jansa@gmail.com> wrote:
>
>> what's the value of COREBASE in that build? Wasn't full path already in
>> your sstate signatures - in that case this change didn't break your builds,
>> just highlighted an issue in your builds which was already there.
>>
>> Our COREBASE is,
>
>   COREBASE="/home/builds/oe-rpb-master/layers/openembedded-core"
>
> Expands as follows,
>
> ...
> # $COREBASE [3 operations]
> #   set
> /home/builds/oe-rpb-master/build-845c/conf/../../layers/openembedded-core/meta/conf/layer.conf:20
> #     "${@os.path.normpath("${LAYERDIR}/../")}"
> #   set data_smart.py:931 [expandVarref]
> #
> "${@os.path.normpath("/home/builds/oe-rpb-master/build-845c/conf/../../layers/openembedded-core/meta/../")}"
> #   set
> /home/builds/oe-rpb-master/build-845c/conf/../../layers/openembedded-core/meta/conf/documentation.conf:124
> #     [doc] "Specifies the parent directory of the OpenEmbedded Core
> Metadata layer (i.e. meta)."
> # pre-expansion value:
> #
> "${@os.path.normpath("/home/builds/oe-rpb-master/build-845c/conf/../../layers/openembedded-core/meta/../")}"
> COREBASE="/home/builds/oe-rpb-master/layers/openembedded-core"
> ...
>
>
>> We can fix that by calling os.path.normpath on THISDIR, but I would like
>> to know the reason why THISDIR is set this way for you (also to reproduce
>> it locally to verify the fix).
>>
>
> I tried it and fixes the issue,
>
> -    thisdir = d.getVar("THISDIR")
> +    thisdir = os.path.normpath(d.getVar("THISDIR"))
>
> We use the next config files for bblayers and local.conf,
>
> https://github.com/96boards/oe-rpb-manifest/tree/qcom/master/conf
>
> Regards,
> Anibal
>
>
>>
>>
>>
>> On Tue, Mar 2, 2021 at 6:14 PM Anibal Limon <anibal.limon@linaro.org>
>> wrote:
>>
>>> This breaks our builds,
>>>
>>> ...
>>> ERROR: Cannot add
>>> '/home/builds/oe-rpb-master/layers/openembedded-core/meta/recipes-core/glib-2.0/files/meson.cross.d/common'
>>> to --cross-file, because it's not relative to THISDIR
>>> '/home/builds/oe-rpb-master/build-410c-2/conf/../../layers/openembedded-core/meta/recipes-core/glib-2.0'
>>> and sstate signature would contain this full path
>>> ...
>>>
>>> In,
>>>
>>>
>>> https://ci.linaro.org/job/lt-qcom-openembedded-rpb-master/DISTRO=rpb,MACHINE=dragonboard-410c,label=docker-buster-amd64/470/console
>>>
>>> Regards,
>>> Anibal
>>>
>>> On Thu, 25 Feb 2021 at 11:45, Martin Jansa <Martin.Jansa@gmail.com>
>>> wrote:
>>>
>>>> * COREBASE works correctly only as long as this recipe is inside of
>>>>   COREBASE directory and leaves the full absolute paths if you copy
>>>>   this recipe into your own layer (e.g. to backport newer version)
>>>>
>>>> * use THISDIR as all FILESPATH should be relative to that, unless
>>>>   there is a .bbappend not only extending FILESPATH, but also adding
>>>>   meson.cross.d files there (at which point it also didn't work with
>>>>   COREBASE replacements as well already) - show an ERROR in such case,
>>>>   I don't think it's common scenario to spend more time to support that.
>>>>
>>>> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
>>>> ---
>>>>  meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 8 ++++++--
>>>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>>> b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>>> index 15e9dbaf6f..882a89da7a 100644
>>>> --- a/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>>> +++ b/meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb
>>>> @@ -31,7 +31,7 @@ def find_meson_cross_files(d):
>>>>      if bb.data.inherits_class('native', d):
>>>>          return ""
>>>>
>>>> -    corebase = d.getVar("COREBASE")
>>>> +    thisdir = d.getVar("THISDIR")
>>>>      import collections
>>>>      sitedata = siteinfo_data(d)
>>>>      # filename -> found
>>>> @@ -39,7 +39,11 @@ def find_meson_cross_files(d):
>>>>      for path in d.getVar("FILESPATH").split(":"):
>>>>          for element in sitedata:
>>>>              filename = os.path.normpath(os.path.join(path,
>>>> "meson.cross.d", element))
>>>> -            files[filename.replace(corebase, "${COREBASE}")] =
>>>> os.path.exists(filename)
>>>> +            sanitized_path = filename.replace(thisdir, "${THISDIR}")
>>>> +            if sanitized_path == filename:
>>>> +                bb.error("Cannot add '%s' to --cross-file, because
>>>> it's not relative to THISDIR '%s' and sstate signature would contain this
>>>> full path" % (filename, thisdir))
>>>> +                continue
>>>> +            files[filename.replace(thisdir, "${THISDIR}")] =
>>>> os.path.exists(filename)
>>>>
>>>>      items = ["--cross-file=" + k for k,v in files.items() if v]
>>>>      d.appendVar("EXTRA_OEMESON", " " + " ".join(items))
>>>> --
>>>> 2.27.0
>>>>
>>>>
>>>>
>>>>
>>>>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 8784 bytes --]