[OE-core][PATCH] populate_sdk_ext: record METADATA_REVISION

[OE-core][PATCH] populate_sdk_ext: record METADATA_REVISION

[Chen Qi]

As we delete the .git/ directory, it's impossible to get METADATA_REVISION
inside eSDK. Because of this, we meet the following warning when installing eSDK.

  WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
  but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64

So we record METADATA_REVISION in eSDK generation time to fix this problem.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index e6bf27cf38..9112ab6c5e 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -364,6 +364,9 @@ python copy_buildsystem () {
             # Hide the config information from bitbake output (since it's fixed within the SDK)
             f.write('BUILDCFG_HEADER = ""\n\n')
 
+            # Write METADATA_REVISION
+            f.write('METADATA_REVISION = "%s"\n\n' % d.getVar('METADATA_REVISION'))
+
             f.write('# Provide a flag to indicate we are in the EXT_SDK Context\n')
             f.write('WITHIN_EXT_SDK = "1"\n\n')
 
-- 
2.17.1

Re: [OE-core][PATCH] populate_sdk_ext: record METADATA_REVISION

[Richard Purdie]

On Fri, 2021-03-05 at 18:10 +0800, Chen Qi wrote:
> As we delete the .git/ directory, it's impossible to get METADATA_REVISION
> inside eSDK. Because of this, we meet the following warning when installing eSDK.
> 
>   WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
>   but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64
> 
> So we record METADATA_REVISION in eSDK generation time to fix this problem.
> 
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/classes/populate_sdk_ext.bbclass | 3 +++
>  1 file changed, 3 insertions(+)

I have to wonder why base-files is depending on the metadata-revision? That
implies that base-files and hence anything that depends upon it would rebuild
each time you update to a new revision. Is that really what you want?

Cheers,

Richard

Re: [OE-core][PATCH] populate_sdk_ext: record METADATA_REVISION

[Chen Qi]

[-- Attachment #1: Type: text/plain, Size: 1961 bytes --]

On 03/06/2021 07:29 AM, Richard Purdie wrote:
> On Fri, 2021-03-05 at 18:10 +0800, Chen Qi wrote:
>> As we delete the .git/ directory, it's impossible to get METADATA_REVISION
>> inside eSDK. Because of this, we meet the following warning when installing eSDK.
>>
>>    WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
>>    but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64
>>
>> So we record METADATA_REVISION in eSDK generation time to fix this problem.
>>
>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>> ---
>>   meta/classes/populate_sdk_ext.bbclass | 3 +++
>>   1 file changed, 3 insertions(+)
> I have to wonder why base-files is depending on the metadata-revision? That
> implies that base-files and hence anything that depends upon it would rebuild
> each time you update to a new revision. Is that really what you want?
>
> Cheers,
>
> Richard
>
>
>
>
Hi Richard,

I tested it on pure yocto project. No extra layer or configuration.
I'm also wondering why base-files do_install depends on 
METADATA_REVISION. The strange thing is that things do not rebuild at 
eSDK installation time. There's only a warning about base-files 
do_install signature.

In fact, this is a problem I found when I was dealing with another 
problem with eSDK.
*Yocto's current eSDK cannot be successfully installed on a different 
distro.* For example, you build esdk on ubunt18 and install the esdk on 
ubuntu20, then you'll get an error of pseudo do_fetch task re-run 
unexpectedly.
I think it might be related to NATIVELSBSTRING, but I haven't found out why.
Manually setting NATIVELSBSTRING to some fixed string such as 'Linux' 
will solve the above problem.

I'm still checking the codes. If you have some idea/suggestion, please 
let me know.

Best Regards,
Chen Qi

[-- Attachment #2: Type: text/html, Size: 2644 bytes --]

Re: [OE-core][PATCH] populate_sdk_ext: record METADATA_REVISION

[Chen Qi]

[-- Attachment #1: Type: text/plain, Size: 2655 bytes --]

On 03/08/2021 10:30 AM, Chen Qi wrote:
> On 03/06/2021 07:29 AM, Richard Purdie wrote:
>> On Fri, 2021-03-05 at 18:10 +0800, Chen Qi wrote:
>>> As we delete the .git/ directory, it's impossible to get METADATA_REVISION
>>> inside eSDK. Because of this, we meet the following warning when installing eSDK.
>>>
>>>    WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
>>>    but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64
>>>
>>> So we record METADATA_REVISION in eSDK generation time to fix this problem.
>>>
>>> Signed-off-by: Chen Qi<Qi.Chen@windriver.com>
>>> ---
>>>   meta/classes/populate_sdk_ext.bbclass | 3 +++
>>>   1 file changed, 3 insertions(+)
>> I have to wonder why base-files is depending on the metadata-revision? That
>> implies that base-files and hence anything that depends upon it would rebuild
>> each time you update to a new revision. Is that really what you want?
>>
>> Cheers,
>>
>> Richard
>>
>>
>>
>>
> Hi Richard,
>
> I tested it on pure yocto project. No extra layer or configuration.
> I'm also wondering why base-files do_install depends on 
> METADATA_REVISION. The strange thing is that things do not rebuild at 
> eSDK installation time. There's only a warning about base-files 
> do_install signature.
>

Just found answer for the above problem.
A previous commit changes poky.conf to make DISTRO_VERSION contain 
METADATA_REVISION.
"""

poky.conf: do not write current date into distro version, use git hash instead


"""

I also confirmed that merely updating the git revision will result in a 
rebuild of base-files.
Rebuild of base-files does not result in rebuild of other recipes 
because we have 'base-files' in SIGGEN_EXCLUDERECIPES_ABISAFE.

For the problem below, I haven't found the root cause, still working on it.

Best Regards,
Chen Qi

> In fact, this is a problem I found when I was dealing with another 
> problem with eSDK.
> *Yocto's current eSDK cannot be successfully installed on a different 
> distro.* For example, you build esdk on ubunt18 and install the esdk 
> on ubuntu20, then you'll get an error of pseudo do_fetch task re-run 
> unexpectedly.
> I think it might be related to NATIVELSBSTRING, but I haven't found 
> out why.
> Manually setting NATIVELSBSTRING to some fixed string such as 'Linux' 
> will solve the above problem.
>
> I'm still checking the codes. If you have some idea/suggestion, please 
> let me know.
>
> Best Regards,
> Chen Qi
>
>
> 
>


[-- Attachment #2: Type: text/html, Size: 3929 bytes --]

Re: [OE-core][PATCH] populate_sdk_ext: record METADATA_REVISION

[Chen Qi]

[-- Attachment #1: Type: text/plain, Size: 3774 bytes --]

On 03/08/2021 02:09 PM, Chen Qi wrote:
> On 03/08/2021 10:30 AM, Chen Qi wrote:
>> On 03/06/2021 07:29 AM, Richard Purdie wrote:
>>> On Fri, 2021-03-05 at 18:10 +0800, Chen Qi wrote:
>>>> As we delete the .git/ directory, it's impossible to get METADATA_REVISION
>>>> inside eSDK. Because of this, we meet the following warning when installing eSDK.
>>>>
>>>>    WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
>>>>    but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64
>>>>
>>>> So we record METADATA_REVISION in eSDK generation time to fix this problem.
>>>>
>>>> Signed-off-by: Chen Qi<Qi.Chen@windriver.com>
>>>> ---
>>>>   meta/classes/populate_sdk_ext.bbclass | 3 +++
>>>>   1 file changed, 3 insertions(+)
>>> I have to wonder why base-files is depending on the metadata-revision? That
>>> implies that base-files and hence anything that depends upon it would rebuild
>>> each time you update to a new revision. Is that really what you want?
>>>
>>> Cheers,
>>>
>>> Richard
>>>
>>>
>>>
>>>
>> Hi Richard,
>>
>> I tested it on pure yocto project. No extra layer or configuration.
>> I'm also wondering why base-files do_install depends on 
>> METADATA_REVISION. The strange thing is that things do not rebuild at 
>> eSDK installation time. There's only a warning about base-files 
>> do_install signature.
>>
>
> Just found answer for the above problem.
> A previous commit changes poky.conf to make DISTRO_VERSION contain 
> METADATA_REVISION.
> """
> poky.conf: do not write current date into distro version, use git hash instead
>
> """
>
> I also confirmed that merely updating the git revision will result in 
> a rebuild of base-files.
> Rebuild of base-files does not result in rebuild of other recipes 
> because we have 'base-files' in SIGGEN_EXCLUDERECIPES_ABISAFE.
>
> For the problem below, I haven't found the root cause, still working 
> on it.
>
> Best Regards,
> Chen Qi
>
>> In fact, this is a problem I found when I was dealing with another 
>> problem with eSDK.
>> *Yocto's current eSDK cannot be successfully installed on a different 
>> distro.* For example, you build esdk on ubunt18 and install the esdk 
>> on ubuntu20, then you'll get an error of pseudo do_fetch task re-run 
>> unexpectedly.
>> I think it might be related to NATIVELSBSTRING, but I haven't found 
>> out why.
>> Manually setting NATIVELSBSTRING to some fixed string such as 'Linux' 
>> will solve the above problem.
>>
>> I'm still checking the codes. If you have some idea/suggestion, 
>> please let me know.
>>

I think I've found the problem.

"""
commit 3ecf5d9692fec97b37af6a4e6c747a4e2c2ca292
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Sat Nov 21 16:16:40 2020 +0000

     uninative: Don't use single sstate for pseudo-native

     pseudo-native is a bit special. It conditionally compiles in 
support for
     xattr, statx and statvfs amongst other options. If a pseudo-native 
binary is
     used on a system where these functions are present but it wasn't 
compiled in
     we see hard to debug permissions problems.
[snip]
"""

This commit effectively says that the pseudo-native's sstate cache 
should not be shared among hosts with different distros, as it's 
appending ${ORIGLSBSTRING} to SSTATE_PKGARCH for pseudo-native.

I'm not familiar with the pseudo's codes. But I'm wondering if it's 
possible for pseudo to always build in attr support and behave 
differently on different hosts/filesystems.

Best Regards,
Chen Qi

>> Best Regards,
>> Chen Qi
>>
>>
>>
>
>
>
> 
>


[-- Attachment #2: Type: text/html, Size: 6034 bytes --]

Re: [OE-core][PATCH] populate_sdk_ext: record METADATA_REVISION

[Chen Qi]

[-- Attachment #1: Type: text/plain, Size: 4371 bytes --]

On 03/10/2021 04:31 PM, Chen Qi wrote:
> On 03/08/2021 02:09 PM, Chen Qi wrote:
>> On 03/08/2021 10:30 AM, Chen Qi wrote:
>>> On 03/06/2021 07:29 AM, Richard Purdie wrote:
>>>> On Fri, 2021-03-05 at 18:10 +0800, Chen Qi wrote:
>>>>> As we delete the .git/ directory, it's impossible to get METADATA_REVISION
>>>>> inside eSDK. Because of this, we meet the following warning when installing eSDK.
>>>>>
>>>>>    WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
>>>>>    but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64
>>>>>
>>>>> So we record METADATA_REVISION in eSDK generation time to fix this problem.
>>>>>
>>>>> Signed-off-by: Chen Qi<Qi.Chen@windriver.com>
>>>>> ---
>>>>>   meta/classes/populate_sdk_ext.bbclass | 3 +++
>>>>>   1 file changed, 3 insertions(+)
>>>> I have to wonder why base-files is depending on the metadata-revision? That
>>>> implies that base-files and hence anything that depends upon it would rebuild
>>>> each time you update to a new revision. Is that really what you want?
>>>>
>>>> Cheers,
>>>>
>>>> Richard
>>>>
>>>>
>>>>
>>>>
>>> Hi Richard,
>>>
>>> I tested it on pure yocto project. No extra layer or configuration.
>>> I'm also wondering why base-files do_install depends on 
>>> METADATA_REVISION. The strange thing is that things do not rebuild 
>>> at eSDK installation time. There's only a warning about base-files 
>>> do_install signature.
>>>
>>
>> Just found answer for the above problem.
>> A previous commit changes poky.conf to make DISTRO_VERSION contain 
>> METADATA_REVISION.
>> """
>> poky.conf: do not write current date into distro version, use git hash instead
>>
>> """
>>
>> I also confirmed that merely updating the git revision will result in 
>> a rebuild of base-files.
>> Rebuild of base-files does not result in rebuild of other recipes 
>> because we have 'base-files' in SIGGEN_EXCLUDERECIPES_ABISAFE.
>>
>> For the problem below, I haven't found the root cause, still working 
>> on it.
>>
>> Best Regards,
>> Chen Qi
>>
>>> In fact, this is a problem I found when I was dealing with another 
>>> problem with eSDK.
>>> *Yocto's current eSDK cannot be successfully installed on a 
>>> different distro.* For example, you build esdk on ubunt18 and 
>>> install the esdk on ubuntu20, then you'll get an error of pseudo 
>>> do_fetch task re-run unexpectedly.
>>> I think it might be related to NATIVELSBSTRING, but I haven't found 
>>> out why.
>>> Manually setting NATIVELSBSTRING to some fixed string such as 
>>> 'Linux' will solve the above problem.
>>>
>>> I'm still checking the codes. If you have some idea/suggestion, 
>>> please let me know.
>>>
>
> I think I've found the problem.
>
> """
> commit 3ecf5d9692fec97b37af6a4e6c747a4e2c2ca292
> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date:   Sat Nov 21 16:16:40 2020 +0000
>
>     uninative: Don't use single sstate for pseudo-native
>
>     pseudo-native is a bit special. It conditionally compiles in 
> support for
>     xattr, statx and statvfs amongst other options. If a pseudo-native 
> binary is
>     used on a system where these functions are present but it wasn't 
> compiled in
>     we see hard to debug permissions problems.
> [snip]
> """
>
> This commit effectively says that the pseudo-native's sstate cache 
> should not be shared among hosts with different distros, as it's 
> appending ${ORIGLSBSTRING} to SSTATE_PKGARCH for pseudo-native.
>
> I'm not familiar with the pseudo's codes. But I'm wondering if it's 
> possible for pseudo to always build in attr support and behave 
> differently on different hosts/filesystems.
>


Just checked the pseudo recipe. It seems that attr is always enabled.
DEPENDS = "sqlite3 attr"
PSEUDO_EXTRA_OPTS ?= "--enable-force-async --without-passwd-fallback 
--enable-epoll --enable-xattr"

I think there should be no circumstances where the xattr support is not 
compiled in, yet the pseudo-native sstate cache is still reused. I'll 
send out a patch to revert the commit.

Best Regards,
Chen Qi


> Best Regards,
> Chen Qi
>
>>> Best Regards,
>>> Chen Qi
>>>
>>>
>>
>>
>>
>>
>
>
>
> 
>


[-- Attachment #2: Type: text/html, Size: 7277 bytes --]

Re: [OE-core][PATCH] populate_sdk_ext: record METADATA_REVISION

[Richard Purdie]

On Wed, 2021-03-10 at 17:11 +0800, ChenQi wrote:
>  I think I've found the problem.
>  
>  """
>  commit 3ecf5d9692fec97b37af6a4e6c747a4e2c2ca292
>  Author: Richard Purdie <richard.purdie@linuxfoundation.org>
>  Date:   Sat Nov 21 16:16:40 2020 +0000
>  
>      uninative: Don't use single sstate for pseudo-native
>      
>      pseudo-native is a bit special. It conditionally compiles in support for
>      xattr, statx and statvfs amongst other options. If a pseudo-native binary is
>      used on a system where these functions are present but it wasn't compiled in
>      we see hard to debug permissions problems.
>  [snip]    
>  """
>  
>  This commit effectively says that the pseudo-native's sstate cache should not be shared among hosts with
> different distros, as it's appending ${ORIGLSBSTRING} to SSTATE_PKGARCH for pseudo-native.
>  
>  I'm not familiar with the pseudo's codes. But I'm wondering if it's possible for pseudo to always build in
> attr support and behave differently on different hosts/filesystems.
>  
>   
>  
>  Just checked the pseudo recipe. It seems that attr is always enabled.
>  DEPENDS = "sqlite3 attr"
>  PSEUDO_EXTRA_OPTS ?= "--enable-force-async --without-passwd-fallback --enable-epoll --enable-xattr"
>  
>  I think there should be no circumstances where the xattr support is not compiled in, yet the pseudo-native
> sstate cache is still reused. I'll send out a patch to revert the commit.

I was very reluctant to add that commit in the first place and from what 
I remember, it is a lot more tricky that just adding the right 
dependencies/configuration. I'd need to look at the code to remember
but there was code only being enabled on some distros and we'd have to
have it enabled in all cases even when headers and library functions
were not present.

Cheers,

Richard