[RFC] About rpm multilib packagegroup QA Warning: install files into a shared area when those files already exist

[RFC] About rpm multilib packagegroup QA Warning: install files into a shared area when those files already exist

[Hongxu Jia]

Hi All,

This issue could be reproduced only by building multilib packagegroup
package in the first time. [YOCTO #5532]

*Step:
1) bitbake packagegroup-core-nfs-server

2) bitbake lib32-packagegroup-core-nfs-server
     Only the first time to do the build has this issue:
     ...
    WARNING: The recipe lib32-packagegroup-core-nfs is trying to install 
files into a shared area when those files already exist. Those files  
and their manifest location are:
/home/jiahongxu/yocto/build-20131120-yocto-qemux86-64/tmp/deploy/rpm/all/packagegroup-core-nfs-server-1.0-r2.all.rpm 

       Matched in manifest-allarch-packagegroup-core-nfs.deploy-rpm

*Analysis:

- The following commit message is the background which come
from oe-core d08e64a98316d7659b0fb56812667c534f66a1a8
[YOCTO #4532] Mark Hatle
     In deb/ipk on a multilib package, the package name has specific 
multilib
     references in it.  I.e. the alternative libraries start with something
     like lib32-...  This was done primarily because deb/ipk do not 
allow two
     packages with the same name (but different architectures) to be
     installed at the same time.  So the name has to be unique.

     In RPM however, the names of the packages and matches with the
     architectures and if they are not the same we can do these multilib
     installs.

- For this rpm multilib packagegroup issue, the multilib and non-multilib
packages have the same name 'packagegroup-core-nfs-server', and
the same architectures 'all'.

*Solution

- One possible fix is as Mark Hatle suggested simply to follow the 
deb/ipk package
naming, but this causes a design advantage of rpm.  When a package has a
dependency on 'bash', we really don't care what bash is installed, only 
that -a-
bash is installed.  In the deb/ipk case, the lib32- packages would end 
up with a
lib32-bash dependency and you could potentially end up with two 'bash' 
packages
being installed.

- Tweak oe-core commit 1674541ed83fa4645f2e078f65fe0f878527ee6e
'multilib: fix allarch/kernel/module-base multilib issues', skip the 
packagegroup
allarch recipes in multilib_virtclass_handler and  extend PROVIDES/RPROVIDES
for packagegroup allarch recipes. It will not build the multilib 
packagegroup allarch
recipes if the same packagegroup allarch recipes has already been built.
But I think it's a bad idea, the previous packagegroup allarch check is 
reasonable.

- We could simply add PACKAGE_ARCH = "${MACHINE_ARCH}" in each
packagegroup allarch recipe to avoid the QA Warning, but it actually 
doesn't work
at the image do_rootfs time, the smart could not correctly find the RDEPENDS
between multilib and non-multilib packagegroup packages.

- Is it necessary to fix the multilib packagegroup issue, it seems the 
warning
occurs on the world building, we rarely build both of the multilib and 
non-multilib
for one package at the same time , we could simple tweak the sanity check
to ignore the warning.

Any suggestion is welcomed.

//Hongxu

//Hongxu

Re: [RFC] About rpm multilib packagegroup QA Warning: install files into a shared area when those files already exist

[Mark Hatle]

On 4/21/14, 10:17 PM, Hongxu Jia wrote:
> Hi All,
>
> This issue could be reproduced only by building multilib packagegroup
> package in the first time. [YOCTO #5532]
>
> *Step:
> 1) bitbake packagegroup-core-nfs-server
>
> 2) bitbake lib32-packagegroup-core-nfs-server
>       Only the first time to do the build has this issue:
>       ...
>      WARNING: The recipe lib32-packagegroup-core-nfs is trying to install
> files into a shared area when those files already exist. Those files
> and their manifest location are:
> /home/jiahongxu/yocto/build-20131120-yocto-qemux86-64/tmp/deploy/rpm/all/packagegroup-core-nfs-server-1.0-r2.all.rpm
>
>         Matched in manifest-allarch-packagegroup-core-nfs.deploy-rpm

There are two types of 'packagegroup' packages.  Ones that are specific to an 
architecture (or machine), and one that are 'all' packages.

All packages in indeed generic and should not have references to 'lib32' or 
specific other package types on RPM.

> *Analysis:
>
> - The following commit message is the background which come
> from oe-core d08e64a98316d7659b0fb56812667c534f66a1a8
> [YOCTO #4532] Mark Hatle
>       In deb/ipk on a multilib package, the package name has specific
> multilib
>       references in it.  I.e. the alternative libraries start with something
>       like lib32-...  This was done primarily because deb/ipk do not
> allow two
>       packages with the same name (but different architectures) to be
>       installed at the same time.  So the name has to be unique.
>
>       In RPM however, the names of the packages and matches with the
>       architectures and if they are not the same we can do these multilib
>       installs.
>
> - For this rpm multilib packagegroup issue, the multilib and non-multilib
> packages have the same name 'packagegroup-core-nfs-server', and
> the same architectures 'all'.

This is correct.  For a packagegroup that only wants generically named 
functionality (and all package) then we could end up trying to generate two of 
them...

> *Solution
>
> - One possible fix is as Mark Hatle suggested simply to follow the
> deb/ipk package
> naming, but this causes a design advantage of rpm.  When a package has a
> dependency on 'bash', we really don't care what bash is installed, only
> that -a-
> bash is installed.  In the deb/ipk case, the lib32- packages would end
> up with a
> lib32-bash dependency and you could potentially end up with two 'bash'
> packages
> being installed.

Yes, we want to avoid that unless we find there simply is no other way.  One 
possible fix (using the above approach), if this is an 'all' package, and we're 
not in the primary multilib, we could simply skip the package_write_rpm step. 
This may prevent the warning and will avoid repackaging things unnecessarily.

We only want to do this for 'all' types, and not for a tune or machine type, as 
those are allowed to have specific architectural references in them.

> - Tweak oe-core commit 1674541ed83fa4645f2e078f65fe0f878527ee6e
> 'multilib: fix allarch/kernel/module-base multilib issues', skip the
> packagegroup
> allarch recipes in multilib_virtclass_handler and  extend PROVIDES/RPROVIDES
> for packagegroup allarch recipes. It will not build the multilib
> packagegroup allarch
> recipes if the same packagegroup allarch recipes has already been built.
> But I think it's a bad idea, the previous packagegroup allarch check is
> reasonable.

In this case, it would likely break deb and opkg.  Even though we don't use 
them, I still don't want to break them unintentionally.

> - We could simply add PACKAGE_ARCH = "${MACHINE_ARCH}" in each
> packagegroup allarch recipe to avoid the QA Warning, but it actually
> doesn't work
> at the image do_rootfs time, the smart could not correctly find the RDEPENDS
> between multilib and non-multilib packagegroup packages.
>
> - Is it necessary to fix the multilib packagegroup issue, it seems the
> warning
> occurs on the world building, we rarely build both of the multilib and
> non-multilib
> for one package at the same time , we could simple tweak the sanity check
> to ignore the warning.

It is a warning that really doesn't cause a problem.  Tweaking the sanity check 
is another option -- but the actions of the system are incorrect.  It should 
know for RPM that we're not REALLY rebuilding the packagegroup, and not 
repackage it.

If we were to modify the sanity check, then we should change it to verify what 
has been installed already, and what is about to be installed/overwritten have 
the same contents.  If the contents differ, then it's still a warning. 
Otherwise we could silently ignore the warning for our customers (and try to 
resolve this in the community for WR 7, if it's still an issue there.)

> Any suggestion is welcomed.

For WR 6, either we should:

Adjust the package generation to avoid generating 'all', secondary multilib 
packages using rpm.

or

Adjust the sanity check to verify the second package overwriting the first has 
the same contents.  [Specifically package set.]

--Mark

> //Hongxu
>
> //Hongxu
>

Re: [RFC] About rpm multilib packagegroup QA Warning: install files into a shared area when those files already exist

[Mark Hatle]

On 4/22/14, 9:52 AM, Mark Hatle wrote:
> On 4/21/14, 10:17 PM, Hongxu Jia wrote:
>> Hi All,
>>
>> This issue could be reproduced only by building multilib packagegroup
>> package in the first time. [YOCTO #5532]
>>
>> *Step:
>> 1) bitbake packagegroup-core-nfs-server
>>
>> 2) bitbake lib32-packagegroup-core-nfs-server
>>        Only the first time to do the build has this issue:
>>        ...
>>       WARNING: The recipe lib32-packagegroup-core-nfs is trying to install
>> files into a shared area when those files already exist. Those files
>> and their manifest location are:
>> /home/jiahongxu/yocto/build-20131120-yocto-qemux86-64/tmp/deploy/rpm/all/packagegroup-core-nfs-server-1.0-r2.all.rpm
>>
>>          Matched in manifest-allarch-packagegroup-core-nfs.deploy-rpm
>
> There are two types of 'packagegroup' packages.  Ones that are specific to an
> architecture (or machine), and one that are 'all' packages.
>
> All packages in indeed generic and should not have references to 'lib32' or
> specific other package types on RPM.
>
>> *Analysis:
>>
>> - The following commit message is the background which come
>> from oe-core d08e64a98316d7659b0fb56812667c534f66a1a8
>> [YOCTO #4532] Mark Hatle
>>        In deb/ipk on a multilib package, the package name has specific
>> multilib
>>        references in it.  I.e. the alternative libraries start with something
>>        like lib32-...  This was done primarily because deb/ipk do not
>> allow two
>>        packages with the same name (but different architectures) to be
>>        installed at the same time.  So the name has to be unique.
>>
>>        In RPM however, the names of the packages and matches with the
>>        architectures and if they are not the same we can do these multilib
>>        installs.
>>
>> - For this rpm multilib packagegroup issue, the multilib and non-multilib
>> packages have the same name 'packagegroup-core-nfs-server', and
>> the same architectures 'all'.
>
> This is correct.  For a packagegroup that only wants generically named
> functionality (and all package) then we could end up trying to generate two of
> them...
>
>> *Solution
>>
>> - One possible fix is as Mark Hatle suggested simply to follow the
>> deb/ipk package
>> naming, but this causes a design advantage of rpm.  When a package has a
>> dependency on 'bash', we really don't care what bash is installed, only
>> that -a-
>> bash is installed.  In the deb/ipk case, the lib32- packages would end
>> up with a
>> lib32-bash dependency and you could potentially end up with two 'bash'
>> packages
>> being installed.
>
> Yes, we want to avoid that unless we find there simply is no other way.  One
> possible fix (using the above approach), if this is an 'all' package, and we're
> not in the primary multilib, we could simply skip the package_write_rpm step.
> This may prevent the warning and will avoid repackaging things unnecessarily.
>
> We only want to do this for 'all' types, and not for a tune or machine type, as
> those are allowed to have specific architectural references in them.
>
>> - Tweak oe-core commit 1674541ed83fa4645f2e078f65fe0f878527ee6e
>> 'multilib: fix allarch/kernel/module-base multilib issues', skip the
>> packagegroup
>> allarch recipes in multilib_virtclass_handler and  extend PROVIDES/RPROVIDES
>> for packagegroup allarch recipes. It will not build the multilib
>> packagegroup allarch
>> recipes if the same packagegroup allarch recipes has already been built.
>> But I think it's a bad idea, the previous packagegroup allarch check is
>> reasonable.
>
> In this case, it would likely break deb and opkg.  Even though we don't use
> them, I still don't want to break them unintentionally.
>
>> - We could simply add PACKAGE_ARCH = "${MACHINE_ARCH}" in each
>> packagegroup allarch recipe to avoid the QA Warning, but it actually
>> doesn't work
>> at the image do_rootfs time, the smart could not correctly find the RDEPENDS
>> between multilib and non-multilib packagegroup packages.
>>
>> - Is it necessary to fix the multilib packagegroup issue, it seems the
>> warning
>> occurs on the world building, we rarely build both of the multilib and
>> non-multilib
>> for one package at the same time , we could simple tweak the sanity check
>> to ignore the warning.
>
> It is a warning that really doesn't cause a problem.  Tweaking the sanity check
> is another option -- but the actions of the system are incorrect.  It should
> know for RPM that we're not REALLY rebuilding the packagegroup, and not
> repackage it.
>
> If we were to modify the sanity check, then we should change it to verify what
> has been installed already, and what is about to be installed/overwritten have
> the same contents.  If the contents differ, then it's still a warning.
> Otherwise we could silently ignore the warning for our customers (and try to
> resolve this in the community for WR 7, if it's still an issue there.)
>
>> Any suggestion is welcomed.
>
> For WR 6, either we should:

My apologies, I thought this was an internal discussion  (nothing internal about 
my reply, but it may be a bit confusing.)  Just so people can follow along WR6 
is Yocto Project/OE 1.5, and WR 7 is "future work" (i.e. what the community is 
working on now.)

The tone of the email doesn't change.  We've got a problem with existing 
versions and multilibs, and we need to figure out if this is something we can 
solve in the community and then either backport (or if the community work has 
changed to much) figure out a workaround for our existing products.

--Mark

> Adjust the package generation to avoid generating 'all', secondary multilib
> packages using rpm.
>
> or
>
> Adjust the sanity check to verify the second package overwriting the first has
> the same contents.  [Specifically package set.]
>
> --Mark
>
>> //Hongxu
>>
>> //Hongxu
>>
>