Library Patches Causing Many Packages to Rebuild

Library Patches Causing Many Packages to Rebuild

[Connor Smith]

Hello,

I've been doing research to improve the process of applying security
patches to our images. Specifically, I've been looking into why BitBake
seems to be rebuilding more than is necessary when a library is patched.
For example, applying a patch to openssl causes just under 2000 ipks
to be rebuilt. Many of these make sense, because they depend directly or
indirectly on openssl. These packages are being recompiled and rebuilt,
and that makes sense. However, there are other packages being rebuilt
that don't seem to depend on openssl at all (based off the dependency
information generated by "bitbake -g"). These packages are not getting
recompiled, but the do_package and do_package_write_ipk tasks are being
run, and the ipks are being replaced (the only difference I've seen
between the original ipk and the updated one is a change in revision
number). I've tried using "bitbake-whatchanged -v" to see why these
packages are being rebuilt, but the output is not clear.

To give a concrete example, vlan and zip are two of the packages that
are being rebuilt but not recompiled. Based off the dependency
information I got from using "bitbake -g",  these packages do not depend
on openssl. bitbake-whatchanged appears to state that both of these are
being repackaged because the hashes for three dependent tasks are
changing:

    glibc: do_packagedata
    rpm (native): do_populate_sysroot
    gcc-runtime: do_packagedata

glibc and gcc-runtime are both being repackaged but not recompiled
themselves, while rpm-native is being recompiled. What I don't
understand is why these three packages are changing. As far as I can
tell, none of them depend on openssl.

Searching the whatchanged output for why glibc is changing reveals some
strange information. It appears the basehash has changed, which doesn't
make sense to me because nothing directly related to glibc should have
changed (unless I'm confused on what is included in the basehash for a
task). I also see that BB_TASKDEPDATA has changed from a very large map
of values to None. I'm unfamiliar with this variable and couldn't find
any references to it in the documentation, so I'm not sure what's
happening here.

rpm-native appears to be recompiling because the do_populate_sysroot
hash is changing for the native versions of openssl and python. I'm not
sure why either of these tasks are changing.

I also see no explanation of why gcc-runtime is being repackaged.

To see if this behavior was consistent for other libraries, I tried
applying a patch to cracklib. The list of ipks that were rebuilt was
much more predictable, and the corresponding packages are all being
recompiled before being repackaged. There are no ipks that were just
repackaged, as far as I can tell. The whatchanged output still appears
to be somewhat incomplete, however, since there is not an explanation
for all of the packages that were rebuilt, but the information that is
given makes sense.

If anyone could help clear up why patching openssl (and perhaps other
libraries) would cause this behavior, it would be greatly appreciated.

Thanks,
Connor Smith

Re: Library Patches Causing Many Packages to Rebuild

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

On 10 July 2015 at 00:24, Connor Smith <connor.smith@ni.com> wrote:

> rpm-native appears to be recompiling because the do_populate_sysroot
> hash is changing for the native versions of openssl and python. I'm not
> sure why either of these tasks are changing.
>

This is likely the source of a lot of the rebuilds.  openssl was changed,
openssl-native rebuilds, causing a rebuild of rpm-native, which likely
causes a re-package of everything.

If this doesn't explain the situation then attaching the cooker log
demonstrating the problem might help identify it.

Ross

[-- Attachment #2: Type: text/html, Size: 1057 bytes --]

Re: Library Patches Causing Many Packages to Rebuild

[Connor Smith]

[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]



On 07/10/2015 04:20 AM, Burton, Ross wrote:
>
> On 10 July 2015 at 00:24, Connor Smith <connor.smith@ni.com> wrote:
>
>     rpm-native appears to be recompiling because the do_populate_sysroot
>     hash is changing for the native versions of openssl and python. I'm not
>     sure why either of these tasks are changing.
>
>
> This is likely the source of a lot of the rebuilds.  openssl was changed, openssl-native rebuilds, causing a rebuild of rpm-native, which likely causes a re-package of everything.
>
> If this doesn't explain the situation then attaching the cooker log demonstrating the problem might help identify it.
>
> Ross

Ah, that makes sense. I investigated some more, and the do_package task
has a hard-coded dependency on rpm-native. Removing this dependency and
rebuilding resolved the issue; the only ipks being rebuilt now are from
packages that depend on openssl and need to be recompiled.

Should this dependency be based off PACKAGE_CLASSES, or is it necessary
to always depend on rpm-native? We're using opkg, not rpm.

Thanks,
Connor Smith


[-- Attachment #2: Type: text/html, Size: 1743 bytes --]

Re: Library Patches Causing Many Packages to Rebuild

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

On 10 July 2015 at 18:09, Connor Smith <connor.smith@ni.com> wrote:

> Ah, that makes sense. I investigated some more, and the do_package task
> has a hard-coded dependency on rpm-native. Removing this dependency and
> rebuilding resolved the issue; the only ipks being rebuilt now are from
> packages that depend on openssl and need to be recompiled.
>
> Should this dependency be based off PACKAGE_CLASSES, or is it necessary
> to always depend on rpm-native? We're using opkg, not rpm.
>

do_package() uses rpmdeps-oecore built by rpm-native to inject
dependencies, so you'll always need to build rpm-native.

Ross

[-- Attachment #2: Type: text/html, Size: 1086 bytes --]