Missing dependencies after setscene, postinst fails

Missing dependencies after setscene, postinst fails

[Jack_Fewx]

Hello,

I am seeing build problems with a number of poky packages.  I know why the build is failing, but these should work "out of the box".  I have figured out a workaround, but it seems very hack-ish to me, and I'd really like to know if there is a more elegant solution or if this is a bug?  Has anyone else seen this?

- Setup: 
Poky version is 1.7.1.  Working directories have been removed, forcing all packages to be restored through the setscene process. The rootfs is read-only, so postinst causes build failure instead of being postponed to first boot.

- Problem:
In this scenario I am consistently seeing 3 Poky recipes fail at postinst.  I have triaged the problem to be missing native-class dependencies for each of the 3 packages.  

The problem packages are: ca-certificates, nss, and gdk-pixbuf-native.  ca-certificates and nss both depend on their own native-class builds, so ca-certificates-native and nss-native respectively, and they fail after doing do_package_write_ipk_setscene.  gdk-pixbuf-native needs jpeg-native in order to complete do_populate_sysroot_setscene, but fails due to its absence.

Each of these recipes has a DEPENDS or PACKAGECONFIG appropriate to pull in the needed package during the normal build process, but these dependencies don't seem to carry over to the setscene process.

- My attempt at a solution:
My workaround is to write a bbappend for each of the 3 recipes that looks something like below, that explicitly add the missing dependencies:

--- /dev/null
+++ b/nss/nss_%.bbappend
+do_package_write_ipk_setscene[depends] += "nss-native:do_populate_sysroot"

Re: Missing dependencies after setscene, postinst fails

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

On 28 July 2015 at 15:12, <Jack_Fewx@dell.com> wrote:

> In this scenario I am consistently seeing 3 Poky recipes fail at
> postinst.  I have triaged the problem to be missing native-class
> dependencies for each of the 3 packages.
>
> The problem packages are: ca-certificates, nss, and gdk-pixbuf-native.
> ca-certificates and nss both depend on their own native-class builds, so
> ca-certificates-native and nss-native respectively, and they fail after
> doing do_package_write_ipk_setscene.  gdk-pixbuf-native needs jpeg-native
> in order to complete do_populate_sysroot_setscene, but fails due to its
> absence.
>

The problem is an ordering one - as I understand it setscene depends are
restored in the reverse order to what you'd expect so any hooks that run in
populate_sysroot need to explicitly add dependencies.  The pixbuf one is
known and we should probably merge the ugly fix before doing the right
thing, but the others are unknown to me at least.

A good start would be to put those missing depends statements in the
recipe, can you prepare a patch for oe-core?

Ross

[-- Attachment #2: Type: text/html, Size: 1560 bytes --]

Re: Missing dependencies after setscene, postinst fails

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

On 28 July 2015 at 15:12, <Jack_Fewx@dell.com> wrote:

> The problem packages are: ca-certificates, nss, and gdk-pixbuf-native.
> ca-certificates and nss both depend on their own native-class builds, so
> ca-certificates-native and nss-native respectively, and they fail after
> doing do_package_write_ipk_setscene.  gdk-pixbuf-native needs jpeg-native
> in order to complete do_populate_sysroot_setscene, but fails due to its
> absence.


So I can't replicate any problem with ca-certificates, can you provide a
log of what fails for you?

Ross

[-- Attachment #2: Type: text/html, Size: 897 bytes --]

Re: Missing dependencies after setscene, postinst fails

[Jack_Fewx]

[-- Attachment #1: Type: text/plain, Size: 1703 bytes --]

Do you have the read-only Rootfs flag set?  And did you remove all of the /tmp directory (including /work and /sysroots)? Those are the keys to seeing the failure:

log.do_rootfs (relevant lines):

Installing ca-certificates (20140325-r0) on root.
Package ca-certificates (20140325-r0) installed in root is up to date.
……./rootfs//var/lib/opkg/info/ca-certificates.postinst: line 1: update-ca-certificates: command not found
Configuring ca-certificates.
ca-certificates.postinst returned 127, marking as unpacked only, configuration required on target.
ERROR: The following packages could not be configured offline and rootfs is read-only: ['ca-certificates']
DEBUG: Python function do_rootfs finished
ERROR: Function failed: do_rootfs

ca-certificates.postinst:
SYSROOT="$D" update-ca-certificates


The command it is choking on is “update-ca-certificates”, which is installed in the rootfs, but what’s missing is the native version under sysroots that can run on the build machine during build time.  “ca-certificates-native” is not re-populated in sysroots during setscene.

Jack Fewx

---------

On 28 July 2015 at 15:12, <Jack_Fewx@dell.com<mailto:Jack_Fewx@dell.com>> wrote:
The problem packages are: ca-certificates, nss, and gdk-pixbuf-native.  ca-certificates and nss both depend on their own native-class builds, so ca-certificates-native and nss-native respectively, and they fail after doing do_package_write_ipk_setscene.  gdk-pixbuf-native needs jpeg-native in order to complete do_populate_sysroot_setscene, but fails due to its absence.

So I can't replicate any problem with ca-certificates, can you provide a log of what fails for you?

Ross

[-- Attachment #2: Type: text/html, Size: 6895 bytes --]

Re: Missing dependencies after setscene, postinst fails

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 946 bytes --]

On 28 July 2015 at 16:46, <Jack_Fewx@dell.com> wrote:

> The command it is choking on is “update-ca-certificates”, which is
> installed in the rootfs, but what’s missing is the native version under
> sysroots that can run on the build machine during build time.
> “ca-certificates-native” is not re-populated in sysroots during setscene.
>
>
Yeah, that's an optimisation (no need for build dependencies if we're not
building anything) going too far in this specific case.  The same problem
exists for update-rcd and update-alternatives, which is "fixed" by having
image.bbclass add explicit dependencies for those.

Your bbappend is close but I think you could generalise it (it hardcodes
ipkg right now) by adding the dependency to do_populate_sysroot and
do_populate_sysroot_setscene instead.  I'm about to go see the kids before
they go to bed, so if you could test that before I can that would be
awesome.

Ross

[-- Attachment #2: Type: text/html, Size: 1507 bytes --]

Re: Missing dependencies after setscene, postinst fails

[Jack_Fewx]

[-- Attachment #1: Type: text/plain, Size: 294 bytes --]

>> Your bbappend is close but I think you could generalise it (it hardcodes ipkg right now)

Very good point.  After a few attempts, the following seems to work, so it won’t be just ipk specific:

do_package_setscene[depends] += "ca-certificates-native:do_populate_sysroot"

Jack Fewx

[-- Attachment #2: Type: text/html, Size: 3379 bytes --]

Re: Missing dependencies after setscene, postinst fails

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 624 bytes --]

On 28 July 2015 at 18:11, <Jack_Fewx@dell.com> wrote:

> Very good point.  After a few attempts, the following seems to work, so it
> won’t be just ipk specific:
>
>
>
> do_package_setscene[depends] += "ca-certificates
> -native:do_populate_sysroot"
>
>
Interestingly I can't replicate this.  Switched to 1.7.1, added
ca-certificates to core-image-minimal, wiped away tmp, bitbake
core-image-minimal and the log shows ca-certificates-native being installed
before the image is constructed.

Can you share a cooker log (tmp/log/cooker) without your fixes?  That might
help verify what's going on.

Ross

[-- Attachment #2: Type: text/html, Size: 1525 bytes --]

Re: Missing dependencies after setscene, postinst fails

[Jack_Fewx]

[-- Attachment #1: Type: text/plain, Size: 7296 bytes --]

I can’t dump all of that without highly redacting it, so instead I just pulled all instances of ca-certificates and nss with both successful and non-successful builds (i.e. with and without my fixes).  If you need other specific data please let me know and I’ll try to provide it.

Ca-certificates (success):
                Line 174: NOTE: Running setscene task 78 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/ca-certificates/ca-certificates_20140325.bb, do_package_write_ipk_setscene)
                Line 179: NOTE: recipe ca-certificates-20140325-r0: task do_package_write_ipk_setscene: Started
                Line 183: NOTE: recipe ca-certificates-20140325-r0: task do_package_write_ipk_setscene: Succeeded
                Line 1477: NOTE: Running setscene task 718 of 1876 (virtual:native:/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/ca-certificates/ca-certificates_20140325.bb, do_populate_sysroot_setscene)
                Line 1496: NOTE: recipe ca-certificates-native-20140325-r0: task do_populate_sysroot_setscene: Started
                Line 1511: NOTE: recipe ca-certificates-native-20140325-r0: task do_populate_sysroot_setscene: Succeeded
                Line 1820: NOTE: Running setscene task 899 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/ca-certificates/ca-certificates_20140325.bb, do_packagedata_setscene)
                Line 1825: NOTE: recipe ca-certificates-20140325-r0: task do_packagedata_setscene: Started
                Line 1828: NOTE: recipe ca-certificates-20140325-r0: task do_packagedata_setscene: Succeeded
                Line 1829: NOTE: Running setscene task 905 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/ca-certificates/ca-certificates_20140325.bb, do_package_setscene)
                Line 1850: NOTE: recipe ca-certificates-20140325-r0: task do_package_setscene: Started
                Line 1864: NOTE: recipe ca-certificates-20140325-r0: task do_package_setscene: Succeeded

Ca-certificates (fail):

                Line 155: NOTE: Running setscene task 81 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/ca-certificates/ca-certificates_20140325.bb, do_package_write_ipk_setscene)
                Line 162: NOTE: recipe ca-certificates-20140325-r0: task do_package_write_ipk_setscene: Started
                Line 172: NOTE: recipe ca-certificates-20140325-r0: task do_package_write_ipk_setscene: Succeeded
                Line 1783: NOTE: Running setscene task 892 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/ca-certificates/ca-certificates_20140325.bb, do_packagedata_setscene)
                Line 1791: NOTE: recipe ca-certificates-20140325-r0: task do_packagedata_setscene: Started
                Line 1798: NOTE: recipe ca-certificates-20140325-r0: task do_packagedata_setscene: Succeeded
                Line 3772: ERROR: The following packages could not be configured offline and rootfs is read-only: ['ca-certificates']

Nss (success):

                Line 1621: NOTE: Running setscene task 812 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/nss/nss_3.15.1.bb, do_populate_sysroot_setscene)
                Line 1624: NOTE: Running setscene task 813 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/nss/nss_3.15.1.bb, do_package_write_ipk_setscene)
                Line 1627: NOTE: recipe nss-3.15.1-r0: task do_populate_sysroot_setscene: Started
                Line 1632: NOTE: recipe nss-3.15.1-r0: task do_package_write_ipk_setscene: Started
                Line 1635: NOTE: recipe nss-3.15.1-r0: task do_populate_sysroot_setscene: Succeeded
                Line 1643: NOTE: recipe nss-3.15.1-r0: task do_package_write_ipk_setscene: Succeeded
                Line 1645: NOTE: Running setscene task 824 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/nss/nss_3.15.1.bb, do_packagedata_setscene)
                Line 1655: NOTE: recipe nss-3.15.1-r0: task do_packagedata_setscene: Started
                Line 1660: NOTE: recipe nss-3.15.1-r0: task do_packagedata_setscene: Succeeded
                Line 2024: NOTE: Running setscene task 1069 of 1876 (virtual:native:/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/nss/nss_3.15.1.bb, do_populate_sysroot_setscene)
                Line 2036: NOTE: recipe nss-native-3.15.1-r0: task do_populate_sysroot_setscene: Started
                Line 2056: NOTE: recipe nss-native-3.15.1-r0: task do_populate_sysroot_setscene: Succeeded
                Line 2069: NOTE: Running setscene task 1096 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/nss/nss_3.15.1.bb, do_package_setscene)
                Line 2081: NOTE: recipe nss-3.15.1-r0: task do_package_setscene: Started
                Line 2102: NOTE: recipe nss-3.15.1-r0: task do_package_setscene: Succeeded

Nss (fail):
                Line 1659: NOTE: Running setscene task 813 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/nss/nss_3.15.1.bb, do_populate_sysroot_setscene)
                Line 1662: NOTE: Running setscene task 814 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/nss/nss_3.15.1.bb, do_package_write_ipk_setscene)
                Line 1663: NOTE: recipe nss-3.15.1-r0: task do_populate_sysroot_setscene: Started
                Line 1670: NOTE: recipe nss-3.15.1-r0: task do_package_write_ipk_setscene: Started
                Line 1671: NOTE: recipe nss-3.15.1-r0: task do_populate_sysroot_setscene: Succeeded
                Line 1675: NOTE: recipe nss-3.15.1-r0: task do_package_write_ipk_setscene: Succeeded
                Line 1676: NOTE: Running setscene task 819 of 1876 (/home/jack_fewx/master/build/refboard/xrev/../../../poky/meta/recipes-support/nss/nss_3.15.1.bb, do_packagedata_setscene)
                Line 1684: NOTE: recipe nss-3.15.1-r0: task do_packagedata_setscene: Started
                Line 1694: NOTE: recipe nss-3.15.1-r0: task do_packagedata_setscene: Succeeded
                Line 3433: ERROR: The following packages could not be configured offline and rootfs is read-only: ['nss']

Jack Fewx

From: Burton, Ross [mailto:ross.burton@intel.com]
Sent: Wednesday, July 29, 2015 2:22 PM
To: Fewx, Jack
Cc: Poky Project
Subject: Re: [poky] Missing dependencies after setscene, postinst fails


On 28 July 2015 at 18:11, <Jack_Fewx@dell.com<mailto:Jack_Fewx@dell.com>> wrote:
Very good point.  After a few attempts, the following seems to work, so it won’t be just ipk specific:

do_package_setscene[depends] += "ca-certificates-native:do_populate_sysroot"

Interestingly I can't replicate this.  Switched to 1.7.1, added ca-certificates to core-image-minimal, wiped away tmp, bitbake core-image-minimal and the log shows ca-certificates-native being installed before the image is constructed.

Can you share a cooker log (tmp/log/cooker) without your fixes?  That might help verify what's going on.

Ross

[-- Attachment #2: Type: text/html, Size: 18142 bytes --]

Missing dependencies after setscene, postinst fails

[Jack_Fewx]

Hello,

I need some help figuring out a build problem I'm seeing.  I know why the build is failing, I just don't know the best way to fix it.  I have figured out a workaround, but it seems very hack-ish to me, and I'd really like to know if there is a more elegant solution?  Is this a defect or something I'm doing wrong?  Has anyone else seen this?

- Setup: 
Poky version is 1.7.1.  Working directories have been removed, forcing all packages to be restored through the setscene process. The rootfs is read-only, so postinst causes build failure instead of being postponed to first boot.

- Problem:
In this scenario I am consistently seeing 3 Poky recipes fail at postinst.  I have triaged the problem to be missing native-class dependencies for each of the 3 packages.  

The problem packages are: ca-certificates, nss, and gdk-pixbuf-native.  ca-certificates and nss both depend on their own native-class builds, so ca-certificates-native and nss-native respectively, and they fail after doing do_package_write_ipk_setscene.  gdk-pixbuf-native needs jpeg-native in order to complete do_populate_sysroot_setscene, but fails due to its absence.

Each of these recipes has a DEPENDS or PACKAGECONFIG appropriate to pull in the needed package during the normal build process, but these dependencies don't seem to carry over to the setscene process.

- My attempt at a solution:
My workaround is to write a bbappend for each of the 3 recipes that looks something like below, that explicitly add the missing dependencies:

--- /dev/null
+++ b/nss/nss_%.bbappend
+do_package_write_ipk_setscene[depends] += "nss-native:do_populate_sysroot"