Troubleshooting sstate cache

Troubleshooting sstate cache

[Nick Winters]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

Often when I use an sstate cache from another build, I run into problems
where builds fail. Usually the failure is a result of not being able to
find a shared library. If I delete the sstate cache, the build succeeds,
although it is 8 hours later. I switched to using sstate mirrors and had
similar problems.

Questions:

* Are there any tools or bitbake commands to use for troubleshooting
failures like this?

* I am creating one sstate cache per architecture being built. Should this
really be necessary?

* Any other advice to help troubleshoot sstate cache issues?

Thanks,
Nick

[-- Attachment #2: Type: text/html, Size: 740 bytes --]

Re: Troubleshooting sstate cache

[Alexander Kanavin]

2018-08-28 7:53 GMT+02:00 Nick Winters <nick@wintersteam.org>:
> Often when I use an sstate cache from another build, I run into problems
> where builds fail. Usually the failure is a result of not being able to find
> a shared library. If I delete the sstate cache, the build succeeds, although
> it is 8 hours later. I switched to using sstate mirrors and had similar
> problems.

If you provide an example of a specific failure, we might be better
able to help you.

It's fine to use one global sstate cache for absolutely everything.
You can share between builds, target machines and distros.

Alex

Re: Troubleshooting sstate cache

[Nick Winters]

[-- Attachment #1: Type: text/plain, Size: 3183 bytes --]

Thanks for your response Alex.

Below is an excerpt of one failure I've encountered. When attempting to
do_rootfs, the following error occurs:

error while loading shared libraries: libopkg.so.1: cannot open shared
object file: No such file or directory

Below is more detail:

NOTE: recipe core-image-biamp-linux-ppc-1.0-r0: task do_rootfs: Started
ERROR: Unable to update the package index files. Command
'/var/lib/jenkins/workspace/SingleNetworkConnectionPipeline/repos/poky/ppc/build-biamp-linux-ppc/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl
-f
/var/lib/jenkins/workspace/SingleNetworkConnectionPipeline/repos/poky/ppc/build-biamp-linux-ppc/tmp/work/mpc83xx-poky-linux/core-image-biamp-linux-ppc/1.0-r0/opkg.conf
-o
/var/lib/jenkins/workspace/SingleNetworkConnectionPipeline/repos/poky/ppc/build-biamp-linux-ppc/tmp/work/mpc83xx-poky-linux/core-image-biamp-linux-ppc/1.0-r0/rootfs
--force_postinstall --prefer-arch-to-version  --add-exclude linux-dummy
--add-exclude opkg --add-exclude dbus --add-exclude gdbm --force-overwrite
update' returned 127:
/var/lib/jenkins/workspace/SingleNetworkConnectionPipeline/repos/poky/ppc/build-biamp-linux-ppc/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl:
error while loading shared libraries: libopkg.so.1: cannot open shared
object file: No such file or directory
ERROR: Function failed: do_rootfs
ERROR: Logfile of failure stored in:
/var/lib/jenkins/workspace/SingleNetworkConnectionPipeline/repos/poky/ppc/build-biamp-linux-ppc/tmp/work/mpc83xx-poky-linux/core-image-biamp-linux-ppc/1.0-r0/temp/log.do_rootfs.4871
NOTE: recipe core-image-biamp-linux-ppc-1.0-r0: task do_rootfs: Failed
ERROR: Task 7
(/var/lib/jenkins/workspace/SingleNetworkConnectionPipeline/repos/poky/ppc/build-biamp-linux-ppc/../meta-biamp/recipes-ppc/images/
core-image-biamp-linux-ppc.bb, do_rootfs) failed with exit code '1'

When I execute ldd on opkg-cl, I get the following:
linux-vdso.so.1 (0x00007ffda9bd8000)
libopkg.so.1 => not found
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f72dbac3000)
/lib64/ld-linux-x86-64.so.2 (0x00007f72dc067000)

Looking inside the tmp/sysroots directory, I can find libopkg.so.1. It
appears as though opkg-cl was built to look for the shared library in the
host path rather than the tmp/sysroots path.

My guess is that this is something other than an sstate-cache problem but
when I clear the sstate-cache, everything builds fine.

Nick

On Tue, Aug 28, 2018 at 12:10 AM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> 2018-08-28 7:53 GMT+02:00 Nick Winters <nick@wintersteam.org>:
> > Often when I use an sstate cache from another build, I run into problems
> > where builds fail. Usually the failure is a result of not being able to
> find
> > a shared library. If I delete the sstate cache, the build succeeds,
> although
> > it is 8 hours later. I switched to using sstate mirrors and had similar
> > problems.
>
> If you provide an example of a specific failure, we might be better
> able to help you.
>
> It's fine to use one global sstate cache for absolutely everything.
> You can share between builds, target machines and distros.
>
> Alex
>

[-- Attachment #2: Type: text/html, Size: 4065 bytes --]

Re: Troubleshooting sstate cache

[Nick Winters]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

The problem may be a result of executables being built with absolute paths
to their shared libraries. When I do the following:

ldd -v
./tmp/sysroots/x86_64-linux/usr/bin/powerpc-poky-linux/powerpc-poky-linux-ar

I see the Library runpath for a shared library is an absolute path to the
original Jenkins job that was building the sstate cache, not the current
one that I am building. How do I configure Yocto not to use absolute paths
to the shared libraries?

Nick

[-- Attachment #2: Type: text/html, Size: 560 bytes --]