[PATCH 1/2] pixbufcache: Ensure sysroot dependencies are satisfied

[PATCH 1/2] pixbufcache: Ensure sysroot dependencies are satisfied

[Kyle Russell]

Reverts part of b411085fefc8582d24c172db5f6610705eb44c2f

Makes sure gdk-pixbuf-native populates the sysroot during setscene;
otherwise, gdk-pixbuf-query-loaders may not actually be available at the
time the completion function runs, causing a build failure.

Similarly, packages providing loader libraries should append to
PIXBUFCACHE_SYSROOT_DEPS with their runtime dependencies.

Signed-off-by: Kyle Russell <bkylerussell@gmail.com>
---
 meta/classes/pixbufcache.bbclass | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/classes/pixbufcache.bbclass b/meta/classes/pixbufcache.bbclass
index dbe11e1..3c1e398 100644
--- a/meta/classes/pixbufcache.bbclass
+++ b/meta/classes/pixbufcache.bbclass
@@ -65,3 +65,17 @@ pixbufcache_sstate_postinst() {
 	fi
 }
 
+# Add all of the dependencies of gdk-pixbuf as dependencies of
+# do_populate_sysroot_setscene so that pixbufcache_sstate_postinst can work
+# (otherwise gdk-pixbuf-query-loaders may not exist or link). Only add
+# gdk-pixbuf-native if we're not building gdk-pixbuf itself.
+#
+# Packages that use this class should extend this variable with their runtime
+# dependencies.
+PIXBUFCACHE_SYSROOT_DEPS = ""
+PIXBUFCACHE_SYSROOT_DEPS_class-native = "\
+    ${@['gdk-pixbuf-native:do_populate_sysroot_setscene', '']['${BPN}' == 'gdk-pixbuf']} \
+"
+
+do_populate_sysroot_setscene[depends] += "${PIXBUFCACHE_SYSROOT_DEPS}"
+do_populate_sysroot[depends] += "${@d.getVar('PIXBUFCACHE_SYSROOT_DEPS', True).replace('_setscene','')}"
-- 
2.7.4

[PATCH 2/2] librsvg: Add missing pixbufcache dependencies

[Kyle Russell]

Forces do_populate_sysroot_setscene to run for libxdmcp-native and
pixman-native before the pixbufcache sstate postinst hook so that the
svg pixbufloader has all of its dependencies available.

g_module_open() failed for libpixbufloader-svg.so: libpixman-1.so.0:
cannot open shared object file: No such file or directory

g_module_open() failed for libpixbufloader-svg.so: libXdmcp.so.6:
cannot open shared object file: No such file or directory

Signed-off-by: Kyle Russell <bkylerussell@gmail.com>
---
 meta/recipes-gnome/librsvg/librsvg_2.40.16.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-gnome/librsvg/librsvg_2.40.16.bb b/meta/recipes-gnome/librsvg/librsvg_2.40.16.bb
index ed54d9d..3015f1b 100644
--- a/meta/recipes-gnome/librsvg/librsvg_2.40.16.bb
+++ b/meta/recipes-gnome/librsvg/librsvg_2.40.16.bb
@@ -42,4 +42,4 @@ FILES_librsvg-gtk = "${libdir}/gdk-pixbuf-2.0/*/*/*.so"
 
 PIXBUF_PACKAGES = "librsvg-gtk"
 
-PIXBUFCACHE_SYSROOT_DEPS_append_class-native = " harfbuzz-native:do_populate_sysroot_setscene pango-native:do_populate_sysroot_setscene icu-native:do_populate_sysroot_setscene"
+PIXBUFCACHE_SYSROOT_DEPS_append_class-native = " harfbuzz-native:do_populate_sysroot_setscene pango-native:do_populate_sysroot_setscene icu-native:do_populate_sysroot_setscene libxdmcp-native:do_populate_sysroot_setscene pixman-native:do_populate_sysroot_setscene"
-- 
2.7.4

Re: [PATCH 2/2] librsvg: Add missing pixbufcache dependencies

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

On 11 October 2016 at 15:17, Kyle Russell <bkylerussell@gmail.com> wrote:

> g_module_open() failed for libpixbufloader-svg.so: libXdmcp.so.6:
> cannot open shared object file: No such file or directory
>

Can you replicate this failure on demand?  I can't see why eg xdmcp needs
to be in this list but say libxcb doesn't, which is presumably there that
linkage is coming from.

One solution for xdmcp would be to remove it entirely...

Ross

[-- Attachment #2: Type: text/html, Size: 889 bytes --]

Re: [PATCH 2/2] librsvg: Add missing pixbufcache dependencies

[Kyle Russell]

[-- Attachment #1: Type: text/plain, Size: 1646 bytes --]

No, I haven't been able to reproduce it on demand, so there appears to be
some race scenario.  I have only seen the failure about libXdmcp a couple
of times.  It does look like libpixbufloader-svg.so depends on libxcb.so as
well, so if libxcb-native would be more appropriate here I can use that
instead (or in addition to libxdmcp?).

I suspect this list may still not be entirely complete, because ldd shows
several dependencies for libpixbufloader-svg.so.  However, I'm not really
that familiar with all the dependencies going on here; we are definitely
seeing sporadic failures for various native libraries during the
pixbufcache postinst task though.  Best I could tell, it looked like the
PIXBUFCACHE_SYSROOT_DEPS functionality was accidentally removed from the
pixbufcache.bbclass a while back (hence, my first patch).  Adding that hunk
back in has seemed to help most of the failures, but we still see lingering
failures for pixman and libXdmcp on occasion.

If PIXBUFCACHE_SYSROOT_DEPS is supposed to be satisfied another way, I'll
be glad to reevaluate.

On Tue, Oct 11, 2016 at 10:57 AM, Burton, Ross <ross.burton@intel.com>
wrote:

>
> On 11 October 2016 at 15:17, Kyle Russell <bkylerussell@gmail.com> wrote:
>
>> g_module_open() failed for libpixbufloader-svg.so: libXdmcp.so.6:
>> cannot open shared object file: No such file or directory
>>
>
> Can you replicate this failure on demand?  I can't see why eg xdmcp needs
> to be in this list but say libxcb doesn't, which is presumably there that
> linkage is coming from.
>
> One solution for xdmcp would be to remove it entirely...
>
> Ross
>

[-- Attachment #2: Type: text/html, Size: 2473 bytes --]

Re: [PATCH 1/2] pixbufcache: Ensure sysroot dependencies are satisfied

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2088 bytes --]

On Tue, Oct 11, 2016 at 10:17:16AM -0400, Kyle Russell wrote:
> Reverts part of b411085fefc8582d24c172db5f6610705eb44c2f
> 
> Makes sure gdk-pixbuf-native populates the sysroot during setscene;
> otherwise, gdk-pixbuf-query-loaders may not actually be available at the
> time the completion function runs, causing a build failure.
> 
> Similarly, packages providing loader libraries should append to
> PIXBUFCACHE_SYSROOT_DEPS with their runtime dependencies.

Is this related to
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10102
? Or are you seeing completely different errors?

> Signed-off-by: Kyle Russell <bkylerussell@gmail.com>
> ---
>  meta/classes/pixbufcache.bbclass | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/meta/classes/pixbufcache.bbclass b/meta/classes/pixbufcache.bbclass
> index dbe11e1..3c1e398 100644
> --- a/meta/classes/pixbufcache.bbclass
> +++ b/meta/classes/pixbufcache.bbclass
> @@ -65,3 +65,17 @@ pixbufcache_sstate_postinst() {
>  	fi
>  }
>  
> +# Add all of the dependencies of gdk-pixbuf as dependencies of
> +# do_populate_sysroot_setscene so that pixbufcache_sstate_postinst can work
> +# (otherwise gdk-pixbuf-query-loaders may not exist or link). Only add
> +# gdk-pixbuf-native if we're not building gdk-pixbuf itself.
> +#
> +# Packages that use this class should extend this variable with their runtime
> +# dependencies.
> +PIXBUFCACHE_SYSROOT_DEPS = ""
> +PIXBUFCACHE_SYSROOT_DEPS_class-native = "\
> +    ${@['gdk-pixbuf-native:do_populate_sysroot_setscene', '']['${BPN}' == 'gdk-pixbuf']} \
> +"
> +
> +do_populate_sysroot_setscene[depends] += "${PIXBUFCACHE_SYSROOT_DEPS}"
> +do_populate_sysroot[depends] += "${@d.getVar('PIXBUFCACHE_SYSROOT_DEPS', True).replace('_setscene','')}"
> -- 
> 2.7.4
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

Re: [PATCH 1/2] pixbufcache: Ensure sysroot dependencies are satisfied

[Kyle Russell]

[-- Attachment #1: Type: text/plain, Size: 3504 bytes --]

The symptoms sound very similar as far as the postinst exception; however,
I don't think I've seen an error specifically for libpcre before.  I think
in one instance, gdk-pixbuf-query-loaders itself was missing, and in
another instance, one of the libraries provided by harfbuzz. Looking
through the logs, the do_populate_sysroot_setscene tasks for these packages
never ran. (My memory might be off slightly about the specific packages
that were missing, though in each failure instance they were listed in the
librsvg PIXBUFCACHE_SYSROOT_DEPS variable.)

Since PIXBUFCACHE_SYSROOT_DEPS didn't appear to be used anywhere, and we
kept hitting what appeared to be dependency issues with setscene tasks not
running, we thought maybe this hunk was inadvertantly removed from
pixbufcache.bbclass. We've been running with this patch for a few weeks on
our jethro branch and haven't seen any subsequent postinst failures (except
the two packages I addressed in my second patch). The logs from the failed
builds have already been reaped from the build slave, or I could go back to
verify exactly which error messages we encountered in postinst.

Should these setscene dependencies be satisfied another way?

On Tue, Oct 11, 2016 at 2:29 PM, Martin Jansa <martin.jansa@gmail.com>
wrote:

> On Tue, Oct 11, 2016 at 10:17:16AM -0400, Kyle Russell wrote:
> > Reverts part of b411085fefc8582d24c172db5f6610705eb44c2f
> >
> > Makes sure gdk-pixbuf-native populates the sysroot during setscene;
> > otherwise, gdk-pixbuf-query-loaders may not actually be available at the
> > time the completion function runs, causing a build failure.
> >
> > Similarly, packages providing loader libraries should append to
> > PIXBUFCACHE_SYSROOT_DEPS with their runtime dependencies.
>
> Is this related to
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=10102
> ? Or are you seeing completely different errors?
>
> > Signed-off-by: Kyle Russell <bkylerussell@gmail.com>
> > ---
> >  meta/classes/pixbufcache.bbclass | 14 ++++++++++++++
> >  1 file changed, 14 insertions(+)
> >
> > diff --git a/meta/classes/pixbufcache.bbclass
> b/meta/classes/pixbufcache.bbclass
> > index dbe11e1..3c1e398 100644
> > --- a/meta/classes/pixbufcache.bbclass
> > +++ b/meta/classes/pixbufcache.bbclass
> > @@ -65,3 +65,17 @@ pixbufcache_sstate_postinst() {
> >       fi
> >  }
> >
> > +# Add all of the dependencies of gdk-pixbuf as dependencies of
> > +# do_populate_sysroot_setscene so that pixbufcache_sstate_postinst can
> work
> > +# (otherwise gdk-pixbuf-query-loaders may not exist or link). Only add
> > +# gdk-pixbuf-native if we're not building gdk-pixbuf itself.
> > +#
> > +# Packages that use this class should extend this variable with their
> runtime
> > +# dependencies.
> > +PIXBUFCACHE_SYSROOT_DEPS = ""
> > +PIXBUFCACHE_SYSROOT_DEPS_class-native = "\
> > +    ${@['gdk-pixbuf-native:do_populate_sysroot_setscene', '']['${BPN}'
> == 'gdk-pixbuf']} \
> > +"
> > +
> > +do_populate_sysroot_setscene[depends] += "${PIXBUFCACHE_SYSROOT_DEPS}"
> > +do_populate_sysroot[depends] += "${@d.getVar('PIXBUFCACHE_SYSROOT_DEPS',
> True).replace('_setscene','')}"
> > --
> > 2.7.4
> >
> > --
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
> --
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com
>

[-- Attachment #2: Type: text/html, Size: 5063 bytes --]

Re: [PATCH 2/2] librsvg: Add missing pixbufcache dependencies

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 2184 bytes --]

On 11 October 2016 at 17:44, Kyle Russell <bkylerussell@gmail.com> wrote:

> I suspect this list may still not be entirely complete, because ldd shows
> several dependencies for libpixbufloader-svg.so.  However, I'm not really
> that familiar with all the dependencies going on here; we are definitely
> seeing sporadic failures for various native libraries during the
> pixbufcache postinst task though.  Best I could tell, it looked like the
> PIXBUFCACHE_SYSROOT_DEPS functionality was accidentally removed from the
> pixbufcache.bbclass a while back (hence, my first patch).  Adding that hunk
> back in has seemed to help most of the failures, but we still see lingering
> failures for pixman and libXdmcp on occasion.


This bit of code is particularly tangled, so I'm hoping I can remember the
details.  The AB just hit a related problem earlier in the week so I'm
trying to remember how all this ties together.

The SYSROOT_DEPS bit wasn't removed accidentally because the way the
query-loaders script is executed has changed.  It used to be ran via
SSTATEPOSTINSTFUNCS so would execute the moment that gdk-pixbuf-native or
librsvg-native was unpacked.  As sstate installation is top-down instead of
bottom-up this would generally mean that librsvg-native would be installed
before even gdk-pixbuf-native was installed, and typically before stuff
like libpng-native or the rest of the stack.

This was changed so that the SSTATEPOSTINSTFUNC simply writes a
sstate-completion script that is executed once all of sstate has been
unpacked, so ordering isn't a problem. The logic to add explicit
dependencies isn't required anymore.

Of course native dependencies are not as neat as target dependencies which
is something I'd really like to see fixed, but in general I can't
understand why this happens for you: if gdk-pixbuf-native can be extracted
from sstate then all of its dependencies must have been too, and they
include recipes such as libxdmcp-native.  If the dependencies need a
rebuild then gdk-pixbuf should need a rebuild too...

A full cooker log with the bug would be appreciated, but I know that's
unlikely to happen!

Ross

[-- Attachment #2: Type: text/html, Size: 2844 bytes --]

Re: [PATCH 2/2] librsvg: Add missing pixbufcache dependencies

[Kyle Russell]

[-- Attachment #1: Type: text/plain, Size: 3356 bytes --]

Actually, we just hit another failure mode running with these two patches.
Again, we're currently running on the jethro branch.

g_module_open() failed for
/poky/build/tmp/sysroots/x86_64-linux/usr/lib/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-jpeg.so:
libjpeg.so.9: cannot open shared object file: No such file or directory

Looking at the build output itself, I was reminded of another signature of
this failure.  It seems like the do_populate_lice_setscene runs for the
package that should have provided libjpeg.so.9, but no
do_populate_sysroot_setscene.  I remember also seeing this on the other
failures I mentioned.

$ cat build-output.log | grep jpeg-native
NOTE: recipe jpeg-native-9a-r0: task do_populate_lic_setscene: Started
NOTE: recipe jpeg-native-9a-r0: task do_populate_lic_setscene: Succeeded

I can try to get the full cooker log from the build slave if you want.  Do
you mean the file from tmp/log/cooker/<machine>/?  If I can get it, how
would you prefer me send that to you?

On Thu, Oct 13, 2016 at 9:09 AM, Burton, Ross <ross.burton@intel.com> wrote:

>
> On 11 October 2016 at 17:44, Kyle Russell <bkylerussell@gmail.com> wrote:
>
>> I suspect this list may still not be entirely complete, because ldd shows
>> several dependencies for libpixbufloader-svg.so.  However, I'm not really
>> that familiar with all the dependencies going on here; we are definitely
>> seeing sporadic failures for various native libraries during the
>> pixbufcache postinst task though.  Best I could tell, it looked like the
>> PIXBUFCACHE_SYSROOT_DEPS functionality was accidentally removed from the
>> pixbufcache.bbclass a while back (hence, my first patch).  Adding that hunk
>> back in has seemed to help most of the failures, but we still see lingering
>> failures for pixman and libXdmcp on occasion.
>
>
> This bit of code is particularly tangled, so I'm hoping I can remember the
> details.  The AB just hit a related problem earlier in the week so I'm
> trying to remember how all this ties together.
>
> The SYSROOT_DEPS bit wasn't removed accidentally because the way the
> query-loaders script is executed has changed.  It used to be ran via
> SSTATEPOSTINSTFUNCS so would execute the moment that gdk-pixbuf-native or
> librsvg-native was unpacked.  As sstate installation is top-down instead of
> bottom-up this would generally mean that librsvg-native would be installed
> before even gdk-pixbuf-native was installed, and typically before stuff
> like libpng-native or the rest of the stack.
>
> This was changed so that the SSTATEPOSTINSTFUNC simply writes a
> sstate-completion script that is executed once all of sstate has been
> unpacked, so ordering isn't a problem. The logic to add explicit
> dependencies isn't required anymore.
>
> Of course native dependencies are not as neat as target dependencies which
> is something I'd really like to see fixed, but in general I can't
> understand why this happens for you: if gdk-pixbuf-native can be extracted
> from sstate then all of its dependencies must have been too, and they
> include recipes such as libxdmcp-native.  If the dependencies need a
> rebuild then gdk-pixbuf should need a rebuild too...
>
> A full cooker log with the bug would be appreciated, but I know that's
> unlikely to happen!
>
> Ross
>

[-- Attachment #2: Type: text/html, Size: 4444 bytes --]

Re: [PATCH 2/2] librsvg: Add missing pixbufcache dependencies

[Kyle Russell]

[-- Attachment #1: Type: text/plain, Size: 3864 bytes --]

We were able to capture a full cooker log with the libpixbufloader-jpeg
failure, and I only see the populate_lic_setscene tasks being run for
jpeg-native.  We were running jethro with the two patches I originally
provided in this thread.

Is there a specific task dependency chain you'd like for me to look for in
the log?

On Fri, Oct 14, 2016 at 5:55 AM, Kyle Russell <bkylerussell@gmail.com>
wrote:

> Actually, we just hit another failure mode running with these two
> patches.  Again, we're currently running on the jethro branch.
>
> g_module_open() failed for /poky/build/tmp/sysroots/x86_
> 64-linux/usr/lib/gdk-pixbuf-2.0/2.10.0/loaders/libpixbufloader-jpeg.so:
> libjpeg.so.9: cannot open shared object file: No such file or directory
>
> Looking at the build output itself, I was reminded of another signature of
> this failure.  It seems like the do_populate_lice_setscene runs for the
> package that should have provided libjpeg.so.9, but no
> do_populate_sysroot_setscene.  I remember also seeing this on the other
> failures I mentioned.
>
> $ cat build-output.log | grep jpeg-native
> NOTE: recipe jpeg-native-9a-r0: task do_populate_lic_setscene: Started
> NOTE: recipe jpeg-native-9a-r0: task do_populate_lic_setscene: Succeeded
>
> I can try to get the full cooker log from the build slave if you want.  Do
> you mean the file from tmp/log/cooker/<machine>/?  If I can get it, how
> would you prefer me send that to you?
>
> On Thu, Oct 13, 2016 at 9:09 AM, Burton, Ross <ross.burton@intel.com>
> wrote:
>
>>
>> On 11 October 2016 at 17:44, Kyle Russell <bkylerussell@gmail.com> wrote:
>>
>>> I suspect this list may still not be entirely complete, because ldd
>>> shows several dependencies for libpixbufloader-svg.so.  However, I'm not
>>> really that familiar with all the dependencies going on here; we are
>>> definitely seeing sporadic failures for various native libraries during the
>>> pixbufcache postinst task though.  Best I could tell, it looked like the
>>> PIXBUFCACHE_SYSROOT_DEPS functionality was accidentally removed from the
>>> pixbufcache.bbclass a while back (hence, my first patch).  Adding that hunk
>>> back in has seemed to help most of the failures, but we still see lingering
>>> failures for pixman and libXdmcp on occasion.
>>
>>
>> This bit of code is particularly tangled, so I'm hoping I can remember
>> the details.  The AB just hit a related problem earlier in the week so I'm
>> trying to remember how all this ties together.
>>
>> The SYSROOT_DEPS bit wasn't removed accidentally because the way the
>> query-loaders script is executed has changed.  It used to be ran via
>> SSTATEPOSTINSTFUNCS so would execute the moment that gdk-pixbuf-native or
>> librsvg-native was unpacked.  As sstate installation is top-down instead of
>> bottom-up this would generally mean that librsvg-native would be installed
>> before even gdk-pixbuf-native was installed, and typically before stuff
>> like libpng-native or the rest of the stack.
>>
>> This was changed so that the SSTATEPOSTINSTFUNC simply writes a
>> sstate-completion script that is executed once all of sstate has been
>> unpacked, so ordering isn't a problem. The logic to add explicit
>> dependencies isn't required anymore.
>>
>> Of course native dependencies are not as neat as target dependencies
>> which is something I'd really like to see fixed, but in general I can't
>> understand why this happens for you: if gdk-pixbuf-native can be extracted
>> from sstate then all of its dependencies must have been too, and they
>> include recipes such as libxdmcp-native.  If the dependencies need a
>> rebuild then gdk-pixbuf should need a rebuild too...
>>
>> A full cooker log with the bug would be appreciated, but I know that's
>> unlikely to happen!
>>
>> Ross
>>
>
>

[-- Attachment #2: Type: text/html, Size: 5248 bytes --]