[PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Yao Zhao]

when bzip2-native is installed in parallel to sysroot, it is possible that
some packages are using bzip2 to unpack, there are chances that bzip2 is
installed to sysroot but libbz2.so.0 not installed yet because parallel
installation.
link bzip2 and bzip2recover statically to avoid this problem and don't lose
parallel installation. libbz2.so is still available.

Signed-off-by: Yao Zhao <yao.zhao@windriver.com>
---
 meta/recipes-extended/bzip2/bzip2_1.0.6.bb |    7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/meta/recipes-extended/bzip2/bzip2_1.0.6.bb b/meta/recipes-extended/bzip2/bzip2_1.0.6.bb
index 43b462a..4a0ad0c 100644
--- a/meta/recipes-extended/bzip2/bzip2_1.0.6.bb
+++ b/meta/recipes-extended/bzip2/bzip2_1.0.6.bb
@@ -25,6 +25,13 @@ ALTERNATIVE_PRIORITY = "100"
 ALTERNATIVE_${PN} = "bunzip2 bzcat"
 
 do_configure_prepend () {
+	#link libbz2 statically to avoid problems when bzip2-native was
+	#installed parallel, libbz2.so.0 was not available but bzip2 is
+	if [ "${PN}" = "${BPN}-native" ]; then
+	  sed -i -e '/^bzip2_DEPENDENCIES/a bzip2_LDFLAGS = -static' \
+	    -e '/^bzip2recover_DEPENDENCIES/a bzip2recover_LDFLAGS = -static' \
+	    ${WORKDIR}/Makefile.am
+	fi
 	cp ${WORKDIR}/configure.ac ${S}/
 	cp ${WORKDIR}/Makefile.am ${S}/
 	cp ${STAGING_DATADIR_NATIVE}/automake*/install-sh ${S}/
-- 
1.7.9.5

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Burton, Ross]

On 24 July 2012 14:49, Yao Zhao <yao.zhao@windriver.com> wrote:
> when bzip2-native is installed in parallel to sysroot, it is possible that
> some packages are using bzip2 to unpack, there are chances that bzip2 is
> installed to sysroot but libbz2.so.0 not installed yet because parallel
> installation.
> link bzip2 and bzip2recover statically to avoid this problem and don't lose
> parallel installation. libbz2.so is still available.

Is it me, or is this officially getting silly?  This probably happens
for *every* binary in the sysroot that links to a library, which is
probably a fair proportion of them.  Statically linking every single
one and then special-casing further problems where a static link isn't
sufficient (see pythonnative) just isn't going to scale.

Ross

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Yao Zhao]

On 12-07-24 09:57 AM, Burton, Ross wrote:
> On 24 July 2012 14:49, Yao Zhao <yao.zhao@windriver.com> wrote:
>> when bzip2-native is installed in parallel to sysroot, it is possible that
>> some packages are using bzip2 to unpack, there are chances that bzip2 is
>> installed to sysroot but libbz2.so.0 not installed yet because parallel
>> installation.
>> link bzip2 and bzip2recover statically to avoid this problem and don't lose
>> parallel installation. libbz2.so is still available.
> Is it me, or is this officially getting silly?  This probably happens
> for *every* binary in the sysroot that links to a library, which is
> probably a fair proportion of them.  Statically linking every single
> one and then special-casing further problems where a static link isn't
> sufficient (see pythonnative) just isn't going to scale.
HI Ross,

I agree.
I sent a discussion to openembedded-devel@lists.openembedded.org, 
probably I sent to a wrong list.

Can you give any suggestion or we need a whole solution for all those 
native binaries?

yao
Just forward here:

Hi all,

Got a problem about bzip2-native:

The reason is that when unpacking xextproto-7.2.0.tar.bz2, 
bzip2-native's build is triggered too because of another package's 
dependency. And loader can't find libbz2.so.0 which is in sysroot(host 
bzip2 used libbz2.so.1 so it is sure the libbz2.so.0 in sysroot) I have 
searched the paths in PATH below and only find bzip2 in 2 places: host 
/bin/bzip2 and sysroot bzip2.

In bzip2's Makefile, the install is make -j xxx ... although the install 
is first to install libbz2.so.0 then bzip2 but because of the "-j xxx" 
so it is possible that bzip2 is installed first but libbz2.so.0 is not 
ready yet(in a subshell this link is created).
So I have to make the install atomic or at least don't install in parallel.

so there are a couple of workarounds:
1.set PARALLEL_MAKEINST="" (one line change, and only one library, 2 
binaries installed(bzip2, bzip2recover) + 3 scripts + a couple of links)
2.make a statically linked bzip2 and bzip2recover
3.like perl or python native to make a bzip2-native subdir in sysroot 
then only packages explicitly inherit bzip2native which will set the path.

I need your advice!

thanks,
yao
> Ross
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Richard Purdie]

On Tue, 2012-07-24 at 14:57 +0100, Burton, Ross wrote:
> On 24 July 2012 14:49, Yao Zhao <yao.zhao@windriver.com> wrote:
> > when bzip2-native is installed in parallel to sysroot, it is possible that
> > some packages are using bzip2 to unpack, there are chances that bzip2 is
> > installed to sysroot but libbz2.so.0 not installed yet because parallel
> > installation.
> > link bzip2 and bzip2recover statically to avoid this problem and don't lose
> > parallel installation. libbz2.so is still available.
> 
> Is it me, or is this officially getting silly?  This probably happens
> for *every* binary in the sysroot that links to a library, which is
> probably a fair proportion of them.  Statically linking every single
> one and then special-casing further problems where a static link isn't
> sufficient (see pythonnative) just isn't going to scale.

It happens for things in ASSUME_PROVIDED so there is only a finite list
of these issues. I'm curious what is actually triggering bzip2-native to
build given its in ASSUME_PROVIDED...

Cheers,

Richard

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Phil Blundell]

On Tue, 2012-07-24 at 09:49 -0400, Yao Zhao wrote:
> when bzip2-native is installed in parallel to sysroot, it is possible that
> some packages are using bzip2 to unpack, there are chances that bzip2 is
> installed to sysroot but libbz2.so.0 not installed yet because parallel
> installation.
> link bzip2 and bzip2recover statically to avoid this problem and don't lose
> parallel installation. libbz2.so is still available.

Can the makefile not be fixed to get libbz2.so.0 installed before bzip?

p.

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Yao Zhao]

On 12-07-24 11:39 AM, Phil Blundell wrote:
> On Tue, 2012-07-24 at 09:49 -0400, Yao Zhao wrote:
>> when bzip2-native is installed in parallel to sysroot, it is possible that
>> some packages are using bzip2 to unpack, there are chances that bzip2 is
>> installed to sysroot but libbz2.so.0 not installed yet because parallel
>> installation.
>> link bzip2 and bzip2recover statically to avoid this problem and don't lose
>> parallel installation. libbz2.so is still available.
> Can the makefile not be fixed to get libbz2.so.0 installed before bzip?
In the Makefile, libbz2.so.0 is installed before bzip2, but the 
installation is "make -j xxx install". Then this could happen.

yao
> p.
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Phil Blundell]

On Tue, 2012-07-24 at 11:45 -0400, Yao Zhao wrote:
> In the Makefile, libbz2.so.0 is installed before bzip2, but the 
> installation is "make -j xxx install". Then this could happen.

Well, yes, but you could fix that by adding an appropriate dependency.
Right?

p.

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Yao Zhao]

On 12-07-24 11:34 AM, Richard Purdie wrote:
> On Tue, 2012-07-24 at 14:57 +0100, Burton, Ross wrote:
>> On 24 July 2012 14:49, Yao Zhao <yao.zhao@windriver.com> wrote:
>>> when bzip2-native is installed in parallel to sysroot, it is possible that
>>> some packages are using bzip2 to unpack, there are chances that bzip2 is
>>> installed to sysroot but libbz2.so.0 not installed yet because parallel
>>> installation.
>>> link bzip2 and bzip2recover statically to avoid this problem and don't lose
>>> parallel installation. libbz2.so is still available.
>> Is it me, or is this officially getting silly?  This probably happens
>> for *every* binary in the sysroot that links to a library, which is
>> probably a fair proportion of them.  Statically linking every single
>> one and then special-casing further problems where a static link isn't
>> sufficient (see pythonnative) just isn't going to scale.
> It happens for things in ASSUME_PROVIDED so there is only a finite list
> of these issues. I'm curious what is actually triggering bzip2-native to
> build given its in ASSUME_PROVIDED...
Yeah, it is in the ASSUME_PROVIDED but somehow it was built too. I will 
take a further look at why it was built.

thanks,
yao
> Cheers,
>
> Richard
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Yao Zhao]

On 12-07-24 11:47 AM, Phil Blundell wrote:
> On Tue, 2012-07-24 at 11:45 -0400, Yao Zhao wrote:
>> In the Makefile, libbz2.so.0 is installed before bzip2, but the
>> installation is "make -j xxx install". Then this could happen.
> Well, yes, but you could fix that by adding an appropriate dependency.
> Right?
I relooked at the Makefile again:
install-binPROGRAMS: $(bin_PROGRAMS)
         @$(NORMAL_INSTALL)
         @list='$(bin_PROGRAMS)'; test -n "$(bindir)" || list=; \
         if test -n "$$list"; then \
           echo " $(MKDIR_P) '$(DESTDIR)$(bindir)'"; \
           $(MKDIR_P) "$(DESTDIR)$(bindir)" || exit 1; \
         fi; \
         for p in $$list; do echo "$$p $$p"; done | \
         sed 's/$(EXEEXT)$$//' | \
         while read p p1; do if test -f $$p || test -f $$p1; \
           then echo "$$p"; echo "$$p"; else :; fi; \
         done | \
         sed -e 'p;s,.*/,,;n;h' -e 's|.*|.|' \
             -e 
'p;x;s,.*/,,;s/$(EXEEXT)$$//;$(transform);s/$$/$(EXEEXT)/' | \
         sed 'N;N;N;s,\n, ,g' | \
         $(AWK) 'BEGIN { files["."] = ""; dirs["."] = 1 } \
           { d=$$3; if (dirs[d] != 1) { print "d", d; dirs[d] = 1 } \
             if ($$2 == $$4) files[d] = files[d] " " $$1; \
             else { print "f", $$3 "/" $$4, $$1; } } \
           END { for (d in files) print "f", d, files[d] }' | \
         while read type dir files; do \
             if test "$$dir" = .; then dir=; else dir=/$$dir; fi; \
             test -z "$$files" || { \
.....

install-binPROGRAMS: install-libLTLIBRARIES

install-exec-am: install-binPROGRAMS install-binSCRIPTS \
         install-libLTLIBRARIES
         @$(NORMAL_INSTALL)
         $(MAKE) $(AM_MAKEFLAGS) install-exec-hook

My first thought is that install-libLTLIBRARIES is a dependency of 
install-binPROGRAMS, but there are 2 install-binPROGRAMS targets, how it 
works?

thanks,
yao
> p.
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Mark Hatle]

On 7/24/12 11:39 AM, Yao Zhao wrote:
> On 12-07-24 11:47 AM, Phil Blundell wrote:
>> On Tue, 2012-07-24 at 11:45 -0400, Yao Zhao wrote:
>>> In the Makefile, libbz2.so.0 is installed before bzip2, but the
>>> installation is "make -j xxx install". Then this could happen.
>> Well, yes, but you could fix that by adding an appropriate dependency.
>> Right?
> I relooked at the Makefile again:
> install-binPROGRAMS: $(bin_PROGRAMS)
>           @$(NORMAL_INSTALL)
>           @list='$(bin_PROGRAMS)'; test -n "$(bindir)" || list=; \
>           if test -n "$$list"; then \
>             echo " $(MKDIR_P) '$(DESTDIR)$(bindir)'"; \
>             $(MKDIR_P) "$(DESTDIR)$(bindir)" || exit 1; \
>           fi; \
>           for p in $$list; do echo "$$p $$p"; done | \
>           sed 's/$(EXEEXT)$$//' | \
>           while read p p1; do if test -f $$p || test -f $$p1; \
>             then echo "$$p"; echo "$$p"; else :; fi; \
>           done | \
>           sed -e 'p;s,.*/,,;n;h' -e 's|.*|.|' \
>               -e
> 'p;x;s,.*/,,;s/$(EXEEXT)$$//;$(transform);s/$$/$(EXEEXT)/' | \
>           sed 'N;N;N;s,\n, ,g' | \
>           $(AWK) 'BEGIN { files["."] = ""; dirs["."] = 1 } \
>             { d=$$3; if (dirs[d] != 1) { print "d", d; dirs[d] = 1 } \
>               if ($$2 == $$4) files[d] = files[d] " " $$1; \
>               else { print "f", $$3 "/" $$4, $$1; } } \
>             END { for (d in files) print "f", d, files[d] }' | \
>           while read type dir files; do \
>               if test "$$dir" = .; then dir=; else dir=/$$dir; fi; \
>               test -z "$$files" || { \
> .....
>
> install-binPROGRAMS: install-libLTLIBRARIES
>
> install-exec-am: install-binPROGRAMS install-binSCRIPTS \
>           install-libLTLIBRARIES
>           @$(NORMAL_INSTALL)
>           $(MAKE) $(AM_MAKEFLAGS) install-exec-hook
>
> My first thought is that install-libLTLIBRARIES is a dependency of
> install-binPROGRAMS, but there are 2 install-binPROGRAMS targets, how it
> works?

In the second one, as long as there are no white space below the line, it's just 
a \n.  Then it adds the item to the dependency list..  So effectively

install-binPROGRAMS: $(bin_PROGRAMS) install-libLTLIBRARIES

(if there is a tab or other white space it should generate a warning or error 
about multiple definitions of install-binPROGRAMS, and in that case, the last 
definition wins.)

--Mark

> thanks,
> yao
>> p.
>>
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Yao Zhao]

On 12-07-24 12:52 PM, Mark Hatle wrote:
> On 7/24/12 11:39 AM, Yao Zhao wrote:
>> On 12-07-24 11:47 AM, Phil Blundell wrote:
>>> On Tue, 2012-07-24 at 11:45 -0400, Yao Zhao wrote:
>>>> In the Makefile, libbz2.so.0 is installed before bzip2, but the
>>>> installation is "make -j xxx install". Then this could happen.
>>> Well, yes, but you could fix that by adding an appropriate dependency.
>>> Right?
>> I relooked at the Makefile again:
>> install-binPROGRAMS: $(bin_PROGRAMS)
>>           @$(NORMAL_INSTALL)
>>           @list='$(bin_PROGRAMS)'; test -n "$(bindir)" || list=; \
>>           if test -n "$$list"; then \
>>             echo " $(MKDIR_P) '$(DESTDIR)$(bindir)'"; \
>>             $(MKDIR_P) "$(DESTDIR)$(bindir)" || exit 1; \
>>           fi; \
>>           for p in $$list; do echo "$$p $$p"; done | \
>>           sed 's/$(EXEEXT)$$//' | \
>>           while read p p1; do if test -f $$p || test -f $$p1; \
>>             then echo "$$p"; echo "$$p"; else :; fi; \
>>           done | \
>>           sed -e 'p;s,.*/,,;n;h' -e 's|.*|.|' \
>>               -e
>> 'p;x;s,.*/,,;s/$(EXEEXT)$$//;$(transform);s/$$/$(EXEEXT)/' | \
>>           sed 'N;N;N;s,\n, ,g' | \
>>           $(AWK) 'BEGIN { files["."] = ""; dirs["."] = 1 } \
>>             { d=$$3; if (dirs[d] != 1) { print "d", d; dirs[d] = 1 } \
>>               if ($$2 == $$4) files[d] = files[d] " " $$1; \
>>               else { print "f", $$3 "/" $$4, $$1; } } \
>>             END { for (d in files) print "f", d, files[d] }' | \
>>           while read type dir files; do \
>>               if test "$$dir" = .; then dir=; else dir=/$$dir; fi; \
>>               test -z "$$files" || { \
>> .....
>>
>> install-binPROGRAMS: install-libLTLIBRARIES
>>
>> install-exec-am: install-binPROGRAMS install-binSCRIPTS \
>>           install-libLTLIBRARIES
>>           @$(NORMAL_INSTALL)
>>           $(MAKE) $(AM_MAKEFLAGS) install-exec-hook
>>
>> My first thought is that install-libLTLIBRARIES is a dependency of
>> install-binPROGRAMS, but there are 2 install-binPROGRAMS targets, how it
>> works?
>
> In the second one, as long as there are no white space below the line, 
> it's just a \n.  Then it adds the item to the dependency list..  So 
> effectively
>
> install-binPROGRAMS: $(bin_PROGRAMS) install-libLTLIBRARIES
>
> (if there is a tab or other white space it should generate a warning 
> or error about multiple definitions of install-binPROGRAMS, and in 
> that case, the last definition wins.)
>
Thanks Mark!
So I didn't understand wrongly about the dependency, 
install-libLTLIBRARIES is install-binPROGRAMS's dependency.
according to make document, -j jobs specifies the number of 
jobs(commands) to run simultaneously. My understanding is the commands 
for each target is the jobs or command the doc talks about, am I right?
In this case, will "make -j xx" even didn't care the dependency? or make 
did care but they are paralleled to executed.

thanks,
yao
> --Mark
>
>> thanks,
>> yao
>>> p.
>>>
>>>
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Yao Zhao]

On 12-07-24 12:00 PM, Yao Zhao wrote:
> On 12-07-24 11:34 AM, Richard Purdie wrote:
>> On Tue, 2012-07-24 at 14:57 +0100, Burton, Ross wrote:
>>> On 24 July 2012 14:49, Yao Zhao <yao.zhao@windriver.com> wrote:
>>>> when bzip2-native is installed in parallel to sysroot, it is 
>>>> possible that
>>>> some packages are using bzip2 to unpack, there are chances that 
>>>> bzip2 is
>>>> installed to sysroot but libbz2.so.0 not installed yet because 
>>>> parallel
>>>> installation.
>>>> link bzip2 and bzip2recover statically to avoid this problem and 
>>>> don't lose
>>>> parallel installation. libbz2.so is still available.
>>> Is it me, or is this officially getting silly?  This probably happens
>>> for *every* binary in the sysroot that links to a library, which is
>>> probably a fair proportion of them.  Statically linking every single
>>> one and then special-casing further problems where a static link isn't
>>> sufficient (see pythonnative) just isn't going to scale.
>> It happens for things in ASSUME_PROVIDED so there is only a finite list
>> of these issues. I'm curious what is actually triggering bzip2-native to
>> build given its in ASSUME_PROVIDED...
> Yeah, it is in the ASSUME_PROVIDED but somehow it was built too. I 
> will take a further look at why it was built.
>
It seems that python-native is depending on "bzip2-full-native" and 
bzip2 does provide this "bzip2-full-native".
Change it from "bzip2-full-native" to "bzip2-native" , bzip2-native is 
not built any more. The code may not check the
Why we need the bzip2-full-native? any idea?

thanks,
yao
> thanks,
> yao
>> Cheers,
>>
>> Richard
>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Mark Hatle]

On 7/24/12 8:57 AM, Burton, Ross wrote:
> On 24 July 2012 14:49, Yao Zhao <yao.zhao@windriver.com> wrote:
>> when bzip2-native is installed in parallel to sysroot, it is possible that
>> some packages are using bzip2 to unpack, there are chances that bzip2 is
>> installed to sysroot but libbz2.so.0 not installed yet because parallel
>> installation.
>> link bzip2 and bzip2recover statically to avoid this problem and don't lose
>> parallel installation. libbz2.so is still available.
>
> Is it me, or is this officially getting silly?  This probably happens
> for *every* binary in the sysroot that links to a library, which is
> probably a fair proportion of them.  Statically linking every single
> one and then special-casing further problems where a static link isn't
> sufficient (see pythonnative) just isn't going to scale.

The problem is that there are a handful of things that are needed, and for some 
reason (valid or not) packages are not "requiring", either because they assume 
the system is valid or they simply don't know they have the requirement.

Both python and perl may be used by a number of auto* utilities as well as by 
packages themselves.  We could attempt to add DEPENDS for all of the cases, but 
is it really worth it?  I suspect we'll end up specify the DEPENDS for 90% of 
the things in the system, simply due to them incidentally running some system 
level python or perl script.  If we make the python/perl into external items 
that only get loaded into the environment when we're building python/perl 
modules then things become increasingly easy.

As for the bzip2 issue.. This is another of those "we assume it's valid 
binaries"..  And the problem appears to be when it gets build, it's not valid 
for a small window of time.

 From what Yao has explained to me (offline) the issue is that the 
assume_provided for bzip2-native works just fine, we use the system 
version...but then python (or other) utilities come in and say they need 
"bzip2-full-native".  Which triggers the build.. and opens a small window of 
time where bzip2 is being installed, where it isn't functional -- something else 
comes in and has a requirement of bzip2-native, which is satisfied by the 
assume_provided and we get into the race where it executes a broken binary.

The underlying issue is simply that if we're installing tooling that is either 
assume-provided (based on an alternative 'full' version) or incidental usage 
like python or perl we need to take precautions to ensure that the build system 
never sees the "broken" version during the short install window triggering the 
race condition.

I'll let Yao comment further on this particular issue, but there is an overall 
class of issues with -native that we need to track to get rid of these subtle races.

--Mark

> Ross
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Richard Purdie]

On Tue, 2012-07-24 at 13:39 -0500, Mark Hatle wrote:
> On 7/24/12 8:57 AM, Burton, Ross wrote:
> > On 24 July 2012 14:49, Yao Zhao <yao.zhao@windriver.com> wrote:
> >> when bzip2-native is installed in parallel to sysroot, it is possible that
> >> some packages are using bzip2 to unpack, there are chances that bzip2 is
> >> installed to sysroot but libbz2.so.0 not installed yet because parallel
> >> installation.
> >> link bzip2 and bzip2recover statically to avoid this problem and don't lose
> >> parallel installation. libbz2.so is still available.
> >
> > Is it me, or is this officially getting silly?  This probably happens
> > for *every* binary in the sysroot that links to a library, which is
> > probably a fair proportion of them.  Statically linking every single
> > one and then special-casing further problems where a static link isn't
> > sufficient (see pythonnative) just isn't going to scale.
> 
> The problem is that there are a handful of things that are needed, and for some 
> reason (valid or not) packages are not "requiring", either because they assume 
> the system is valid or they simply don't know they have the requirement.
> 
> Both python and perl may be used by a number of auto* utilities as well as by 
> packages themselves.  We could attempt to add DEPENDS for all of the cases, but 
> is it really worth it?  I suspect we'll end up specify the DEPENDS for 90% of 
> the things in the system, simply due to them incidentally running some system 
> level python or perl script.  If we make the python/perl into external items 
> that only get loaded into the environment when we're building python/perl 
> modules then things become increasingly easy.
> 
> As for the bzip2 issue.. This is another of those "we assume it's valid 
> binaries"..  And the problem appears to be when it gets build, it's not valid 
> for a small window of time.
> 
>  From what Yao has explained to me (offline) the issue is that the 
> assume_provided for bzip2-native works just fine, we use the system 
> version...but then python (or other) utilities come in and say they need 
> "bzip2-full-native".  Which triggers the build.. and opens a small window of 
> time where bzip2 is being installed, where it isn't functional -- something else 
> comes in and has a requirement of bzip2-native, which is satisfied by the 
> assume_provided and we get into the race where it executes a broken binary.
> 
> The underlying issue is simply that if we're installing tooling that is either 
> assume-provided (based on an alternative 'full' version) or incidental usage 
> like python or perl we need to take precautions to ensure that the build system 
> never sees the "broken" version during the short install window triggering the 
> race condition.
> 
> I'll let Yao comment further on this particular issue, but there is an overall 
> class of issues with -native that we need to track to get rid of these subtle races.

The set of problems are cases where we have things in ASSUME_PROVIDED.
It is near impossible to have an empty set there and sometimes we need
to build things ourselves which are already listed there. python-native
and perl-native are the pathological cases but as also have issues with
bzip2, tar and likely undiscovered yet, now git.

I'd been searching for bzip2-replacement-native rather than full which
is why I didn't see this earlier. I suspect in the python/bzip2 case,
not installing the bzip2 binary might well solve many of the problems as
python really wants a native libbz2 and the host bzip2 binary is ok.

The races themselves are over binaries in PATH. As far as I
know/understand things we're close to having them resolved. What I am
considering as an improvement is instead of the pythonnative/perlnative
bbclass files, having a variable like "NATIVEPATHADDITIONS" which would
take values like "python-native", "perl-native" and would simply add
${STAGING_BINDIR}/xxx to PATH for each item listed. The bbclass files
are useful for the dependency additions too though which sometimes are
not trivial. This would at least remove the case by case bbclass
addition issue we currently have for the simple cases though.

Cheers,

Richard

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Richard Purdie]

On Tue, 2012-07-24 at 14:32 -0400, Yao Zhao wrote:
> On 12-07-24 12:00 PM, Yao Zhao wrote:
> > On 12-07-24 11:34 AM, Richard Purdie wrote:
> >> On Tue, 2012-07-24 at 14:57 +0100, Burton, Ross wrote:
> >>> On 24 July 2012 14:49, Yao Zhao <yao.zhao@windriver.com> wrote:
> >>>> when bzip2-native is installed in parallel to sysroot, it is 
> >>>> possible that
> >>>> some packages are using bzip2 to unpack, there are chances that 
> >>>> bzip2 is
> >>>> installed to sysroot but libbz2.so.0 not installed yet because 
> >>>> parallel
> >>>> installation.
> >>>> link bzip2 and bzip2recover statically to avoid this problem and 
> >>>> don't lose
> >>>> parallel installation. libbz2.so is still available.
> >>> Is it me, or is this officially getting silly?  This probably happens
> >>> for *every* binary in the sysroot that links to a library, which is
> >>> probably a fair proportion of them.  Statically linking every single
> >>> one and then special-casing further problems where a static link isn't
> >>> sufficient (see pythonnative) just isn't going to scale.
> >> It happens for things in ASSUME_PROVIDED so there is only a finite list
> >> of these issues. I'm curious what is actually triggering bzip2-native to
> >> build given its in ASSUME_PROVIDED...
> > Yeah, it is in the ASSUME_PROVIDED but somehow it was built too. I 
> > will take a further look at why it was built.
> >
> It seems that python-native is depending on "bzip2-full-native" and 
> bzip2 does provide this "bzip2-full-native".
> Change it from "bzip2-full-native" to "bzip2-native" , bzip2-native is 
> not built any more. The code may not check the
> Why we need the bzip2-full-native? any idea?

See my other reply but I think its libbz2 that it wants.

Cheers,

Richard

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Yao Zhao]

On 12-07-24 03:07 PM, Richard Purdie wrote:
> On Tue, 2012-07-24 at 13:39 -0500, Mark Hatle wrote:
>> On 7/24/12 8:57 AM, Burton, Ross wrote:
>>> On 24 July 2012 14:49, Yao Zhao <yao.zhao@windriver.com> wrote:
>>>> when bzip2-native is installed in parallel to sysroot, it is possible that
>>>> some packages are using bzip2 to unpack, there are chances that bzip2 is
>>>> installed to sysroot but libbz2.so.0 not installed yet because parallel
>>>> installation.
>>>> link bzip2 and bzip2recover statically to avoid this problem and don't lose
>>>> parallel installation. libbz2.so is still available.
>>> Is it me, or is this officially getting silly?  This probably happens
>>> for *every* binary in the sysroot that links to a library, which is
>>> probably a fair proportion of them.  Statically linking every single
>>> one and then special-casing further problems where a static link isn't
>>> sufficient (see pythonnative) just isn't going to scale.
>> The problem is that there are a handful of things that are needed, and for some
>> reason (valid or not) packages are not "requiring", either because they assume
>> the system is valid or they simply don't know they have the requirement.
>>
>> Both python and perl may be used by a number of auto* utilities as well as by
>> packages themselves.  We could attempt to add DEPENDS for all of the cases, but
>> is it really worth it?  I suspect we'll end up specify the DEPENDS for 90% of
>> the things in the system, simply due to them incidentally running some system
>> level python or perl script.  If we make the python/perl into external items
>> that only get loaded into the environment when we're building python/perl
>> modules then things become increasingly easy.
>>
>> As for the bzip2 issue.. This is another of those "we assume it's valid
>> binaries"..  And the problem appears to be when it gets build, it's not valid
>> for a small window of time.
>>
>>   From what Yao has explained to me (offline) the issue is that the
>> assume_provided for bzip2-native works just fine, we use the system
>> version...but then python (or other) utilities come in and say they need
>> "bzip2-full-native".  Which triggers the build.. and opens a small window of
>> time where bzip2 is being installed, where it isn't functional -- something else
>> comes in and has a requirement of bzip2-native, which is satisfied by the
>> assume_provided and we get into the race where it executes a broken binary.
>>
>> The underlying issue is simply that if we're installing tooling that is either
>> assume-provided (based on an alternative 'full' version) or incidental usage
>> like python or perl we need to take precautions to ensure that the build system
>> never sees the "broken" version during the short install window triggering the
>> race condition.
>>
>> I'll let Yao comment further on this particular issue, but there is an overall
>> class of issues with -native that we need to track to get rid of these subtle races.
> The set of problems are cases where we have things in ASSUME_PROVIDED.
> It is near impossible to have an empty set there and sometimes we need
> to build things ourselves which are already listed there. python-native
> and perl-native are the pathological cases but as also have issues with
> bzip2, tar and likely undiscovered yet, now git.
>
> I'd been searching for bzip2-replacement-native rather than full which
> is why I didn't see this earlier. I suspect in the python/bzip2 case,
> not installing the bzip2 binary might well solve many of the problems as
> python really wants a native libbz2 and the host bzip2 binary is ok.
Hi Richard,

Can I provide a patch to change bzip2-full-native to bzip2-native to 
finish this problem?
I have tested that the bzip2-native is not built any more.

thanks,
yao
> The races themselves are over binaries in PATH. As far as I
> know/understand things we're close to having them resolved. What I am
> considering as an improvement is instead of the pythonnative/perlnative
> bbclass files, having a variable like "NATIVEPATHADDITIONS" which would
> take values like "python-native", "perl-native" and would simply add
> ${STAGING_BINDIR}/xxx to PATH for each item listed. The bbclass files
> are useful for the dependency additions too though which sometimes are
> not trivial. This would at least remove the case by case bbclass
> addition issue we currently have for the simple cases though.
>
> Cheers,
>
> Richard
>
>
>
>
>
>

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Richard Purdie]

On Tue, 2012-07-24 at 15:25 -0400, Yao Zhao wrote:
> On 12-07-24 03:07 PM, Richard Purdie wrote:
> > On Tue, 2012-07-24 at 13:39 -0500, Mark Hatle wrote:
> >> On 7/24/12 8:57 AM, Burton, Ross wrote:
> >>> On 24 July 2012 14:49, Yao Zhao <yao.zhao@windriver.com> wrote:
> >>>> when bzip2-native is installed in parallel to sysroot, it is possible that
> >>>> some packages are using bzip2 to unpack, there are chances that bzip2 is
> >>>> installed to sysroot but libbz2.so.0 not installed yet because parallel
> >>>> installation.
> >>>> link bzip2 and bzip2recover statically to avoid this problem and don't lose
> >>>> parallel installation. libbz2.so is still available.
> >>> Is it me, or is this officially getting silly?  This probably happens
> >>> for *every* binary in the sysroot that links to a library, which is
> >>> probably a fair proportion of them.  Statically linking every single
> >>> one and then special-casing further problems where a static link isn't
> >>> sufficient (see pythonnative) just isn't going to scale.
> >> The problem is that there are a handful of things that are needed, and for some
> >> reason (valid or not) packages are not "requiring", either because they assume
> >> the system is valid or they simply don't know they have the requirement.
> >>
> >> Both python and perl may be used by a number of auto* utilities as well as by
> >> packages themselves.  We could attempt to add DEPENDS for all of the cases, but
> >> is it really worth it?  I suspect we'll end up specify the DEPENDS for 90% of
> >> the things in the system, simply due to them incidentally running some system
> >> level python or perl script.  If we make the python/perl into external items
> >> that only get loaded into the environment when we're building python/perl
> >> modules then things become increasingly easy.
> >>
> >> As for the bzip2 issue.. This is another of those "we assume it's valid
> >> binaries"..  And the problem appears to be when it gets build, it's not valid
> >> for a small window of time.
> >>
> >>   From what Yao has explained to me (offline) the issue is that the
> >> assume_provided for bzip2-native works just fine, we use the system
> >> version...but then python (or other) utilities come in and say they need
> >> "bzip2-full-native".  Which triggers the build.. and opens a small window of
> >> time where bzip2 is being installed, where it isn't functional -- something else
> >> comes in and has a requirement of bzip2-native, which is satisfied by the
> >> assume_provided and we get into the race where it executes a broken binary.
> >>
> >> The underlying issue is simply that if we're installing tooling that is either
> >> assume-provided (based on an alternative 'full' version) or incidental usage
> >> like python or perl we need to take precautions to ensure that the build system
> >> never sees the "broken" version during the short install window triggering the
> >> race condition.
> >>
> >> I'll let Yao comment further on this particular issue, but there is an overall
> >> class of issues with -native that we need to track to get rid of these subtle races.
> > The set of problems are cases where we have things in ASSUME_PROVIDED.
> > It is near impossible to have an empty set there and sometimes we need
> > to build things ourselves which are already listed there. python-native
> > and perl-native are the pathological cases but as also have issues with
> > bzip2, tar and likely undiscovered yet, now git.
> >
> > I'd been searching for bzip2-replacement-native rather than full which
> > is why I didn't see this earlier. I suspect in the python/bzip2 case,
> > not installing the bzip2 binary might well solve many of the problems as
> > python really wants a native libbz2 and the host bzip2 binary is ok.
> Hi Richard,
> 
> Can I provide a patch to change bzip2-full-native to bzip2-native to 
> finish this problem?
> I have tested that the bzip2-native is not built any more.

No, since I think the libbz2 component that python needs will then not
be present and this causes other problems (see the previous commit or
mailing list archives for what the exact issue was, I don't remember
offhand).

Cheers,

Richard

Re: [PATCH] bzip2-native: fix problems when bzip2-native is installed in parallel

[Yao Zhao]

On 12-07-24 03:32 PM, Richard Purdie wrote:
> On Tue, 2012-07-24 at 15:25 -0400, Yao Zhao wrote:
>> On 12-07-24 03:07 PM, Richard Purdie wrote:
>>> On Tue, 2012-07-24 at 13:39 -0500, Mark Hatle wrote:
>>>> On 7/24/12 8:57 AM, Burton, Ross wrote:
>>>>> On 24 July 2012 14:49, Yao Zhao <yao.zhao@windriver.com> wrote:
>>>>>> when bzip2-native is installed in parallel to sysroot, it is possible that
>>>>>> some packages are using bzip2 to unpack, there are chances that bzip2 is
>>>>>> installed to sysroot but libbz2.so.0 not installed yet because parallel
>>>>>> installation.
>>>>>> link bzip2 and bzip2recover statically to avoid this problem and don't lose
>>>>>> parallel installation. libbz2.so is still available.
>>>>> Is it me, or is this officially getting silly?  This probably happens
>>>>> for *every* binary in the sysroot that links to a library, which is
>>>>> probably a fair proportion of them.  Statically linking every single
>>>>> one and then special-casing further problems where a static link isn't
>>>>> sufficient (see pythonnative) just isn't going to scale.
>>>> The problem is that there are a handful of things that are needed, and for some
>>>> reason (valid or not) packages are not "requiring", either because they assume
>>>> the system is valid or they simply don't know they have the requirement.
>>>>
>>>> Both python and perl may be used by a number of auto* utilities as well as by
>>>> packages themselves.  We could attempt to add DEPENDS for all of the cases, but
>>>> is it really worth it?  I suspect we'll end up specify the DEPENDS for 90% of
>>>> the things in the system, simply due to them incidentally running some system
>>>> level python or perl script.  If we make the python/perl into external items
>>>> that only get loaded into the environment when we're building python/perl
>>>> modules then things become increasingly easy.
>>>>
>>>> As for the bzip2 issue.. This is another of those "we assume it's valid
>>>> binaries"..  And the problem appears to be when it gets build, it's not valid
>>>> for a small window of time.
>>>>
>>>>    From what Yao has explained to me (offline) the issue is that the
>>>> assume_provided for bzip2-native works just fine, we use the system
>>>> version...but then python (or other) utilities come in and say they need
>>>> "bzip2-full-native".  Which triggers the build.. and opens a small window of
>>>> time where bzip2 is being installed, where it isn't functional -- something else
>>>> comes in and has a requirement of bzip2-native, which is satisfied by the
>>>> assume_provided and we get into the race where it executes a broken binary.
>>>>
>>>> The underlying issue is simply that if we're installing tooling that is either
>>>> assume-provided (based on an alternative 'full' version) or incidental usage
>>>> like python or perl we need to take precautions to ensure that the build system
>>>> never sees the "broken" version during the short install window triggering the
>>>> race condition.
>>>>
>>>> I'll let Yao comment further on this particular issue, but there is an overall
>>>> class of issues with -native that we need to track to get rid of these subtle races.
>>> The set of problems are cases where we have things in ASSUME_PROVIDED.
>>> It is near impossible to have an empty set there and sometimes we need
>>> to build things ourselves which are already listed there. python-native
>>> and perl-native are the pathological cases but as also have issues with
>>> bzip2, tar and likely undiscovered yet, now git.
>>>
>>> I'd been searching for bzip2-replacement-native rather than full which
>>> is why I didn't see this earlier. I suspect in the python/bzip2 case,
>>> not installing the bzip2 binary might well solve many of the problems as
>>> python really wants a native libbz2 and the host bzip2 binary is ok.
>> Hi Richard,
>>
>> Can I provide a patch to change bzip2-full-native to bzip2-native to
>> finish this problem?
>> I have tested that the bzip2-native is not built any more.
> No, since I think the libbz2 component that python needs will then not
> be present and this causes other problems (see the previous commit or
> mailing list archives for what the exact issue was, I don't remember
> offhand).
An update on this thread.

I made a mistake on this problem. I thought do_install will install to 
sysroot directly for native build but actually not.
The do_install only installs to image so it doesn't matter whether 
do_install is installed paralleled. The installation to sysroot happens 
on populate_sysroot stage and the code do a copytree to copy files,dirs 
from sysroot-destdir to sysroot dir. I didn't debug this python code, 
but debugged the code below when writting to the manifest file.
bzip2 is copied earlier than libbz2.so... so it will happen if in this 
small window, unpacking referencing bzip2.

To me this routine should be written to install by dependency order, 
like packages-split dir. This should fix a lot of native tools problem 
without changing code in upper level. Of course I am not sure it will 
fix python problem.

And today we see another problem:
/bin/sh: 
/builds-2012-07-25-003940/qemux86_std/bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/bzip2: 
Text file busy
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

So looks like even just a copy it may cause problem too.


Is it possible that let python-native only depends on libbz2 sub 
package? I looked at python-native package, when bzip2-native is not 
built, it seems it has problems to find bz2 library although it builds 
still ok.

thanks,
yao
> Cheers,
>
> Richard
>
>
>