Race issues (lzma-native)

Race issues (lzma-native)

[Andrea Adami]

Hello,

during first build of console image I notice a strange fault:
subsequent builds are ok.
My scripts builds console-image and opie-image.

ERROR: Task 2411
(virtual:native:/oe/openembedded/recipes/lzma/lzma_4.65.bb,
do_distribute_sources) failed with exit code '1'
NOTE: Unpacking ../sources/lzma465.tar.bz2 to
tmp/work/i686-linux/lzma-native-4.65-r4.1/
NOTE: package lzma-native-4.65-r4.1: task do_distribute_sources: Started
NOTE: package lzma-native-4.65-r4.1: task SRC_DISTRIBUTECOMMAND: Failed
ERROR: Function 'SRC_DISTRIBUTECOMMAND' failed


Looking at the
run.SRC_DISTRIBUTECOMMAND.18268   16384 (failing)
run.SRC_DISTRIBUTECOMMAND.9570     34334

it looks like the file is not complete / malformed:

package_generate_ipkg_conf() {
<------>mkdir -p /oe/build/tmp/sysroots/i686-linux/etc/
<------>echo "src oe file:/oe/build/tmp/deploy/glibc/ipk" >
/oe/build/tmp/sysroots/i686-linux/etc/opkg.conf
<------>echo "src oe file:/
EOF

run.do_distribute_sources.18268  is empty, while second time
run.do_distribute_sources.9570   1051 contains  def do_distribute_sources(d):

This seems indeed a race issue...

Andrea

Re: Race issues (lzma-native)

[Graham Gower]

On 10/24/2010 07:29 PM, Andrea Adami wrote:
> Hello,
> 
> during first build of console image I notice a strange fault:
> subsequent builds are ok.
> My scripts builds console-image and opie-image.
> 
> ERROR: Task 2411
> (virtual:native:/oe/openembedded/recipes/lzma/lzma_4.65.bb,
> do_distribute_sources) failed with exit code '1'
> NOTE: Unpacking ../sources/lzma465.tar.bz2 to
> tmp/work/i686-linux/lzma-native-4.65-r4.1/
> NOTE: package lzma-native-4.65-r4.1: task do_distribute_sources: Started
> NOTE: package lzma-native-4.65-r4.1: task SRC_DISTRIBUTECOMMAND: Failed
> ERROR: Function 'SRC_DISTRIBUTECOMMAND' failed
> 
> 
> Looking at the
> run.SRC_DISTRIBUTECOMMAND.18268   16384 (failing)
> run.SRC_DISTRIBUTECOMMAND.9570     34334
> 
> it looks like the file is not complete / malformed:
> 
> package_generate_ipkg_conf() {
> <------>mkdir -p /oe/build/tmp/sysroots/i686-linux/etc/
> <------>echo "src oe file:/oe/build/tmp/deploy/glibc/ipk" >
> /oe/build/tmp/sysroots/i686-linux/etc/opkg.conf
> <------>echo "src oe file:/
> EOF
> 
> run.do_distribute_sources.18268  is empty, while second time
> run.do_distribute_sources.9570   1051 contains  def do_distribute_sources(d):
> 
> This seems indeed a race issue...
> 
> Andrea
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

I can reproduce this one quite easily. Reverting a6a23f73 seems to make it
disappear. Alternately the change below works, but I'm not convinced this is 
the right fix.

Can someone with greater insight explain what causes this race?

-Graham


diff --git a/classes/src_distribute.bbclass b/classes/src_distribute.bbclass
index 795a5cf..98d8d88 100644
--- a/classes/src_distribute.bbclass
+++ b/classes/src_distribute.bbclass
@@ -1,7 +1,7 @@
 SRC_DISTRIBUTE_DLONLY ?= "0"
 SRC_DISTRIBUTECOMMAND[func] = "1"
 
-addtask distribute_sources before do_build after do_fetch
+addtask distribute_sources before do_build after do_unpack
 python do_distribute_sources () {
 	bb.build.exec_func("do_fetch", d)
 

Re: Race issues (lzma-native)

[Andrea Adami]

Just to add this time the race appears building on Ubuntu 10.4 32bit.
Last race-case (gtk+-native) was only on Gentoo and not on Ubuntu...

Both distros are on the same Quad host and share  an EXT3 OpenEmbedded
partition, though.

PARALLEL_MAKE = "-j5"
BB_NUMBER_THREADS = "4"


Perplexed...

Andrea

On Mon, Oct 25, 2010 at 5:59 AM, Graham Gower <graham.gower@gmail.com> wrote:
> On 10/24/2010 07:29 PM, Andrea Adami wrote:
>> Hello,
>>
>> during first build of console image I notice a strange fault:
>> subsequent builds are ok.
>> My scripts builds console-image and opie-image.
>>
>> ERROR: Task 2411
>> (virtual:native:/oe/openembedded/recipes/lzma/lzma_4.65.bb,
>> do_distribute_sources) failed with exit code '1'
>> NOTE: Unpacking ../sources/lzma465.tar.bz2 to
>> tmp/work/i686-linux/lzma-native-4.65-r4.1/
>> NOTE: package lzma-native-4.65-r4.1: task do_distribute_sources: Started
>> NOTE: package lzma-native-4.65-r4.1: task SRC_DISTRIBUTECOMMAND: Failed
>> ERROR: Function 'SRC_DISTRIBUTECOMMAND' failed
>>
>>
>> Looking at the
>> run.SRC_DISTRIBUTECOMMAND.18268   16384 (failing)
>> run.SRC_DISTRIBUTECOMMAND.9570     34334
>>
>> it looks like the file is not complete / malformed:
>>
>> package_generate_ipkg_conf() {
>> <------>mkdir -p /oe/build/tmp/sysroots/i686-linux/etc/
>> <------>echo "src oe file:/oe/build/tmp/deploy/glibc/ipk" >
>> /oe/build/tmp/sysroots/i686-linux/etc/opkg.conf
>> <------>echo "src oe file:/
>> EOF
>>
>> run.do_distribute_sources.18268  is empty, while second time
>> run.do_distribute_sources.9570   1051 contains  def do_distribute_sources(d):
>>
>> This seems indeed a race issue...
>>
>> Andrea
>>
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>
> I can reproduce this one quite easily. Reverting a6a23f73 seems to make it
> disappear. Alternately the change below works, but I'm not convinced this is
> the right fix.
>
> Can someone with greater insight explain what causes this race?
>
> -Graham
>
>
> diff --git a/classes/src_distribute.bbclass b/classes/src_distribute.bbclass
> index 795a5cf..98d8d88 100644
> --- a/classes/src_distribute.bbclass
> +++ b/classes/src_distribute.bbclass
> @@ -1,7 +1,7 @@
>  SRC_DISTRIBUTE_DLONLY ?= "0"
>  SRC_DISTRIBUTECOMMAND[func] = "1"
>
> -addtask distribute_sources before do_build after do_fetch
> +addtask distribute_sources before do_build after do_unpack
>  python do_distribute_sources () {
>        bb.build.exec_func("do_fetch", d)
>
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: Race issues (lzma-native)

[Andrea Adami]

FWIW Issue is still present.

I'll try the workaround you posted.

Andrea



On Tue, Oct 26, 2010 at 9:58 AM, Andrea Adami <andrea.adami@gmail.com> wrote:
> Just to add this time the race appears building on Ubuntu 10.4 32bit.
> Last race-case (gtk+-native) was only on Gentoo and not on Ubuntu...
>
> Both distros are on the same Quad host and share  an EXT3 OpenEmbedded
> partition, though.
>
> PARALLEL_MAKE = "-j5"
> BB_NUMBER_THREADS = "4"
>
>
> Perplexed...
>
> Andrea
>
> On Mon, Oct 25, 2010 at 5:59 AM, Graham Gower <graham.gower@gmail.com> wrote:
>> On 10/24/2010 07:29 PM, Andrea Adami wrote:
>>> Hello,
>>>
>>> during first build of console image I notice a strange fault:
>>> subsequent builds are ok.
>>> My scripts builds console-image and opie-image.
>>>
>>> ERROR: Task 2411
>>> (virtual:native:/oe/openembedded/recipes/lzma/lzma_4.65.bb,
>>> do_distribute_sources) failed with exit code '1'
>>> NOTE: Unpacking ../sources/lzma465.tar.bz2 to
>>> tmp/work/i686-linux/lzma-native-4.65-r4.1/
>>> NOTE: package lzma-native-4.65-r4.1: task do_distribute_sources: Started
>>> NOTE: package lzma-native-4.65-r4.1: task SRC_DISTRIBUTECOMMAND: Failed
>>> ERROR: Function 'SRC_DISTRIBUTECOMMAND' failed
>>>
>>>
>>> Looking at the
>>> run.SRC_DISTRIBUTECOMMAND.18268   16384 (failing)
>>> run.SRC_DISTRIBUTECOMMAND.9570     34334
>>>
>>> it looks like the file is not complete / malformed:
>>>
>>> package_generate_ipkg_conf() {
>>> <------>mkdir -p /oe/build/tmp/sysroots/i686-linux/etc/
>>> <------>echo "src oe file:/oe/build/tmp/deploy/glibc/ipk" >
>>> /oe/build/tmp/sysroots/i686-linux/etc/opkg.conf
>>> <------>echo "src oe file:/
>>> EOF
>>>
>>> run.do_distribute_sources.18268  is empty, while second time
>>> run.do_distribute_sources.9570   1051 contains  def do_distribute_sources(d):
>>>
>>> This seems indeed a race issue...
>>>
>>> Andrea
>>>
>>> _______________________________________________
>>> Openembedded-devel mailing list
>>> Openembedded-devel@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>>
>> I can reproduce this one quite easily. Reverting a6a23f73 seems to make it
>> disappear. Alternately the change below works, but I'm not convinced this is
>> the right fix.
>>
>> Can someone with greater insight explain what causes this race?
>>
>> -Graham
>>
>>
>> diff --git a/classes/src_distribute.bbclass b/classes/src_distribute.bbclass
>> index 795a5cf..98d8d88 100644
>> --- a/classes/src_distribute.bbclass
>> +++ b/classes/src_distribute.bbclass
>> @@ -1,7 +1,7 @@
>>  SRC_DISTRIBUTE_DLONLY ?= "0"
>>  SRC_DISTRIBUTECOMMAND[func] = "1"
>>
>> -addtask distribute_sources before do_build after do_fetch
>> +addtask distribute_sources before do_build after do_unpack
>>  python do_distribute_sources () {
>>        bb.build.exec_func("do_fetch", d)
>>
>>
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>>
>

Re: Race issues (lzma-native)

[Andrea Adami]

Strange...

Built today with Metadata rev fda42d9 and no issues.
Tried on Ubuntu and Gentoo 32bit

Regards

Andrea



On Thu, Nov 4, 2010 at 1:27 PM, Andrea Adami <andrea.adami@gmail.com> wrote:
> FWIW Issue is still present.
>
> I'll try the workaround you posted.
>
> Andrea
>
>
>
> On Tue, Oct 26, 2010 at 9:58 AM, Andrea Adami <andrea.adami@gmail.com> wrote:
>> Just to add this time the race appears building on Ubuntu 10.4 32bit.
>> Last race-case (gtk+-native) was only on Gentoo and not on Ubuntu...
>>
>> Both distros are on the same Quad host and share  an EXT3 OpenEmbedded
>> partition, though.
>>
>> PARALLEL_MAKE = "-j5"
>> BB_NUMBER_THREADS = "4"
>>
>>
>> Perplexed...
>>
>> Andrea
>>
>> On Mon, Oct 25, 2010 at 5:59 AM, Graham Gower <graham.gower@gmail.com> wrote:
>>> On 10/24/2010 07:29 PM, Andrea Adami wrote:
>>>> Hello,
>>>>
>>>> during first build of console image I notice a strange fault:
>>>> subsequent builds are ok.
>>>> My scripts builds console-image and opie-image.
>>>>
>>>> ERROR: Task 2411
>>>> (virtual:native:/oe/openembedded/recipes/lzma/lzma_4.65.bb,
>>>> do_distribute_sources) failed with exit code '1'
>>>> NOTE: Unpacking ../sources/lzma465.tar.bz2 to
>>>> tmp/work/i686-linux/lzma-native-4.65-r4.1/
>>>> NOTE: package lzma-native-4.65-r4.1: task do_distribute_sources: Started
>>>> NOTE: package lzma-native-4.65-r4.1: task SRC_DISTRIBUTECOMMAND: Failed
>>>> ERROR: Function 'SRC_DISTRIBUTECOMMAND' failed
>>>>
>>>>
>>>> Looking at the
>>>> run.SRC_DISTRIBUTECOMMAND.18268   16384 (failing)
>>>> run.SRC_DISTRIBUTECOMMAND.9570     34334
>>>>
>>>> it looks like the file is not complete / malformed:
>>>>
>>>> package_generate_ipkg_conf() {
>>>> <------>mkdir -p /oe/build/tmp/sysroots/i686-linux/etc/
>>>> <------>echo "src oe file:/oe/build/tmp/deploy/glibc/ipk" >
>>>> /oe/build/tmp/sysroots/i686-linux/etc/opkg.conf
>>>> <------>echo "src oe file:/
>>>> EOF
>>>>
>>>> run.do_distribute_sources.18268  is empty, while second time
>>>> run.do_distribute_sources.9570   1051 contains  def do_distribute_sources(d):
>>>>
>>>> This seems indeed a race issue...
>>>>
>>>> Andrea
>>>>
>>>> _______________________________________________
>>>> Openembedded-devel mailing list
>>>> Openembedded-devel@lists.openembedded.org
>>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>>>
>>> I can reproduce this one quite easily. Reverting a6a23f73 seems to make it
>>> disappear. Alternately the change below works, but I'm not convinced this is
>>> the right fix.
>>>
>>> Can someone with greater insight explain what causes this race?
>>>
>>> -Graham
>>>
>>>
>>> diff --git a/classes/src_distribute.bbclass b/classes/src_distribute.bbclass
>>> index 795a5cf..98d8d88 100644
>>> --- a/classes/src_distribute.bbclass
>>> +++ b/classes/src_distribute.bbclass
>>> @@ -1,7 +1,7 @@
>>>  SRC_DISTRIBUTE_DLONLY ?= "0"
>>>  SRC_DISTRIBUTECOMMAND[func] = "1"
>>>
>>> -addtask distribute_sources before do_build after do_fetch
>>> +addtask distribute_sources before do_build after do_unpack
>>>  python do_distribute_sources () {
>>>        bb.build.exec_func("do_fetch", d)
>>>
>>>
>>> _______________________________________________
>>> Openembedded-devel mailing list
>>> Openembedded-devel@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>>>
>>
>

Re: Race issues (lzma-native)

[Andrea Adami]

I though tthe issue had disappeared (in the meanwhile I remove an
obsolete dependency on zlib)
but yesterday again...

run.SRC_DISTRIBUTECOMMAND was malformed, broken at line 122 (export
docdir="/oe/bu ).

On second build the size is much bigger (1102 lines).

Retrying..
Andrea

Re: Race issues (lzma-native)

[Graham Gower]

On 22 November 2010 10:42, Andrea Adami <andrea.adami@gmail.com> wrote:
> I though tthe issue had disappeared (in the meanwhile I remove an
> obsolete dependency on zlib)
> but yesterday again...
>
> run.SRC_DISTRIBUTECOMMAND was malformed, broken at line 122 (export
> docdir="/oe/bu ).
>
> On second build the size is much bigger (1102 lines).
>
> Retrying..
> Andrea
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

With pstaging disabled, do_fetch is called twice. The second do_fetch
races with do_unpack. Presumably we see a problem in lzma-native
because it does oe.process.run() in do_unpack_append. I can't see
exactly *what* is racing though.

I previously spoke to Kergoth about it on IRC and he suggested making
bb.build.exec_func("do_fetch", d) conditional on whether the do_fetch
stamp file exists... I've not gotten around to doing that (bad me).
Want to try it?

-Graham

Re: Race issues (lzma-native)

[Frans Meulenbroeks]

2010/11/22 Graham Gower <graham.gower@gmail.com>:
> On 22 November 2010 10:42, Andrea Adami <andrea.adami@gmail.com> wrote:
>> I though tthe issue had disappeared (in the meanwhile I remove an
>> obsolete dependency on zlib)
>> but yesterday again...
>>
>> run.SRC_DISTRIBUTECOMMAND was malformed, broken at line 122 (export
>> docdir="/oe/bu ).
>>
>> On second build the size is much bigger (1102 lines).
>>
>> Retrying..
>> Andrea
>>
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>>
>
> With pstaging disabled, do_fetch is called twice. The second do_fetch
> races with do_unpack. Presumably we see a problem in lzma-native
> because it does oe.process.run() in do_unpack_append. I can't see
> exactly *what* is racing though.
>
> I previously spoke to Kergoth about it on IRC and he suggested making
> bb.build.exec_func("do_fetch", d) conditional on whether the do_fetch
> stamp file exists... I've not gotten around to doing that (bad me).
> Want to try it?

Hm.
I don't know the exact details, but this looks like a bandage, not
like a fix for the root problem.
Why is do_fetch called twice in the first place. I would expect this
not to happen.

Frans

Re: Race issues (lzma-native)

[Graham Gower]

On 22 November 2010 17:24, Frans Meulenbroeks
<fransmeulenbroeks@gmail.com> wrote:
> 2010/11/22 Graham Gower <graham.gower@gmail.com>:
>> On 22 November 2010 10:42, Andrea Adami <andrea.adami@gmail.com> wrote:
>>> I though tthe issue had disappeared (in the meanwhile I remove an
>>> obsolete dependency on zlib)
>>> but yesterday again...
>>>
>>> run.SRC_DISTRIBUTECOMMAND was malformed, broken at line 122 (export
>>> docdir="/oe/bu ).
>>>
>>> On second build the size is much bigger (1102 lines).
>>>
>>> Retrying..
>>> Andrea
>>>
>>> _______________________________________________
>>> Openembedded-devel mailing list
>>> Openembedded-devel@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>>>
>>
>> With pstaging disabled, do_fetch is called twice. The second do_fetch
>> races with do_unpack. Presumably we see a problem in lzma-native
>> because it does oe.process.run() in do_unpack_append. I can't see
>> exactly *what* is racing though.
>>
>> I previously spoke to Kergoth about it on IRC and he suggested making
>> bb.build.exec_func("do_fetch", d) conditional on whether the do_fetch
>> stamp file exists... I've not gotten around to doing that (bad me).
>> Want to try it?
>
> Hm.
> I don't know the exact details, but this looks like a bandage, not
> like a fix for the root problem.

The conditional call of do_fetch is probably what we want, but I'd
prefer to see this race fixed before we do that to ensure we aren't
masking any other related races.

> Why is do_fetch called twice in the first place. I would expect this
> not to happen.

See src_distribute.bbclass and refer to do_distribute_sources().
do_fetch is only called twice if pstaging is disabled (which is not
the default).

-Graham

Re: Race issues (lzma-native)

[Frans Meulenbroeks]

2010/11/22 Graham Gower <graham.gower@gmail.com>:

>
> The conditional call of do_fetch is probably what we want, but I'd
> prefer to see this race fixed before we do that to ensure we aren't
> masking any other related races.

There should definitely be a dependency on the stamp and if the stamp
is newer the fetch should not be called (or generated)
>
>> Why is do_fetch called twice in the first place. I would expect this
>> not to happen.
>
> See src_distribute.bbclass and refer to do_distribute_sources().
> do_fetch is only called twice if pstaging is disabled (which is not
> the default).

Without too much knowledge on the internals, I'd say the 2nd fetch
should be conditional on pstaging then.

Did a quick peek at that bbclass file:

addtask distribute_sources before do_build after do_fetch
python do_distribute_sources () {
        bb.build.exec_func("do_fetch", d)

This greatly puzzles me. distribute_sources comes after do_fetch and
it starts doing a do_fetch ???
Isn't it possible to just eliminate this 2nd fetch ???

Puzzled. Frans