* fetch race problem
@ 2015-03-24 11:00 Steffen Sledz
2015-03-24 11:12 ` [OE-core] " Richard Purdie
0 siblings, 1 reply; 7+ messages in thread
From: Steffen Sledz @ 2015-03-24 11:00 UTC (permalink / raw)
To: openembedded-devel, openembedded-core
We hit the following problem in oe-classic. And we're not sure if this problem may occur in current oe too, or if it is fixed there.
They are two or more recipes (may be in different layers) for packages with sources from the same (git) repository. If the fetching from the repo needs some time we see "file changed as we read it" errors in all likelihood.
------------------> snip <-------------------
ERROR: Error evaluating '${PF}:${P}:${PN}:${BP}:${BPN}:files:.'
Traceback (most recent call last):
File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 116, in expandWithRefs
s = __expand_var_regexp__.sub(varparse.var_sub, s)
File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 60, in var_sub
var = self.d.getVar(key, 1)
File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 260, in getVar
return self.expand(value, var)
File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 132, in expand
return self.expandWithRefs(s, varname).value
File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 116, in expandWithRefs
s = __expand_var_regexp__.sub(varparse.var_sub, s)
FetchError: Fetch command export HOME="/var/lib/jenkins"; export PATH="/CACHE/jenkins-workspace/foo/OE//bin:/CACHE/jenkins-workspace/foo/openembedded/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/armv5te/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/sbin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/sbin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux//bin:/CACHE/jenkins-workspace/foo/OE/bin-HYP-6339:/CACHE/jenkins-workspace/foo/bitbake/bin:/sbin:/usr/sbin:/bin:/usr/bin"; tar -czf /var/lib/jenkins/sources/git_bitbucket.org.dresearchfe.drconfig.git.tar.gz ./.git/* failed with signal 1, output:
/bin/tar: ./.git/objects/pack/tmp_pack_BwThcc: File removed before we read it
/bin/tar: ./.git/objects/pack/tmp_pack_Ans7xK: File removed before we read it
/bin/tar: ./.git/objects/pack: file changed as we read it
------------------> snap <-------------------
Is this a problem in oe-core too (or oe-classic only)?
Any ideas how to avoid such errors?
Steffen
--
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: fetch race problem
2015-03-24 11:00 fetch race problem Steffen Sledz
@ 2015-03-24 11:12 ` Richard Purdie
0 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2015-03-24 11:12 UTC (permalink / raw)
To: Steffen Sledz; +Cc: openembedded-devel, openembedded-core
On Tue, 2015-03-24 at 12:00 +0100, Steffen Sledz wrote:
> We hit the following problem in oe-classic. And we're not sure if this problem may occur in current oe too, or if it is fixed there.
>
> They are two or more recipes (may be in different layers) for packages with sources from the same (git) repository. If the fetching from the repo needs some time we see "file changed as we read it" errors in all likelihood.
>
> ------------------> snip <-------------------
> ERROR: Error evaluating '${PF}:${P}:${PN}:${BP}:${BPN}:files:.'
> Traceback (most recent call last):
> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 116, in expandWithRefs
> s = __expand_var_regexp__.sub(varparse.var_sub, s)
> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 60, in var_sub
> var = self.d.getVar(key, 1)
> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 260, in getVar
> return self.expand(value, var)
> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 132, in expand
> return self.expandWithRefs(s, varname).value
> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 116, in expandWithRefs
> s = __expand_var_regexp__.sub(varparse.var_sub, s)
> FetchError: Fetch command export HOME="/var/lib/jenkins"; export PATH="/CACHE/jenkins-workspace/foo/OE//bin:/CACHE/jenkins-workspace/foo/openembedded/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/armv5te/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/sbin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/sbin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux//bin:/CACHE/jenkins-workspace/foo/OE/bin-HYP-6339:/CACHE/jenkins-workspace/foo/bitbake/bin:/sbin:/usr/sbin:/bin:/usr/bin"; tar -czf /var/lib/jenkins/sources/git_bitbucket.org.dresearchfe.drconfig.git.tar.gz ./.git/* failed with signal 1, output:
> /bin/tar: ./.git/objects/pack/tmp_pack_BwThcc: File removed before we read it
> /bin/tar: ./.git/objects/pack/tmp_pack_Ans7xK: File removed before we read it
> /bin/tar: ./.git/objects/pack: file changed as we read it
> ------------------> snap <-------------------
>
> Is this a problem in oe-core too (or oe-classic only)?
>
> Any ideas how to avoid such errors?
In theory bitbake has locks on files in DL_DIR, even for git
repositories so it should only touch the repo with one fetch process at
a time even if multiple recipes reference the same thing.
There were some big fetcher reworks in more recent versions of bitbake.
For OE-Classic, I'm guessing you're using an older version. You could
try transplanting the modern fetcher core onto your older bitbake I
guess.
Cheers,
Richard
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [OE-core] fetch race problem
@ 2015-03-24 11:12 ` Richard Purdie
0 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2015-03-24 11:12 UTC (permalink / raw)
To: Steffen Sledz; +Cc: openembedded-devel, openembedded-core
On Tue, 2015-03-24 at 12:00 +0100, Steffen Sledz wrote:
> We hit the following problem in oe-classic. And we're not sure if this problem may occur in current oe too, or if it is fixed there.
>
> They are two or more recipes (may be in different layers) for packages with sources from the same (git) repository. If the fetching from the repo needs some time we see "file changed as we read it" errors in all likelihood.
>
> ------------------> snip <-------------------
> ERROR: Error evaluating '${PF}:${P}:${PN}:${BP}:${BPN}:files:.'
> Traceback (most recent call last):
> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 116, in expandWithRefs
> s = __expand_var_regexp__.sub(varparse.var_sub, s)
> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 60, in var_sub
> var = self.d.getVar(key, 1)
> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 260, in getVar
> return self.expand(value, var)
> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 132, in expand
> return self.expandWithRefs(s, varname).value
> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 116, in expandWithRefs
> s = __expand_var_regexp__.sub(varparse.var_sub, s)
> FetchError: Fetch command export HOME="/var/lib/jenkins"; export PATH="/CACHE/jenkins-workspace/foo/OE//bin:/CACHE/jenkins-workspace/foo/openembedded/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/armv5te/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/sbin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/sbin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux//bin:/CACHE/jenkins-workspace/foo/OE/bin-HYP-6339:/CACHE/jenkins-workspace/foo/bitbake/bin:/sbin:/usr/sbin:/bin:/usr/bin"; tar -czf /var/lib/jenkins/sources/git_bitbucket.org.dresearchfe.drconfig.git.tar.gz ./.git/* failed with signal 1, output:
> /bin/tar: ./.git/objects/pack/tmp_pack_BwThcc: File removed before we read it
> /bin/tar: ./.git/objects/pack/tmp_pack_Ans7xK: File removed before we read it
> /bin/tar: ./.git/objects/pack: file changed as we read it
> ------------------> snap <-------------------
>
> Is this a problem in oe-core too (or oe-classic only)?
>
> Any ideas how to avoid such errors?
In theory bitbake has locks on files in DL_DIR, even for git
repositories so it should only touch the repo with one fetch process at
a time even if multiple recipes reference the same thing.
There were some big fetcher reworks in more recent versions of bitbake.
For OE-Classic, I'm guessing you're using an older version. You could
try transplanting the modern fetcher core onto your older bitbake I
guess.
Cheers,
Richard
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: fetch race problem
2015-03-24 11:12 ` [OE-core] " Richard Purdie
@ 2015-04-07 8:35 ` Steffen Sledz
-1 siblings, 0 replies; 7+ messages in thread
From: Steffen Sledz @ 2015-04-07 8:35 UTC (permalink / raw)
To: Richard Purdie; +Cc: openembedded-devel, openembedded-core
Am 24.03.2015 um 12:12 schrieb Richard Purdie:
> On Tue, 2015-03-24 at 12:00 +0100, Steffen Sledz wrote:
>> We hit the following problem in oe-classic. And we're not sure if this problem may occur in current oe too, or if it is fixed there.
>>
>> They are two or more recipes (may be in different layers) for packages with sources from the same (git) repository. If the fetching from the repo needs some time we see "file changed as we read it" errors in all likelihood.
>>
>> ------------------> snip <-------------------
>> ERROR: Error evaluating '${PF}:${P}:${PN}:${BP}:${BPN}:files:.'
>> Traceback (most recent call last):
>> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 116, in expandWithRefs
>> s = __expand_var_regexp__.sub(varparse.var_sub, s)
>> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 60, in var_sub
>> var = self.d.getVar(key, 1)
>> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 260, in getVar
>> return self.expand(value, var)
>> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 132, in expand
>> return self.expandWithRefs(s, varname).value
>> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 116, in expandWithRefs
>> s = __expand_var_regexp__.sub(varparse.var_sub, s)
>> FetchError: Fetch command export HOME="/var/lib/jenkins"; export PATH="/CACHE/jenkins-workspace/foo/OE//bin:/CACHE/jenkins-workspace/foo/openembedded/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/armv5te/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/sbin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/sbin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux//bin:/CACHE/jenkins-workspace/foo/OE/bin-HYP-6339:/CACHE/jenkins-workspace/foo/bitbake/bin:/sbin:/usr/sbin:/bin:/usr/bin"; tar -czf /var/lib/jenkins/sources/git_bitbucket.org.dresearchfe.drconfig.git.tar.gz ./.git/* failed with signal 1, output:
>> /bin/tar: ./.git/objects/pack/tmp_pack_BwThcc: File removed before we read it
>> /bin/tar: ./.git/objects/pack/tmp_pack_Ans7xK: File removed before we read it
>> /bin/tar: ./.git/objects/pack: file changed as we read it
>> ------------------> snap <-------------------
>>
>> Is this a problem in oe-core too (or oe-classic only)?
>>
>> Any ideas how to avoid such errors?
>
> In theory bitbake has locks on files in DL_DIR, even for git
> repositories so it should only touch the repo with one fetch process at
> a time even if multiple recipes reference the same thing.
>
> There were some big fetcher reworks in more recent versions of bitbake.
> For OE-Classic, I'm guessing you're using an older version. You could
> try transplanting the modern fetcher core onto your older bitbake I
> guess.
I do not feel competent enough to try this.
Is it possible (in oe-classic) to mark two or more recipes (or single stages) to run exclusive?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [OE-core] fetch race problem
@ 2015-04-07 8:35 ` Steffen Sledz
0 siblings, 0 replies; 7+ messages in thread
From: Steffen Sledz @ 2015-04-07 8:35 UTC (permalink / raw)
To: Richard Purdie; +Cc: openembedded-devel, openembedded-core
Am 24.03.2015 um 12:12 schrieb Richard Purdie:
> On Tue, 2015-03-24 at 12:00 +0100, Steffen Sledz wrote:
>> We hit the following problem in oe-classic. And we're not sure if this problem may occur in current oe too, or if it is fixed there.
>>
>> They are two or more recipes (may be in different layers) for packages with sources from the same (git) repository. If the fetching from the repo needs some time we see "file changed as we read it" errors in all likelihood.
>>
>> ------------------> snip <-------------------
>> ERROR: Error evaluating '${PF}:${P}:${PN}:${BP}:${BPN}:files:.'
>> Traceback (most recent call last):
>> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 116, in expandWithRefs
>> s = __expand_var_regexp__.sub(varparse.var_sub, s)
>> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 60, in var_sub
>> var = self.d.getVar(key, 1)
>> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 260, in getVar
>> return self.expand(value, var)
>> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 132, in expand
>> return self.expandWithRefs(s, varname).value
>> File "/CACHE/jenkins-workspace/foo/bitbake/lib/bb/data_smart.py", line 116, in expandWithRefs
>> s = __expand_var_regexp__.sub(varparse.var_sub, s)
>> FetchError: Fetch command export HOME="/var/lib/jenkins"; export PATH="/CACHE/jenkins-workspace/foo/OE//bin:/CACHE/jenkins-workspace/foo/openembedded/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/armv5te/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/sbin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/usr/bin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux/sbin:/CACHE/jenkins-workspace/foo/OE/tmp.7/sysroots/x86_64-linux//bin:/CACHE/jenkins-workspace/foo/OE/bin-HYP-6339:/CACHE/jenkins-workspace/foo/bitbake/bin:/sbin:/usr/sbin:/bin:/usr/bin"; tar -czf /var/lib/jenkins/sources/git_bitbucket.org.dresearchfe.drconfig.git.tar.gz ./.git/* failed with signal 1, output:
>> /bin/tar: ./.git/objects/pack/tmp_pack_BwThcc: File removed before we read it
>> /bin/tar: ./.git/objects/pack/tmp_pack_Ans7xK: File removed before we read it
>> /bin/tar: ./.git/objects/pack: file changed as we read it
>> ------------------> snap <-------------------
>>
>> Is this a problem in oe-core too (or oe-classic only)?
>>
>> Any ideas how to avoid such errors?
>
> In theory bitbake has locks on files in DL_DIR, even for git
> repositories so it should only touch the repo with one fetch process at
> a time even if multiple recipes reference the same thing.
>
> There were some big fetcher reworks in more recent versions of bitbake.
> For OE-Classic, I'm guessing you're using an older version. You could
> try transplanting the modern fetcher core onto your older bitbake I
> guess.
I do not feel competent enough to try this.
Is it possible (in oe-classic) to mark two or more recipes (or single stages) to run exclusive?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: fetch race problem
2015-04-07 8:35 ` [OE-core] " Steffen Sledz
@ 2015-04-07 12:29 ` Richard Purdie
-1 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2015-04-07 12:29 UTC (permalink / raw)
To: Steffen Sledz; +Cc: openembedded-devel, openembedded-core
On Tue, 2015-04-07 at 10:35 +0200, Steffen Sledz wrote:
> Am 24.03.2015 um 12:12 schrieb Richard Purdie:
> >>> Is this a problem in oe-core too (or oe-classic only)?
> >>
> >> Any ideas how to avoid such errors?
> >
> > In theory bitbake has locks on files in DL_DIR, even for git
> > repositories so it should only touch the repo with one fetch process at
> > a time even if multiple recipes reference the same thing.
> >
> > There were some big fetcher reworks in more recent versions of bitbake.
> > For OE-Classic, I'm guessing you're using an older version. You could
> > try transplanting the modern fetcher core onto your older bitbake I
> > guess.
>
> I do not feel competent enough to try this.
>
> Is it possible (in oe-classic) to mark two or more recipes (or single stages) to run exclusive?
do_fetch[depends] += "xxx:do_fetch"
in one recipe would make it run after the fetch of the other recipe
(xxx) ...
I will say that oe-classic is dead at this point, its not something I'd
recommend people to be working with.
Cheers,
Richard
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [OE-core] fetch race problem
@ 2015-04-07 12:29 ` Richard Purdie
0 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2015-04-07 12:29 UTC (permalink / raw)
To: Steffen Sledz; +Cc: openembedded-devel, openembedded-core
On Tue, 2015-04-07 at 10:35 +0200, Steffen Sledz wrote:
> Am 24.03.2015 um 12:12 schrieb Richard Purdie:
> >>> Is this a problem in oe-core too (or oe-classic only)?
> >>
> >> Any ideas how to avoid such errors?
> >
> > In theory bitbake has locks on files in DL_DIR, even for git
> > repositories so it should only touch the repo with one fetch process at
> > a time even if multiple recipes reference the same thing.
> >
> > There were some big fetcher reworks in more recent versions of bitbake.
> > For OE-Classic, I'm guessing you're using an older version. You could
> > try transplanting the modern fetcher core onto your older bitbake I
> > guess.
>
> I do not feel competent enough to try this.
>
> Is it possible (in oe-classic) to mark two or more recipes (or single stages) to run exclusive?
do_fetch[depends] += "xxx:do_fetch"
in one recipe would make it run after the fetch of the other recipe
(xxx) ...
I will say that oe-classic is dead at this point, its not something I'd
recommend people to be working with.
Cheers,
Richard
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2015-04-07 12:30 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-03-24 11:00 fetch race problem Steffen Sledz
2015-03-24 11:12 ` Richard Purdie
2015-03-24 11:12 ` [OE-core] " Richard Purdie
2015-04-07 8:35 ` Steffen Sledz
2015-04-07 8:35 ` [OE-core] " Steffen Sledz
2015-04-07 12:29 ` Richard Purdie
2015-04-07 12:29 ` [OE-core] " Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.