[PATCH 0/1] state.bbclass: check before open the manifest

[PATCH 0/1] state.bbclass: check before open the manifest

[Robert Yang]

The following changes since commit 5745e45b18e5099e94b4d5a73bc97dc6d4cdc91f:

  buildtools-tarball: Add python-pkgutil (2013-08-29 00:23:20 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib robert/sstate
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/sstate

Robert Yang (1):
  sstate.bbclass: check before open the manifest

 meta/classes/sstate.bbclass |    3 +++
 1 file changed, 3 insertions(+)

-- 
1.7.10.4

[PATCH 1/1] sstate.bbclass: check before open the manifest

[Robert Yang]

The python stack trace would be printed if we:

$ bitbake make (the make-3.82 will be built)
// Edit make.inc
$ bitbake make-3.81
[snip]
 *** 0004:    mfile = open(manifest)
     0005:    entries = mfile.readlines()
     0006:    mfile.close()
     0007:
     0008:    for entry in entries:
Exception: IOError: [Errno 2] No such file or directory: xxx
[snip]

This because the make-3.81 and make-3.82 are being built at the same
time, the manifest may have been removed by make-3.82, but make-3.81
still opens it, so the error happens.

Check before open the manifest would fix the problem.

[YOCTO #5067]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/sstate.bbclass |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index c86f393..6e71086 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -302,6 +302,9 @@ def sstate_clean_cachefiles(d):
 def sstate_clean_manifest(manifest, d):
     import oe.path
 
+    if not os.path.exists(manifest):
+        return True
+
     mfile = open(manifest)
     entries = mfile.readlines()
     mfile.close()
-- 
1.7.10.4

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

[Richard Purdie]

On Thu, 2013-08-29 at 09:13 -0400, Robert Yang wrote:
> The python stack trace would be printed if we:
> 
> $ bitbake make (the make-3.82 will be built)
> // Edit make.inc
> $ bitbake make-3.81

This is at best hiding the real problem. Why is bitbake building
make-3.82 when you asked for make-3.81?

I tried this here and also see the warning:

ERROR: Multiple .bb files are due to be built which each provide make
(/media/build1/poky/meta/recipes-devtools/make/make_3.81.bb /media/build1/poky/meta/recipes-devtools/make/make_3.82.bb).
 This usually means one provides something the other doesn't and should.

so bitbake is basically telling you there is a problem already. So there
are two issues:

a) Why is make 3.82 being built?
b) If multiple identical PNs are being built we probably should hard
error out since its not supported in the slightest. The sstate race you
mention is the least of the problems :(.

Cheers,

Richard

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

[Robert Yang]

On 08/30/2013 01:00 AM, Richard Purdie wrote:
> On Thu, 2013-08-29 at 09:13 -0400, Robert Yang wrote:
>> The python stack trace would be printed if we:
>>
>> $ bitbake make (the make-3.82 will be built)
>> // Edit make.inc
>> $ bitbake make-3.81
>
> This is at best hiding the real problem. Why is bitbake building
> make-3.82 when you asked for make-3.81?
>
> I tried this here and also see the warning:
>
> ERROR: Multiple .bb files are due to be built which each provide make
> (/media/build1/poky/meta/recipes-devtools/make/make_3.81.bb /media/build1/poky/meta/recipes-devtools/make/make_3.82.bb).
>   This usually means one provides something the other doesn't and should.
>
> so bitbake is basically telling you there is a problem already. So there
> are two issues:
>

What I thought was that let the user know the normal error, but we can do
more to fix it as you pointed out.

> a) Why is make 3.82 being built?

Ah, yes, that's problem, I will do more investigation.

> b) If multiple identical PNs are being built we probably should hard
> error out since its not supported in the slightest. The sstate race you
> mention is the least of the problems :(.
>

OK, I will try to fix such a case:

bitbake make-3.81 make-3.82

// Robert

> Cheers,
>
> Richard
>
>
>

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

[Richard Purdie]

On Fri, 2013-08-30 at 09:24 +0800, Robert Yang wrote:
> 
> On 08/30/2013 01:00 AM, Richard Purdie wrote:
> > On Thu, 2013-08-29 at 09:13 -0400, Robert Yang wrote:
> >> The python stack trace would be printed if we:
> >>
> >> $ bitbake make (the make-3.82 will be built)
> >> // Edit make.inc
> >> $ bitbake make-3.81
> >
> > This is at best hiding the real problem. Why is bitbake building
> > make-3.82 when you asked for make-3.81?
> >
> > I tried this here and also see the warning:
> >
> > ERROR: Multiple .bb files are due to be built which each provide make
> > (/media/build1/poky/meta/recipes-devtools/make/make_3.81.bb /media/build1/poky/meta/recipes-devtools/make/make_3.82.bb).
> >   This usually means one provides something the other doesn't and should.
> >
> > so bitbake is basically telling you there is a problem already. So there
> > are two issues:
> >
> 
> What I thought was that let the user know the normal error, but we can do
> more to fix it as you pointed out.
> 
> > a) Why is make 3.82 being built?
> 
> Ah, yes, that's problem, I will do more investigation.
> 
> > b) If multiple identical PNs are being built we probably should hard
> > error out since its not supported in the slightest. The sstate race you
> > mention is the least of the problems :(.
> >
> 
> OK, I will try to fix such a case:
> 
> bitbake make-3.81 make-3.82

I think this should just give an error.

Cheers,

Richard

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

[Robert Yang]

Hi RP,

Thanks, I've made a draft patch for fixing why make-3.82 is being built,
the problem is that when we run "bitbake make-3.81":
- For 'make' itself, it will build make_3.81.bb
- But for make-dev/dbg, it will build make_3.82.bb since there is no
PREFERRED_VERSION and it will use the highest version.

This draft patch sets the PREFERRED_VERSION for the pkg and will fix the
problem, I will send it to bitbake-devel later.

diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py
index 58fe199..fc8ee6d 100644
--- a/bitbake/lib/bb/taskdata.py
+++ b/bitbake/lib/bb/taskdata.py
@@ -429,6 +429,22 @@ class TaskData:
              return

          all_p = dataCache.providers[item]
+        logger.warn("pppreferred: %s" % dataCache.preferred)
+
+        # Check whether item has the one of following formats and set the
+        # PREFERRED_VERSION if it does:
+        # - <pn>-<pv>
+        # - <pn>-<pv>-<pr> (when no PE)
+        # - <pn>_<pe>-<pv>-<pr> (when PE)
+        for fn in all_p:
+            pn = dataCache.pkg_fn[fn]
+            pe = dataCache.pkg_pepvpr[fn][0]
+            pv = dataCache.pkg_pepvpr[fn][1]
+            pr = dataCache.pkg_pepvpr[fn][2]
+            if not pe and (item == "%s-%s" % (pn, pv) or item == "%s-%s-%s" % 
(pn, pv, pr)):
+                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)
+            elif pe and item == "%s_%s-%s-%s" % (pn, pe, pv, pr):
+                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)

          eligible, foundUnique = bb.providers.filterProviders(all_p, item, 
cfgData, dataCache)
          eligible = [p for p in eligible if not self.getfn_id(p) in 
self.failed_fnids]

// Robert

On 08/30/2013 11:29 PM, Richard Purdie wrote:
> On Fri, 2013-08-30 at 09:24 +0800, Robert Yang wrote:
>>
>> On 08/30/2013 01:00 AM, Richard Purdie wrote:
>>> On Thu, 2013-08-29 at 09:13 -0400, Robert Yang wrote:
>>>> The python stack trace would be printed if we:
>>>>
>>>> $ bitbake make (the make-3.82 will be built)
>>>> // Edit make.inc
>>>> $ bitbake make-3.81
>>>
>>> This is at best hiding the real problem. Why is bitbake building
>>> make-3.82 when you asked for make-3.81?
>>>
>>> I tried this here and also see the warning:
>>>
>>> ERROR: Multiple .bb files are due to be built which each provide make
>>> (/media/build1/poky/meta/recipes-devtools/make/make_3.81.bb /media/build1/poky/meta/recipes-devtools/make/make_3.82.bb).
>>>    This usually means one provides something the other doesn't and should.
>>>
>>> so bitbake is basically telling you there is a problem already. So there
>>> are two issues:
>>>
>>
>> What I thought was that let the user know the normal error, but we can do
>> more to fix it as you pointed out.
>>
>>> a) Why is make 3.82 being built?
>>
>> Ah, yes, that's problem, I will do more investigation.
>>
>>> b) If multiple identical PNs are being built we probably should hard
>>> error out since its not supported in the slightest. The sstate race you
>>> mention is the least of the problems :(.
>>>
>>
>> OK, I will try to fix such a case:
>>
>> bitbake make-3.81 make-3.82
>
> I think this should just give an error.
>
> Cheers,
>
> Richard
>
>
>

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

[Otavio Salvador]

On Sat, Aug 31, 2013 at 7:51 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
>
> Hi RP,
>
> Thanks, I've made a draft patch for fixing why make-3.82 is being built,
> the problem is that when we run "bitbake make-3.81":
> - For 'make' itself, it will build make_3.81.bb
> - But for make-dev/dbg, it will build make_3.82.bb since there is no
> PREFERRED_VERSION and it will use the highest version.
>
> This draft patch sets the PREFERRED_VERSION for the pkg and will fix the
> problem, I will send it to bitbake-devel later.

Nice discovery :-) This does seem to address a core issue.

> diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py
> index 58fe199..fc8ee6d 100644
> --- a/bitbake/lib/bb/taskdata.py
> +++ b/bitbake/lib/bb/taskdata.py
> @@ -429,6 +429,22 @@ class TaskData:
>              return
>
>          all_p = dataCache.providers[item]
> +        logger.warn("pppreferred: %s" % dataCache.preferred)
> +
> +        # Check whether item has the one of following formats and set the
> +        # PREFERRED_VERSION if it does:
> +        # - <pn>-<pv>
> +        # - <pn>-<pv>-<pr> (when no PE)
> +        # - <pn>_<pe>-<pv>-<pr> (when PE)
> +        for fn in all_p:
> +            pn = dataCache.pkg_fn[fn]
> +            pe = dataCache.pkg_pepvpr[fn][0]
> +            pv = dataCache.pkg_pepvpr[fn][1]
> +            pr = dataCache.pkg_pepvpr[fn][2]
> +            if not pe and (item == "%s-%s" % (pn, pv) or item == "%s-%s-%s"
> % (pn, pv, pr)):
> +                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)
> +            elif pe and item == "%s_%s-%s-%s" % (pn, pe, pv, pr):
> +                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)

I understand the problem here but maybe this should be checked in
bitbake and avoid iterate in all packages?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 1/1] sstate.bbclass: check before open the manifest

[Robert Yang]

On 09/01/2013 06:07 AM, Otavio Salvador wrote:
> On Sat, Aug 31, 2013 at 7:51 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
>>
>> Hi RP,
>>
>> Thanks, I've made a draft patch for fixing why make-3.82 is being built,
>> the problem is that when we run "bitbake make-3.81":
>> - For 'make' itself, it will build make_3.81.bb
>> - But for make-dev/dbg, it will build make_3.82.bb since there is no
>> PREFERRED_VERSION and it will use the highest version.
>>
>> This draft patch sets the PREFERRED_VERSION for the pkg and will fix the
>> problem, I will send it to bitbake-devel later.
>
> Nice discovery :-) This does seem to address a core issue.
>
>> diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py
>> index 58fe199..fc8ee6d 100644
>> --- a/bitbake/lib/bb/taskdata.py
>> +++ b/bitbake/lib/bb/taskdata.py
>> @@ -429,6 +429,22 @@ class TaskData:
>>               return
>>
>>           all_p = dataCache.providers[item]
>> +        logger.warn("pppreferred: %s" % dataCache.preferred)
>> +
>> +        # Check whether item has the one of following formats and set the
>> +        # PREFERRED_VERSION if it does:
>> +        # - <pn>-<pv>
>> +        # - <pn>-<pv>-<pr> (when no PE)
>> +        # - <pn>_<pe>-<pv>-<pr> (when PE)
>> +        for fn in all_p:
>> +            pn = dataCache.pkg_fn[fn]
>> +            pe = dataCache.pkg_pepvpr[fn][0]
>> +            pv = dataCache.pkg_pepvpr[fn][1]
>> +            pr = dataCache.pkg_pepvpr[fn][2]
>> +            if not pe and (item == "%s-%s" % (pn, pv) or item == "%s-%s-%s"
>> % (pn, pv, pr)):
>> +                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)
>> +            elif pe and item == "%s_%s-%s-%s" % (pn, pe, pv, pr):
>> +                cfgData.setVar("PREFERRED_VERSION_" + pn, pv)
>
> I understand the problem here but maybe this should be checked in
> bitbake and avoid iterate in all packages?
>

Yes, you are right, it should be sent to bitbake-devel mailing list,
and I'm trying to avoid iterating in all packages.

// Robert