bitbake dependency cache problem

bitbake dependency cache problem

[Steffen Sledz]

I'm working with oe-classic and BitBake Build Tool Core version 1.12.0, bitbake version 1.12.0.

Because of some special development requirements we like to generate the Package Version from the SVN Revision (not the Last Changed Rev!) of the bitbake recipe in the local workspace.

Therefor the recipe contains this:

---------------->snip<-----------------
PR = "r8"
PV = "svnr${@svn_revision(d)}"

inherit svn-helper
---------------->snip<-----------------

The svn-helper.bbclass contains a little helper function to determine the needed value:

---------------->snip<-----------------
def svn_revision(d):
     import subprocess
     bbpath = os.path.dirname(bb.data.getVar('FILE',d,1))
     return subprocess.check_output(["svn", "info", bbpath]).partition("Revision: ")[2].splitlines()[0]
---------------->snip<-----------------

After this changes i make a first build and everything works fine. Assuming the SVN Revsion is 42 a package called foo-svnr42-r8.ipk is generated.

Now i make an "svn update" inside the workspace and the SVN Revision increases to 66.

A call of "bitbake foo" now results in an "Tasks Summary: Attempted 1182 tasks of which 1182 didn't need to be rerun and 0 failed." an no new ipg is generated. :(

The log generated by "bitbake -DDDDDvvvvv foo" contains

---------------->snip<-----------------
DEBUG: providers for foo are: ['foo']
NOTE: checking PREFERRED_PROVIDER_foo
NOTE: checking PREFERRED_PROVIDER_foo-svnr42
NOTE: checking PREFERRED_PROVIDER_foo-svnr42-r8
---------------->snip<-----------------

Why does bitbake not respects the new PV and generates a foo-svnr66-r8.ipk here???

Regards,
Steffen

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: bitbake dependency cache problem

[Steffen Sledz]

On 01.03.2012 14:46, Steffen Sledz wrote:
> I'm working with oe-classic and BitBake Build Tool Core version 1.12.0, bitbake version 1.12.0.
> 
> Because of some special development requirements we like to generate the Package Version from the SVN Revision (not the Last Changed Rev!) of the bitbake recipe in the local workspace.
> 
> Therefor the recipe contains this:
> 
> ---------------->snip<-----------------
> PR = "r8"
> PV = "svnr${@svn_revision(d)}"
> 
> inherit svn-helper
> ---------------->snip<-----------------
> 
> The svn-helper.bbclass contains a little helper function to determine the needed value:
> 
> ---------------->snip<-----------------
> def svn_revision(d):
>      import subprocess
>      bbpath = os.path.dirname(bb.data.getVar('FILE',d,1))
>      return subprocess.check_output(["svn", "info", bbpath]).partition("Revision: ")[2].splitlines()[0]
> ---------------->snip<-----------------
> 
> After this changes i make a first build and everything works fine. Assuming the SVN Revsion is 42 a package called foo-svnr42-r8.ipk is generated.
> 
> Now i make an "svn update" inside the workspace and the SVN Revision increases to 66.
> 
> A call of "bitbake foo" now results in an "Tasks Summary: Attempted 1182 tasks of which 1182 didn't need to be rerun and 0 failed." an no new ipg is generated. :(
> 
> The log generated by "bitbake -DDDDDvvvvv foo" contains
> 
> ---------------->snip<-----------------
> DEBUG: providers for foo are: ['foo']
> NOTE: checking PREFERRED_PROVIDER_foo
> NOTE: checking PREFERRED_PROVIDER_foo-svnr42
> NOTE: checking PREFERRED_PROVIDER_foo-svnr42-r8
> ---------------->snip<-----------------
> 
> Why does bitbake not respects the new PV and generates a foo-svnr66-r8.ipk here???

Ping!

Can anyone of the bitbake gurus lease give a feedback here?

Thx,
Steffen

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: bitbake dependency cache problem

[Steffen Sledz]

On 01.03.2012 14:46, Steffen Sledz wrote:
> I'm working with oe-classic and BitBake Build Tool Core version 1.12.0, bitbake version 1.12.0.
> 
> Because of some special development requirements we like to generate the Package Version from the SVN Revision (not the Last Changed Rev!) of the bitbake recipe in the local workspace.
> 
> Therefor the recipe contains this:
> 
> ---------------->snip<-----------------
> PR = "r8"
> PV = "svnr${@svn_revision(d)}"
> 
> inherit svn-helper
> ---------------->snip<-----------------
> 
> The svn-helper.bbclass contains a little helper function to determine the needed value:
> 
> ---------------->snip<-----------------
> def svn_revision(d):
>      import subprocess
>      bbpath = os.path.dirname(bb.data.getVar('FILE',d,1))
>      return subprocess.check_output(["svn", "info", bbpath]).partition("Revision: ")[2].splitlines()[0]
> ---------------->snip<-----------------
> 
> After this changes i make a first build and everything works fine. Assuming the SVN Revsion is 42 a package called foo-svnr42-r8.ipk is generated.
> 
> Now i make an "svn update" inside the workspace and the SVN Revision increases to 66.
> 
> A call of "bitbake foo" now results in an "Tasks Summary: Attempted 1182 tasks of which 1182 didn't need to be rerun and 0 failed." an no new ipg is generated. :(
> 
> The log generated by "bitbake -DDDDDvvvvv foo" contains
> 
> ---------------->snip<-----------------
> DEBUG: providers for foo are: ['foo']
> NOTE: checking PREFERRED_PROVIDER_foo
> NOTE: checking PREFERRED_PROVIDER_foo-svnr42
> NOTE: checking PREFERRED_PROVIDER_foo-svnr42-r8
> ---------------->snip<-----------------
> 
> Why does bitbake not respects the new PV and generates a foo-svnr66-r8.ipk here???

Ping!

Can anyone of the bitbake gurus lease give a feedback here?

Thx,
Steffen

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: bitbake dependency cache problem

[Richard Purdie]

On Thu, 2012-03-01 at 14:46 +0100, Steffen Sledz wrote:
> I'm working with oe-classic and BitBake Build Tool Core version
> 1.12.0, bitbake version 1.12.0.
> 
> Because of some special development requirements we like to generate
> the Package Version from the SVN Revision (not the Last Changed Rev!)
> of the bitbake recipe in the local workspace.
> 
> Therefor the recipe contains this:
> 
> ---------------->snip<-----------------
> PR = "r8"
> PV = "svnr${@svn_revision(d)}"
> 
> inherit svn-helper
> ---------------->snip<-----------------
> 
> The svn-helper.bbclass contains a little helper function to determine
> the needed value:
> 
> ---------------->snip<-----------------
> def svn_revision(d):
>      import subprocess
>      bbpath = os.path.dirname(bb.data.getVar('FILE',d,1))
>      return subprocess.check_output(["svn", "info", bbpath]).partition("Revision: ")[2].splitlines()[0]
> ---------------->snip<-----------------
> 
> After this changes i make a first build and everything works fine.
> Assuming the SVN Revsion is 42 a package called foo-svnr42-r8.ipk is
> generated.
> 
> Now i make an "svn update" inside the workspace and the SVN Revision
> increases to 66.
> 
> A call of "bitbake foo" now results in an "Tasks Summary: Attempted
> 1182 tasks of which 1182 didn't need to be rerun and 0 failed." an no
> new ipg is generated. :(
> 
> The log generated by "bitbake -DDDDDvvvvv foo" contains
> 
> ---------------->snip<-----------------
> DEBUG: providers for foo are: ['foo']
> NOTE: checking PREFERRED_PROVIDER_foo
> NOTE: checking PREFERRED_PROVIDER_foo-svnr42
> NOTE: checking PREFERRED_PROVIDER_foo-svnr42-r8
> ---------------->snip<-----------------
> 
> Why does bitbake not respects the new PV and generates a
> foo-svnr66-r8.ipk here???

Bitbake has no idea that the PV has changed and that it needs to ignore
the cache value and reparse the metadata for that file. The similar
example is when autorev is used with a source control system. You can
can see the code for this in lib/bb/fetch2/__init__.py in get_autorev().
In summary, if you set __BB_DONT_CACHE = "1" in your recipe, I think it
will do what you expect, at the cost that it will reparse the recipe
every time (which it has to in case PV changes).

Cheers,

Richard

Re: bitbake dependency cache problem

[Steffen Sledz]

On 05.03.2012 14:28, Richard Purdie wrote:
> On Thu, 2012-03-01 at 14:46 +0100, Steffen Sledz wrote:
>> I'm working with oe-classic and BitBake Build Tool Core version
>> 1.12.0, bitbake version 1.12.0.
>>
>> Because of some special development requirements we like to generate
>> the Package Version from the SVN Revision (not the Last Changed Rev!)
>> of the bitbake recipe in the local workspace.
>>
>> Therefor the recipe contains this:
>>
>> ---------------->snip<-----------------
>> PR = "r8"
>> PV = "svnr${@svn_revision(d)}"
>>
>> inherit svn-helper
>> ---------------->snip<-----------------
>>
>> The svn-helper.bbclass contains a little helper function to determine
>> the needed value:
>>
>> ---------------->snip<-----------------
>> def svn_revision(d):
>>      import subprocess
>>      bbpath = os.path.dirname(bb.data.getVar('FILE',d,1))
>>      return subprocess.check_output(["svn", "info", bbpath]).partition("Revision: ")[2].splitlines()[0]
>> ---------------->snip<-----------------
>>
>> After this changes i make a first build and everything works fine.
>> Assuming the SVN Revsion is 42 a package called foo-svnr42-r8.ipk is
>> generated.
>>
>> Now i make an "svn update" inside the workspace and the SVN Revision
>> increases to 66.
>>
>> A call of "bitbake foo" now results in an "Tasks Summary: Attempted
>> 1182 tasks of which 1182 didn't need to be rerun and 0 failed." an no
>> new ipg is generated. :(
>>
>> The log generated by "bitbake -DDDDDvvvvv foo" contains
>>
>> ---------------->snip<-----------------
>> DEBUG: providers for foo are: ['foo']
>> NOTE: checking PREFERRED_PROVIDER_foo
>> NOTE: checking PREFERRED_PROVIDER_foo-svnr42
>> NOTE: checking PREFERRED_PROVIDER_foo-svnr42-r8
>> ---------------->snip<-----------------
>>
>> Why does bitbake not respects the new PV and generates a
>> foo-svnr66-r8.ipk here???
> 
> Bitbake has no idea that the PV has changed and that it needs to ignore
> the cache value and reparse the metadata for that file. The similar
> example is when autorev is used with a source control system. You can
> can see the code for this in lib/bb/fetch2/__init__.py in get_autorev().
> In summary, if you set __BB_DONT_CACHE = "1" in your recipe, I think it
> will do what you expect, at the cost that it will reparse the recipe
> every time (which it has to in case PV changes).

This solution seems to work. :)

Thx,
Steffen

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058