sstate info

sstate info

[Gary Thomas]

I'm trying to understand why sstate sometimes [most times in my case]
is not reusable.  Comparing some of the .siginfo files gives me some
info, e.g.

$ bitbake-diffsigs p60_poky/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-66ad8e10f260d506d065831a738c04a3_populate-sysroot.tgz.siginfo 
p60_test_pass1/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-c7e01d52b467bbe3af0556a43806521a_populate-sysroot.tgz.siginfo
Hash for dependent task virtual:nativepseudo_1.2.bb.do_install changed from 9edd41e331b29f08941225709c325a0d to 9865df7c4aa84fdb288c06faecf63426

How do I figure out what went into the hash(es) that changed?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate info

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

On Thu, Jan 12, 2012 at 02:57:21AM -0700, Gary Thomas wrote:
> I'm trying to understand why sstate sometimes [most times in my case]
> is not reusable.  Comparing some of the .siginfo files gives me some
> info, e.g.
> 
> $ bitbake-diffsigs p60_poky/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-66ad8e10f260d506d065831a738c04a3_populate-sysroot.tgz.siginfo 
> p60_test_pass1/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-c7e01d52b467bbe3af0556a43806521a_populate-sysroot.tgz.siginfo
> Hash for dependent task virtual:nativepseudo_1.2.bb.do_install changed from 9edd41e331b29f08941225709c325a0d to 9865df7c4aa84fdb288c06faecf63426
> 
> How do I figure out what went into the hash(es) that changed?

Try to look in stamps directory

something like:
tmp-eglibc/stamps/x86_64-linux/pseudo-native-1.2-r4.do_install.sigdata.504f9aac443e1f135aa2d1cbcf84c614

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: sstate info

[Gary Thomas]

On 2012-01-12 06:13, Martin Jansa wrote:
> On Thu, Jan 12, 2012 at 02:57:21AM -0700, Gary Thomas wrote:
>> I'm trying to understand why sstate sometimes [most times in my case]
>> is not reusable.  Comparing some of the .siginfo files gives me some
>> info, e.g.
>>
>> $ bitbake-diffsigs p60_poky/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-66ad8e10f260d506d065831a738c04a3_populate-sysroot.tgz.siginfo
>> p60_test_pass1/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-c7e01d52b467bbe3af0556a43806521a_populate-sysroot.tgz.siginfo
>> Hash for dependent task virtual:nativepseudo_1.2.bb.do_install changed from 9edd41e331b29f08941225709c325a0d to 9865df7c4aa84fdb288c06faecf63426
>>
>> How do I figure out what went into the hash(es) that changed?
>
> Try to look in stamps directory
>
> something like:
> tmp-eglibc/stamps/x86_64-linux/pseudo-native-1.2-r4.do_install.sigdata.504f9aac443e1f135aa2d1cbcf84c614

How can I interpret the contents of that file?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate info

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1427 bytes --]

On Thu, Jan 12, 2012 at 06:31:01AM -0700, Gary Thomas wrote:
> On 2012-01-12 06:13, Martin Jansa wrote:
> > On Thu, Jan 12, 2012 at 02:57:21AM -0700, Gary Thomas wrote:
> >> I'm trying to understand why sstate sometimes [most times in my case]
> >> is not reusable.  Comparing some of the .siginfo files gives me some
> >> info, e.g.
> >>
> >> $ bitbake-diffsigs p60_poky/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-66ad8e10f260d506d065831a738c04a3_populate-sysroot.tgz.siginfo
> >> p60_test_pass1/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-c7e01d52b467bbe3af0556a43806521a_populate-sysroot.tgz.siginfo
> >> Hash for dependent task virtual:nativepseudo_1.2.bb.do_install changed from 9edd41e331b29f08941225709c325a0d to 9865df7c4aa84fdb288c06faecf63426
> >>
> >> How do I figure out what went into the hash(es) that changed?
> >
> > Try to look in stamps directory
> >
> > something like:
> > tmp-eglibc/stamps/x86_64-linux/pseudo-native-1.2-r4.do_install.sigdata.504f9aac443e1f135aa2d1cbcf84c614
> 
> How can I interpret the contents of that file?

find file for virtual:nativepseudo_1.2.bb.do_install
with checksum 9edd41e331b29f08941225709c325a0d and 
with checksum 9865df7c4aa84fdb288c06faecf63426 and
then compare them with bitbake-diffsigs again

or use bitbake-diffsigs for one sigdata file

Cheers,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: sstate info

[Gary Thomas]

On 2012-01-12 06:37, Martin Jansa wrote:
> On Thu, Jan 12, 2012 at 06:31:01AM -0700, Gary Thomas wrote:
>> On 2012-01-12 06:13, Martin Jansa wrote:
>>> On Thu, Jan 12, 2012 at 02:57:21AM -0700, Gary Thomas wrote:
>>>> I'm trying to understand why sstate sometimes [most times in my case]
>>>> is not reusable.  Comparing some of the .siginfo files gives me some
>>>> info, e.g.
>>>>
>>>> $ bitbake-diffsigs p60_poky/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-66ad8e10f260d506d065831a738c04a3_populate-sysroot.tgz.siginfo
>>>> p60_test_pass1/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-c7e01d52b467bbe3af0556a43806521a_populate-sysroot.tgz.siginfo
>>>> Hash for dependent task virtual:nativepseudo_1.2.bb.do_install changed from 9edd41e331b29f08941225709c325a0d to 9865df7c4aa84fdb288c06faecf63426
>>>>
>>>> How do I figure out what went into the hash(es) that changed?
>>>
>>> Try to look in stamps directory
>>>
>>> something like:
>>> tmp-eglibc/stamps/x86_64-linux/pseudo-native-1.2-r4.do_install.sigdata.504f9aac443e1f135aa2d1cbcf84c614
>>
>> How can I interpret the contents of that file?
>
> find file for virtual:nativepseudo_1.2.bb.do_install
> with checksum 9edd41e331b29f08941225709c325a0d and
> with checksum 9865df7c4aa84fdb288c06faecf63426 and
> then compare them with bitbake-diffsigs again
>
> or use bitbake-diffsigs for one sigdata file

I worked this down to the do_patch in pseudo-native:
$ bitbake-diffsigs p60_poky/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.6e97a41d825568db298fdc9dbaed563c 
p60_test_pass1/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.a0d4e99625c211f4f1d793213014ac07
basehash changed from f5994d735ee2b909b72ee811364befc1 to 564be8db4146afa5fc3579dc4c1618d3
Variable DATE value changed from 20120110 to 20120111
Hash for dependent task quilt-native_0.50.bb.do_populate_sysroot changed from 9307106f3c5dfd49d953e206cb1b6da5 to 24151a370083d625b84a2afb77955021

Why is DATE as a variable important?
Why is it not ignored by this statement in ./meta/classes/patch.bbclass?
    patch_do_patch[vardepsexclude] = "DATE SRCDATE PATCHRESOLVE"

As is, it seems that the sstate cache is only good on the day it
was made :-(

Here's what I tried yesterday:
   * Create a new build, e.g.
     % . /local/poky/oe-init-build-env /local/test_pass1
     -- adjust local.conf to match desired target
     % bitbake core-image-minimal
   * Create a new build, using the previous one's sstate-cache
     % . /local/poky/oe-init-build-env /local/test_pass2
     -- adjust local.conf to match desired target
     -- enable sstate cache via these lines in local.conf:
          SSTATE_MIRRORS ?= "\
          file://.* file:///local/test_pass1/sstate-cache/"
     % bitbake core-image-minimal

This all went to plan - the sstate cache was reused for every package except
one.  For some reason, dbus-1 had to be rebuilt.

Today, I tried it again, just like the second step.  Low and behold, virtually
none of the sstate packages were reused :-(  This seems totally broken to me
and has been filed as bug #1896

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate info

[McClintock Matthew-B29882]

First off, what version of Yocto are you running? I've made a lot of
fixes post 1.1 release (others have made some as well ;)), and I
believe lots of these fixes will be in the 1.1.1 release. They should
all be in master, although I've not tested master sstate-cache in a
while now.

On Thu, Jan 12, 2012 at 9:33 AM, Gary Thomas <gary@mlbassoc.com> wrote:
> I worked this down to the do_patch in pseudo-native:
> $ bitbake-diffsigs
> p60_poky/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.6e97a41d825568db298fdc9dbaed563c
> p60_test_pass1/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.a0d4e99625c211f4f1d793213014ac07
> basehash changed from f5994d735ee2b909b72ee811364befc1 to
> 564be8db4146afa5fc3579dc4c1618d3
> Variable DATE value changed from 20120110 to 20120111
> Hash for dependent task quilt-native_0.50.bb.do_populate_sysroot changed
> from 9307106f3c5dfd49d953e206cb1b6da5 to 24151a370083d625b84a2afb77955021
>

This is telling you that you need to look at the quilt-native task
now. Nothing is wrong about the signature for psuedo-natve.

> Why is DATE as a variable important?
> Why is it not ignored by this statement in ./meta/classes/patch.bbclass?
>   patch_do_patch[vardepsexclude] = "DATE SRCDATE PATCHRESOLVE"

DATE is not important, that is why we are explicitly telling the
signature to ignore this variable.

> As is, it seems that the sstate cache is only good on the day it
> was made :-(

This is possible, but you need to check the upstream task that
changed. Since psuedo-native depends on quilt-native it's getting a
new signature since something that changed in quilt-native might
effect psuedo-native.

> Here's what I tried yesterday:
>  * Create a new build, e.g.
>    % . /local/poky/oe-init-build-env /local/test_pass1
>    -- adjust local.conf to match desired target
>    % bitbake core-image-minimal
>  * Create a new build, using the previous one's sstate-cache
>    % . /local/poky/oe-init-build-env /local/test_pass2
>    -- adjust local.conf to match desired target
>    -- enable sstate cache via these lines in local.conf:
>         SSTATE_MIRRORS ?= "\
>         file://.* file:///local/test_pass1/sstate-cache/"
>    % bitbake core-image-minimal
>
> This all went to plan - the sstate cache was reused for every package except
> one.  For some reason, dbus-1 had to be rebuilt.

This should work. I'm building on one machine and reusing on others
including native cache. I'm doing thing on something close to what
1.1.1 will look like.

> Today, I tried it again, just like the second step.  Low and behold,
> virtually
> none of the sstate packages were reused :-(  This seems totally broken to me
> and has been filed as bug #1896

-M

Re: sstate info

[Gary Thomas]

On 2012-01-12 09:53, McClintock Matthew-B29882 wrote:
> First off, what version of Yocto are you running? I've made a lot of
> fixes post 1.1 release (others have made some as well ;)), and I
> believe lots of these fixes will be in the 1.1.1 release. They should
> all be in master, although I've not tested master sstate-cache in a
> while now.
>
> On Thu, Jan 12, 2012 at 9:33 AM, Gary Thomas<gary@mlbassoc.com>  wrote:
>> I worked this down to the do_patch in pseudo-native:
>> $ bitbake-diffsigs
>> p60_poky/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.6e97a41d825568db298fdc9dbaed563c
>> p60_test_pass1/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.a0d4e99625c211f4f1d793213014ac07
>> basehash changed from f5994d735ee2b909b72ee811364befc1 to
>> 564be8db4146afa5fc3579dc4c1618d3
>> Variable DATE value changed from 20120110 to 20120111
>> Hash for dependent task quilt-native_0.50.bb.do_populate_sysroot changed
>> from 9307106f3c5dfd49d953e206cb1b6da5 to 24151a370083d625b84a2afb77955021
>>
>
> This is telling you that you need to look at the quilt-native task
> now. Nothing is wrong about the signature for psuedo-natve.
>
>> Why is DATE as a variable important?
>> Why is it not ignored by this statement in ./meta/classes/patch.bbclass?
>>    patch_do_patch[vardepsexclude] = "DATE SRCDATE PATCHRESOLVE"
>
> DATE is not important, that is why we are explicitly telling the
> signature to ignore this variable.

It seems that DATE needs to be ignored other places as well.  Digging
further, I find:

$ bitbake-diffsigs p60_poky/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.216968eaca38198b88a7520be02e3fca 
p60_test_pass1/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.0ce7898e700de80ff8afcb0850ec4ed0
basehash changed from 4863b21531f8236184a7c3b16da3ea13 to 7ee430d1251148b2509ef2aefb6e5443
Variable DATE value changed from 20120110 to 20120111

which is why pseudo-native changed which is why almost every other
recipe signature changed :-(

>
>> As is, it seems that the sstate cache is only good on the day it
>> was made :-(
>
> This is possible, but you need to check the upstream task that
> changed. Since psuedo-native depends on quilt-native it's getting a
> new signature since something that changed in quilt-native might
> effect psuedo-native.
>
>> Here's what I tried yesterday:
>>   * Create a new build, e.g.
>>     % . /local/poky/oe-init-build-env /local/test_pass1
>>     -- adjust local.conf to match desired target
>>     % bitbake core-image-minimal
>>   * Create a new build, using the previous one's sstate-cache
>>     % . /local/poky/oe-init-build-env /local/test_pass2
>>     -- adjust local.conf to match desired target
>>     -- enable sstate cache via these lines in local.conf:
>>          SSTATE_MIRRORS ?= "\
>>          file://.* file:///local/test_pass1/sstate-cache/"
>>     % bitbake core-image-minimal
>>
>> This all went to plan - the sstate cache was reused for every package except
>> one.  For some reason, dbus-1 had to be rebuilt.
>
> This should work. I'm building on one machine and reusing on others
> including native cache. I'm doing thing on something close to what
> 1.1.1 will look like.

It did work *YESTERDAY*, but the exact same steps today do not work.
There were no changes to my poky source tree and no new packages
were downloaded.

>
>> Today, I tried it again, just like the second step.  Low and behold,
>> virtually
>> none of the sstate packages were reused :-(  This seems totally broken to me
>> and has been filed as bug #1896
>
> -M

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate info

[McClintock Matthew-B29882]

On Thu, Jan 12, 2012 at 11:06 AM, Gary Thomas <gary@mlbassoc.com> wrote:
>> DATE is not important, that is why we are explicitly telling the
>> signature to ignore this variable.
>
>
> It seems that DATE needs to be ignored other places as well.  Digging
> further, I find:
>
> $ bitbake-diffsigs
> p60_poky/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.216968eaca38198b88a7520be02e3fca
> p60_test_pass1/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.0ce7898e700de80ff8afcb0850ec4ed0
> basehash changed from 4863b21531f8236184a7c3b16da3ea13 to
> 7ee430d1251148b2509ef2aefb6e5443

You might just try adding to BB_HASHBASE_WHITELIST in bitbake.conf
which should ignore DATE in all recipes. I thought this was there, you
can check the history to see...

-M

Re: sstate info

[Gary Thomas]

On 2012-01-12 10:23, McClintock Matthew-B29882 wrote:
> On Thu, Jan 12, 2012 at 11:06 AM, Gary Thomas<gary@mlbassoc.com>  wrote:
>>> DATE is not important, that is why we are explicitly telling the
>>> signature to ignore this variable.
>>
>>
>> It seems that DATE needs to be ignored other places as well.  Digging
>> further, I find:
>>
>> $ bitbake-diffsigs
>> p60_poky/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.216968eaca38198b88a7520be02e3fca
>> p60_test_pass1/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.0ce7898e700de80ff8afcb0850ec4ed0
>> basehash changed from 4863b21531f8236184a7c3b16da3ea13 to
>> 7ee430d1251148b2509ef2aefb6e5443
>
> You might just try adding to BB_HASHBASE_WHITELIST in bitbake.conf
> which should ignore DATE in all recipes. I thought this was there, you
> can check the history to see...

I'm trying this now.  Of course, touching that variable invalidates
the sstate cache, so I won't know until tomorrow if it works.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate info

[Chris Larson]

On Thu, Jan 12, 2012 at 2:07 PM, Gary Thomas <gary@mlbassoc.com> wrote:
> On 2012-01-12 10:23, McClintock Matthew-B29882 wrote:
>>
>> On Thu, Jan 12, 2012 at 11:06 AM, Gary Thomas<gary@mlbassoc.com>  wrote:
>>>>
>>>> DATE is not important, that is why we are explicitly telling the
>>>> signature to ignore this variable.
>>>
>>>
>>>
>>> It seems that DATE needs to be ignored other places as well.  Digging
>>> further, I find:
>>>
>>> $ bitbake-diffsigs
>>>
>>> p60_poky/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.216968eaca38198b88a7520be02e3fca
>>>
>>> p60_test_pass1/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.0ce7898e700de80ff8afcb0850ec4ed0
>>> basehash changed from 4863b21531f8236184a7c3b16da3ea13 to
>>> 7ee430d1251148b2509ef2aefb6e5443
>>
>>
>> You might just try adding to BB_HASHBASE_WHITELIST in bitbake.conf
>> which should ignore DATE in all recipes. I thought this was there, you
>> can check the history to see...
>
>
> I'm trying this now.  Of course, touching that variable invalidates
> the sstate cache, so I won't know until tomorrow if it works.

You can't just override DATE in local.conf? I don't see the point in
holding off, unless your builds will take till tomorrow to finish?
-- 
Christopher Larson

Re: sstate info

[McClintock Matthew-B29882]

On Thu, Jan 12, 2012 at 3:07 PM, Gary Thomas <gary@mlbassoc.com> wrote:
> I'm trying this now.  Of course, touching that variable invalidates
> the sstate cache, so I won't know until tomorrow if it works.

Yes, it's hard (long) to test this. Then you will want to wait another
day to know for sure the DATE dependency is gone to know it's work for
sure. I'd recommend purchasing a large machine to do this testing
on.... it's well worth the investment ;)

-M

Re: sstate info

[McClintock Matthew-B29882]

On Thu, Jan 12, 2012 at 3:12 PM, Chris Larson <clarson@kergoth.com> wrote:
>> I'm trying this now.  Of course, touching that variable invalidates
>> the sstate cache, so I won't know until tomorrow if it works.
>
> You can't just override DATE in local.conf? I don't see the point in
> holding off, unless your builds will take till tomorrow to finish?

This would be a good way to test it out...

-M

Re: sstate info

[Gary Thomas]

On 2012-01-12 14:13, McClintock Matthew-B29882 wrote:
> On Thu, Jan 12, 2012 at 3:07 PM, Gary Thomas<gary@mlbassoc.com>  wrote:
>> I'm trying this now.  Of course, touching that variable invalidates
>> the sstate cache, so I won't know until tomorrow if it works.
>
> Yes, it's hard (long) to test this. Then you will want to wait another
> day to know for sure the DATE dependency is gone to know it's work for
> sure. I'd recommend purchasing a large machine to do this testing
> on.... it's well worth the investment ;)

I have many such large machines at my disposal, but sadly, no time machine :-)

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate info

[Gary Thomas]

On 2012-01-12 14:12, Chris Larson wrote:
> On Thu, Jan 12, 2012 at 2:07 PM, Gary Thomas<gary@mlbassoc.com>  wrote:
>> On 2012-01-12 10:23, McClintock Matthew-B29882 wrote:
>>>
>>> On Thu, Jan 12, 2012 at 11:06 AM, Gary Thomas<gary@mlbassoc.com>    wrote:
>>>>>
>>>>> DATE is not important, that is why we are explicitly telling the
>>>>> signature to ignore this variable.
>>>>
>>>>
>>>>
>>>> It seems that DATE needs to be ignored other places as well.  Digging
>>>> further, I find:
>>>>
>>>> $ bitbake-diffsigs
>>>>
>>>> p60_poky/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.216968eaca38198b88a7520be02e3fca
>>>>
>>>> p60_test_pass1/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.0ce7898e700de80ff8afcb0850ec4ed0
>>>> basehash changed from 4863b21531f8236184a7c3b16da3ea13 to
>>>> 7ee430d1251148b2509ef2aefb6e5443
>>>
>>>
>>> You might just try adding to BB_HASHBASE_WHITELIST in bitbake.conf
>>> which should ignore DATE in all recipes. I thought this was there, you
>>> can check the history to see...
>>
>>
>> I'm trying this now.  Of course, touching that variable invalidates
>> the sstate cache, so I won't know until tomorrow if it works.
>
> You can't just override DATE in local.conf? I don't see the point in
> holding off, unless your builds will take till tomorrow to finish?

Actually, I didn't think of that.  I also didn't see this message
until Friday (the next day) due to some network issues, so it's moot.

That notwithstanding, I tried it today (new DATE) and it behaved as
I would like.  The only problem was that dbus-1 is still being rebuilt.
I compared the siginfo files between the two builds and they are
identical, so I don't know why.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate info

[McClintock Matthew-B29882]

On Fri, Jan 13, 2012 at 9:23 AM, Gary Thomas <gary@mlbassoc.com> wrote:
>>
>> You can't just override DATE in local.conf? I don't see the point in
>> holding off, unless your builds will take till tomorrow to finish?
>
>
> Actually, I didn't think of that.  I also didn't see this message
> until Friday (the next day) due to some network issues, so it's moot.
>
> That notwithstanding, I tried it today (new DATE) and it behaved as
> I would like.  The only problem was that dbus-1 is still being rebuilt.
> I compared the siginfo files between the two builds and they are
> identical, so I don't know why.

Are the hashes the same too?

-M

Re: sstate info

[Richard Purdie]

On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
> On Fri, Jan 13, 2012 at 9:23 AM, Gary Thomas <gary@mlbassoc.com> wrote:
> >>
> >> You can't just override DATE in local.conf? I don't see the point in
> >> holding off, unless your builds will take till tomorrow to finish?
> >
> >
> > Actually, I didn't think of that.  I also didn't see this message
> > until Friday (the next day) due to some network issues, so it's moot.
> >
> > That notwithstanding, I tried it today (new DATE) and it behaved as
> > I would like.  The only problem was that dbus-1 is still being rebuilt.
> > I compared the siginfo files between the two builds and they are
> > identical, so I don't know why.
> 
> Are the hashes the same too?

I've found the root cause of this problem and have posted a fix on the
OE-Core mailing list.

Cheers,

Richard

Re: sstate info

[Gary Thomas]

On 2012-01-13 09:27, Richard Purdie wrote:
> On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
>> On Fri, Jan 13, 2012 at 9:23 AM, Gary Thomas<gary@mlbassoc.com>  wrote:
>>>>
>>>> You can't just override DATE in local.conf? I don't see the point in
>>>> holding off, unless your builds will take till tomorrow to finish?
>>>
>>>
>>> Actually, I didn't think of that.  I also didn't see this message
>>> until Friday (the next day) due to some network issues, so it's moot.
>>>
>>> That notwithstanding, I tried it today (new DATE) and it behaved as
>>> I would like.  The only problem was that dbus-1 is still being rebuilt.
>>> I compared the siginfo files between the two builds and they are
>>> identical, so I don't know why.
>>
>> Are the hashes the same too?
>
> I've found the root cause of this problem and have posted a fix on the
> OE-Core mailing list.

Cool, I'll start testing this now :-)

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate info

[Gary Thomas]

On 2012-01-13 09:53, Gary Thomas wrote:
> On 2012-01-13 09:27, Richard Purdie wrote:
>> On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
>>> On Fri, Jan 13, 2012 at 9:23 AM, Gary Thomas<gary@mlbassoc.com> wrote:
>>>>>
>>>>> You can't just override DATE in local.conf? I don't see the point in
>>>>> holding off, unless your builds will take till tomorrow to finish?
>>>>
>>>>
>>>> Actually, I didn't think of that. I also didn't see this message
>>>> until Friday (the next day) due to some network issues, so it's moot.
>>>>
>>>> That notwithstanding, I tried it today (new DATE) and it behaved as
>>>> I would like. The only problem was that dbus-1 is still being rebuilt.
>>>> I compared the siginfo files between the two builds and they are
>>>> identical, so I don't know why.
>>>
>>> Are the hashes the same too?
>>
>> I've found the root cause of this problem and have posted a fix on the
>> OE-Core mailing list.
>
> Cool, I'll start testing this now :-)
>

I won't know until later as this change itself invalidates the sstate cache,
so I have to start a full rebuild to test it properly.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate info

[Richard Purdie]

On Fri, 2012-01-13 at 09:53 -0700, Gary Thomas wrote:
> On 2012-01-13 09:27, Richard Purdie wrote:
> > On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
> >> On Fri, Jan 13, 2012 at 9:23 AM, Gary Thomas<gary@mlbassoc.com>  wrote:
> >>>>
> >>>> You can't just override DATE in local.conf? I don't see the point in
> >>>> holding off, unless your builds will take till tomorrow to finish?
> >>>
> >>>
> >>> Actually, I didn't think of that.  I also didn't see this message
> >>> until Friday (the next day) due to some network issues, so it's moot.
> >>>
> >>> That notwithstanding, I tried it today (new DATE) and it behaved as
> >>> I would like.  The only problem was that dbus-1 is still being rebuilt.
> >>> I compared the siginfo files between the two builds and they are
> >>> identical, so I don't know why.
> >>
> >> Are the hashes the same too?
> >
> > I've found the root cause of this problem and have posted a fix on the
> > OE-Core mailing list.
> 
> Cool, I'll start testing this now :-)

I've also noticed a problem with gcc being rebuilt. It seems to be from
do_headerfix() and:

diff --git a/meta/recipes-devtools/gcc/gcc-configure-common.inc b/meta/recipes-devtools/gcc/gcc-configure-common.inc
index d014980..3a82720 100644
--- a/meta/recipes-devtools/gcc/gcc-configure-common.inc
+++ b/meta/recipes-devtools/gcc/gcc-configure-common.inc
@@ -77,6 +77,9 @@ do_headerfix () {
 
 addtask headerfix after do_unpack before do_patch
 
+CROSS_TARGET_SYS_DIR[vardepsexclude] = "PN"
+CROSS_TARGET_SYS_DIR[vardepvalue] = "1"
+
 do_configure_prepend () {
        # teach gcc to find correct target includedir when checking libc ssp support
        mkdir -p ${B}/gcc

"fixes" it. I merged the other fix but this one needs a little bit more thought. bitbake-diffsigs can also mislead:

$ ls *headerfix*
gcc-cross-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
gcc-cross-initial-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
$ bitbake-diffsigs *headerfix*
Dependency on task gcc-cross_4.6.bb.do_unpack was added
Dependency on task gcc-cross-initial_4.6.bb.do_unpack was removed

Re: sstate info

[Richard Purdie]

On Fri, 2012-01-13 at 09:53 -0700, Gary Thomas wrote:
> On 2012-01-13 09:27, Richard Purdie wrote:
> > On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
> >> On Fri, Jan 13, 2012 at 9:23 AM, Gary Thomas<gary@mlbassoc.com>  wrote:
> >>>>
> >>>> You can't just override DATE in local.conf? I don't see the point in
> >>>> holding off, unless your builds will take till tomorrow to finish?
> >>>
> >>>
> >>> Actually, I didn't think of that.  I also didn't see this message
> >>> until Friday (the next day) due to some network issues, so it's moot.
> >>>
> >>> That notwithstanding, I tried it today (new DATE) and it behaved as
> >>> I would like.  The only problem was that dbus-1 is still being rebuilt.
> >>> I compared the siginfo files between the two builds and they are
> >>> identical, so I don't know why.
> >>
> >> Are the hashes the same too?
> >
> > I've found the root cause of this problem and have posted a fix on the
> > OE-Core mailing list.
> 
> Cool, I'll start testing this now :-)

I've also noticed a problem with gcc being rebuilt. It seems to be from
do_headerfix() and:

diff --git a/meta/recipes-devtools/gcc/gcc-configure-common.inc b/meta/recipes-devtools/gcc/gcc-configure-common.inc
index d014980..3a82720 100644
--- a/meta/recipes-devtools/gcc/gcc-configure-common.inc
+++ b/meta/recipes-devtools/gcc/gcc-configure-common.inc
@@ -77,6 +77,9 @@ do_headerfix () {
 
 addtask headerfix after do_unpack before do_patch
 
+CROSS_TARGET_SYS_DIR[vardepsexclude] = "PN"
+CROSS_TARGET_SYS_DIR[vardepvalue] = "1"
+
 do_configure_prepend () {
        # teach gcc to find correct target includedir when checking libc ssp support
        mkdir -p ${B}/gcc

"fixes" it. I merged the other fix but this one needs a little bit more thought. bitbake-diffsigs can also mislead:

$ ls *headerfix*
gcc-cross-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
gcc-cross-initial-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
$ bitbake-diffsigs *headerfix*
Dependency on task gcc-cross_4.6.bb.do_unpack was added
Dependency on task gcc-cross-initial_4.6.bb.do_unpack was removed

Note that the hashes are the same and these are in fact identical. The
dependency hashes *do* match and its bitbake-diffsigs that is wrong here
as the file changed names.

Cheers,

Richard

Re: sstate info

[Gary Thomas]

On 2012-01-13 11:09, Richard Purdie wrote:
> On Fri, 2012-01-13 at 09:53 -0700, Gary Thomas wrote:
>> On 2012-01-13 09:27, Richard Purdie wrote:
>>> On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
>>>> On Fri, Jan 13, 2012 at 9:23 AM, Gary Thomas<gary@mlbassoc.com>   wrote:
>>>>>>
>>>>>> You can't just override DATE in local.conf? I don't see the point in
>>>>>> holding off, unless your builds will take till tomorrow to finish?
>>>>>
>>>>>
>>>>> Actually, I didn't think of that.  I also didn't see this message
>>>>> until Friday (the next day) due to some network issues, so it's moot.
>>>>>
>>>>> That notwithstanding, I tried it today (new DATE) and it behaved as
>>>>> I would like.  The only problem was that dbus-1 is still being rebuilt.
>>>>> I compared the siginfo files between the two builds and they are
>>>>> identical, so I don't know why.
>>>>
>>>> Are the hashes the same too?
>>>
>>> I've found the root cause of this problem and have posted a fix on the
>>> OE-Core mailing list.
>>
>> Cool, I'll start testing this now :-)
>
> I've also noticed a problem with gcc being rebuilt. It seems to be from
> do_headerfix() and:
>
> diff --git a/meta/recipes-devtools/gcc/gcc-configure-common.inc b/meta/recipes-devtools/gcc/gcc-configure-common.inc
> index d014980..3a82720 100644
> --- a/meta/recipes-devtools/gcc/gcc-configure-common.inc
> +++ b/meta/recipes-devtools/gcc/gcc-configure-common.inc
> @@ -77,6 +77,9 @@ do_headerfix () {
>
>   addtask headerfix after do_unpack before do_patch
>
> +CROSS_TARGET_SYS_DIR[vardepsexclude] = "PN"
> +CROSS_TARGET_SYS_DIR[vardepvalue] = "1"
> +
>   do_configure_prepend () {
>          # teach gcc to find correct target includedir when checking libc ssp support
>          mkdir -p ${B}/gcc
>
> "fixes" it. I merged the other fix but this one needs a little bit more thought. bitbake-diffsigs can also mislead:
>
> $ ls *headerfix*
> gcc-cross-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
> gcc-cross-initial-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
> $ bitbake-diffsigs *headerfix*
> Dependency on task gcc-cross_4.6.bb.do_unpack was added
> Dependency on task gcc-cross-initial_4.6.bb.do_unpack was removed

While you're looking at gcc, I noticed something else.  I have
multiple MACHINE configurations which are all the same basic
hardware (OMAP3530), the only difference being kernel settings
and other minor target variations.  As far as things like the
compiler and common ARM tools, they are identical.

I tried sharing the sstate between two of these and for the
most part, all was well - only the truly machine dependent
recipes were rebuilt.  Except for gcc-cross-4.6.2 which seems
to have a dependency on the MACHINE variable.  This cascaded
into also building a few other packages (glib, libtool, shadow?
and udev)  [note: I was building a board with SDK tools which
may have brought some of these in].  Does this seem reasonable
to you?  I would hope that these packages, which are placed
in the architecture (armv7a-vfp-neon-amltd-linux-gnueabi) tree
should *not* depend on MACHINE.

Thanks for your help with this.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate info

[McClintock Matthew-B29882]

On Fri, Jan 13, 2012 at 12:19 PM, Gary Thomas <gary@mlbassoc.com> wrote:
> While you're looking at gcc, I noticed something else.  I have
> multiple MACHINE configurations which are all the same basic
> hardware (OMAP3530), the only difference being kernel settings
> and other minor target variations.  As far as things like the
> compiler and common ARM tools, they are identical.
>
> I tried sharing the sstate between two of these and for the
> most part, all was well - only the truly machine dependent
> recipes were rebuilt.  Except for gcc-cross-4.6.2 which seems
> to have a dependency on the MACHINE variable.  This cascaded
> into also building a few other packages (glib, libtool, shadow?
> and udev)  [note: I was building a board with SDK tools which
> may have brought some of these in].  Does this seem reasonable
> to you?  I would hope that these packages, which are placed
> in the architecture (armv7a-vfp-neon-amltd-linux-gnueabi) tree
> should *not* depend on MACHINE.
>
> Thanks for your help with this.

More than likely the MACHINE variable should not be a dependency.
Which variable is depending on MACHINE? You should do some more
investigation.

-M

Re: sstate info

[Gary Thomas]

On 2012-01-13 10:11, Gary Thomas wrote:
> On 2012-01-13 09:53, Gary Thomas wrote:
>> On 2012-01-13 09:27, Richard Purdie wrote:
>>> On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
>>>> On Fri, Jan 13, 2012 at 9:23 AM, Gary Thomas<gary@mlbassoc.com> wrote:
>>>>>>
>>>>>> You can't just override DATE in local.conf? I don't see the point in
>>>>>> holding off, unless your builds will take till tomorrow to finish?
>>>>>
>>>>>
>>>>> Actually, I didn't think of that. I also didn't see this message
>>>>> until Friday (the next day) due to some network issues, so it's moot.
>>>>>
>>>>> That notwithstanding, I tried it today (new DATE) and it behaved as
>>>>> I would like. The only problem was that dbus-1 is still being rebuilt.
>>>>> I compared the siginfo files between the two builds and they are
>>>>> identical, so I don't know why.
>>>>
>>>> Are the hashes the same too?
>>>
>>> I've found the root cause of this problem and have posted a fix on the
>>> OE-Core mailing list.
>>
>> Cool, I'll start testing this now :-)
>>
>
> I won't know until later as this change itself invalidates the sstate cache,
> so I have to start a full rebuild to test it properly.
>

Richard's change works just as well as adding DATE to BB_HASH_WHITELIST.

In my minimal test, there are three packages being rebuilt: dbus-1,
base-files and shadow-sysroot.  It's still not clear to me why this
happened.  I've put the tmp/stamps trees at http://www.mlbassoc.com/misc/poky_stamps.tgz
This has two directories in it
   p60_test_pass1 - the original tree, build without any sstate
   p60_test_pass2 - the dependent tree, build using p60_test_pass1 as sstate
Perhaps someone can give me a clue why these three packages were
rebuilt.  Note: there were no changes to the source repository
between the two builds and I always build with BB_NO_NETWORK="1"

Thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate info

[Richard Purdie]

On Sat, 2012-01-14 at 06:30 -0700, Gary Thomas wrote:
> Richard's change works just as well as adding DATE to BB_HASH_WHITELIST.

Great!

> In my minimal test, there are three packages being rebuilt: dbus-1,
> base-files and shadow-sysroot.  It's still not clear to me why this
> happened.  I've put the tmp/stamps trees at http://www.mlbassoc.com/misc/poky_stamps.tgz
> This has two directories in it
>    p60_test_pass1 - the original tree, build without any sstate
>    p60_test_pass2 - the dependent tree, build using p60_test_pass1 as sstate
> Perhaps someone can give me a clue why these three packages were
> rebuilt.  Note: there were no changes to the source repository
> between the two builds and I always build with BB_NO_NETWORK="1"

I suspect its useradd related and there is already an open bug:

http://bugzilla.yoctoproject.org/show_bug.cgi?id=1721

Cheers,

Richard