How to find out why shared sstate is not being used for a recipe

How to find out why shared sstate is not being used for a recipe

[Mike Looijmans]

I have a deja-vu feeling about this question.

I have this recipe:

https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb

Which includes this one:
https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc

I have a build server that exports its sstate-cache directory through HTTP, 
and a local host that attempts to use that sstate-cache. This works fine, 
except for the recipe above. Building this recipe takes about 1 hour, so i 
really really really want to share that state at any cost. As you can see, 
I've done a big shotgun blast of "vardepdsexclude" to get the recipe to be as 
common as possible. Still any host wants to build its own version.

How can I diagnose the REASON that my machine thinks it isn't building the 
exact same thing as the build server?


Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20
http://medteceurope.com/index.php?page=home-en

Re: How to find out why shared sstate is not being used for a recipe

[Khem Raj]

On Mon, May 26, 2014 at 11:58 PM, Mike Looijmans
<mike.looijmans@topic.nl> wrote:
> I have a deja-vu feeling about this question.
>
> I have this recipe:
>
> https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb
>
> Which includes this one:
> https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc
>
> I have a build server that exports its sstate-cache directory through HTTP,
> and a local host that attempts to use that sstate-cache. This works fine,
> except for the recipe above. Building this recipe takes about 1 hour, so i
> really really really want to share that state at any cost. As you can see,
> I've done a big shotgun blast of "vardepdsexclude" to get the recipe to be
> as common as possible. Still any host wants to build its own version.
>
> How can I diagnose the REASON that my machine thinks it isn't building the
> exact same thing as the build server?

see https://wiki.yoctoproject.org/wiki/Enable_sstate_cache
towards the end it talks about verifying sstate sigs

>
>
> Met vriendelijke groet / kind regards,
>
> Mike Looijmans
>
> TOPIC Embedded Systems
> Eindhovenseweg 32-C, NL-5683 KH Best
> Postbus 440, NL-5680 AK Best
> Telefoon: (+31) (0) 499 33 69 79
> Telefax:  (+31) (0) 499 33 69 70
> E-mail: mike.looijmans@topic.nl
> Website: www.topic.nl
>
> Please consider the environment before printing this e-mail
>
> Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20
> http://medteceurope.com/index.php?page=home-en
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: How to find out why shared sstate is not being used for a recipe

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

On Wed, May 28, 2014 at 1:42 PM, Khem Raj <raj.khem@gmail.com> wrote:

> On Mon, May 26, 2014 at 11:58 PM, Mike Looijmans
> <mike.looijmans@topic.nl> wrote:
> > I have a deja-vu feeling about this question.
> >
> > I have this recipe:
> >
> >
> https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb
> >
> > Which includes this one:
> >
> https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc
> >
> > I have a build server that exports its sstate-cache directory through
> HTTP,
> > and a local host that attempts to use that sstate-cache. This works fine,
> > except for the recipe above. Building this recipe takes about 1 hour, so
> i
> > really really really want to share that state at any cost. As you can
> see,
> > I've done a big shotgun blast of "vardepdsexclude" to get the recipe to
> be
> > as common as possible. Still any host wants to build its own version.
> >
> > How can I diagnose the REASON that my machine thinks it isn't building
> the
> > exact same thing as the build server?
>
> see https://wiki.yoctoproject.org/wiki/Enable_sstate_cache
> towards the end it talks about verifying sstate sigs


If the sstate is local at least, you can use bitbake -S printdiff <target>.
There's also bitbake-whatchanged, but the bitbake one is superior.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2419 bytes --]

Re: How to find out why shared sstate is not being used for a recipe

[Richard Purdie]

On Wed, 2014-05-28 at 13:46 -0700, Christopher Larson wrote:
> 
> On Wed, May 28, 2014 at 1:42 PM, Khem Raj <raj.khem@gmail.com> wrote:
>         On Mon, May 26, 2014 at 11:58 PM, Mike Looijmans
>         <mike.looijmans@topic.nl> wrote:
>         > I have a deja-vu feeling about this question.
>         >
>         > I have this recipe:
>         >
>         >
>         https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb
>         >
>         > Which includes this one:
>         >
>         https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc
>         >
>         > I have a build server that exports its sstate-cache
>         directory through HTTP,
>         > and a local host that attempts to use that sstate-cache.
>         This works fine,
>         > except for the recipe above. Building this recipe takes
>         about 1 hour, so i
>         > really really really want to share that state at any cost.
>         As you can see,
>         > I've done a big shotgun blast of "vardepdsexclude" to get
>         the recipe to be
>         > as common as possible. Still any host wants to build its own
>         version.
>         >
>         > How can I diagnose the REASON that my machine thinks it
>         isn't building the
>         > exact same thing as the build server?
>         
>         
>         see https://wiki.yoctoproject.org/wiki/Enable_sstate_cache
>         towards the end it talks about verifying sstate sigs
> 
> 
> If the sstate is local at least, you can use bitbake -S printdiff
> <target>. There's also bitbake-whatchanged, but the bitbake one is
> superior.

Worst case, you can pull the siginfo files from one build and the
siginfo files from the sstate mirror and then see which ones are
different, then run bitbake-diffsigs X Y to compare the two files.

bitbake -S just tries to automate that process if it can.

Cheers,

Richard

Re: How to find out why shared sstate is not being used for a recipe

[Mike Looijmans]

On 05/29/2014 01:12 AM, Richard Purdie wrote:
> On Wed, 2014-05-28 at 13:46 -0700, Christopher Larson wrote:
>>
>> On Wed, May 28, 2014 at 1:42 PM, Khem Raj <raj.khem@gmail.com> wrote:
>>          On Mon, May 26, 2014 at 11:58 PM, Mike Looijmans
>>          <mike.looijmans@topic.nl> wrote:
>>          > I have a deja-vu feeling about this question.
>>          >
>>          > I have this recipe:
>>          >
>>          >
>>          https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb
>>          >
>>          > Which includes this one:
>>          >
>>          https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc
>>          >
>>          > I have a build server that exports its sstate-cache
>>          directory through HTTP,
>>          > and a local host that attempts to use that sstate-cache.
>>          This works fine,
>>          > except for the recipe above. Building this recipe takes
>>          about 1 hour, so i
>>          > really really really want to share that state at any cost.
>>          As you can see,
>>          > I've done a big shotgun blast of "vardepdsexclude" to get
>>          the recipe to be
>>          > as common as possible. Still any host wants to build its own
>>          version.
>>          >
>>          > How can I diagnose the REASON that my machine thinks it
>>          isn't building the
>>          > exact same thing as the build server?
>>
>>
>>          see https://wiki.yoctoproject.org/wiki/Enable_sstate_cache
>>          towards the end it talks about verifying sstate sigs
>>
>>
>> If the sstate is local at least, you can use bitbake -S printdiff
>> <target>. There's also bitbake-whatchanged, but the bitbake one is
>> superior.

It's on two different machines, so I think that does not qualify as "local".

> Worst case, you can pull the siginfo files from one build and the
> siginfo files from the sstate mirror and then see which ones are
> different, then run bitbake-diffsigs X Y to compare the two files.

How do I find what to pull? I have (ssh) access to both machines. The 
sstate-cache dir contains a bunch of two-digit directories and a gazillion files.

I could just copy the whole thing to one machine, there's gigabit between 
them, but then what do I do with these files?


>
> bitbake -S just tries to automate that process if it can.
>

bitbake -S usually crashes here.



Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20
http://medteceurope.com/index.php?page=home-en

Re: How to find out why shared sstate is not being used for a recipe

[Mike Looijmans]

On 06/03/2014 07:25 AM, Mike Looijmans wrote:
>> Worst case, you can pull the siginfo files from one build and the
>> siginfo files from the sstate mirror and then see which ones are
>> different, then run bitbake-diffsigs X Y to compare the two files.
>
> How do I find what to pull? I have (ssh) access to both machines. The
> sstate-cache dir contains a bunch of two-digit directories and a gazillion files.
>
> I could just copy the whole thing to one machine, there's gigabit between
> them, but then what do I do with these files?

mike:~/zynq_platform/build$ find sstate-cache -name 
'*fpga-image-miami*.siginfo' | wc -l
480

Suppose I copy them. Where do I copy them to, and what do I do with these 480 
files to tell me why the system insists on rebuilding this package?

Mike.

PS: I really really want to find out. Several of these FPGA recipes take 3.5 
hours to build on the fastest i5 we could buy. So you can imagine we really 
want to prevent having to build it more than once.



Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20
http://medteceurope.com/index.php?page=home-en

Re: How to find out why shared sstate is not being used for a recipe

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3771 bytes --]

On Tue, Jun 03, 2014 at 07:25:08AM +0200, Mike Looijmans wrote:
> On 05/29/2014 01:12 AM, Richard Purdie wrote:
> > On Wed, 2014-05-28 at 13:46 -0700, Christopher Larson wrote:
> >>
> >> On Wed, May 28, 2014 at 1:42 PM, Khem Raj <raj.khem@gmail.com> wrote:
> >>          On Mon, May 26, 2014 at 11:58 PM, Mike Looijmans
> >>          <mike.looijmans@topic.nl> wrote:
> >>          > I have a deja-vu feeling about this question.
> >>          >
> >>          > I have this recipe:
> >>          >
> >>          >
> >>          https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb
> >>          >
> >>          > Which includes this one:
> >>          >
> >>          https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc
> >>          >
> >>          > I have a build server that exports its sstate-cache
> >>          directory through HTTP,
> >>          > and a local host that attempts to use that sstate-cache.
> >>          This works fine,
> >>          > except for the recipe above. Building this recipe takes
> >>          about 1 hour, so i
> >>          > really really really want to share that state at any cost.
> >>          As you can see,
> >>          > I've done a big shotgun blast of "vardepdsexclude" to get
> >>          the recipe to be
> >>          > as common as possible. Still any host wants to build its own
> >>          version.
> >>          >
> >>          > How can I diagnose the REASON that my machine thinks it
> >>          isn't building the
> >>          > exact same thing as the build server?
> >>
> >>
> >>          see https://wiki.yoctoproject.org/wiki/Enable_sstate_cache
> >>          towards the end it talks about verifying sstate sigs
> >>
> >>
> >> If the sstate is local at least, you can use bitbake -S printdiff
> >> <target>. There's also bitbake-whatchanged, but the bitbake one is
> >> superior.
> 
> It's on two different machines, so I think that does not qualify as "local".
> 
> > Worst case, you can pull the siginfo files from one build and the
> > siginfo files from the sstate mirror and then see which ones are
> > different, then run bitbake-diffsigs X Y to compare the two files.
> 
> How do I find what to pull? I have (ssh) access to both machines. The 
> sstate-cache dir contains a bunch of two-digit directories and a gazillion files.
> 
> I could just copy the whole thing to one machine, there's gigabit between 
> them, but then what do I do with these files?

You can also use openembedded-core/scripts/sstate-diff-machines.sh
script to create just .sigdata files on both machines and then copy just
this sstate-diff directory, see header of that script for very short
readme.

> > bitbake -S just tries to automate that process if it can.
> >
> 
> bitbake -S usually crashes here.

Are you using latest bitbake? There was fix for that in newer bitbake already/

> Met vriendelijke groet / kind regards,
> 
> Mike Looijmans
> 
> TOPIC Embedded Systems
> Eindhovenseweg 32-C, NL-5683 KH Best
> Postbus 440, NL-5680 AK Best
> Telefoon: (+31) (0) 499 33 69 79
> Telefax:  (+31) (0) 499 33 69 70
> E-mail: mike.looijmans@topic.nl
> Website: www.topic.nl
> 
> Please consider the environment before printing this e-mail
> 
> Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20
> http://medteceurope.com/index.php?page=home-en
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: How to find out why shared sstate is not being used for a recipe

[Richard Purdie]

On Tue, 2014-06-03 at 07:35 +0200, Mike Looijmans wrote:
> On 06/03/2014 07:25 AM, Mike Looijmans wrote:
> >> Worst case, you can pull the siginfo files from one build and the
> >> siginfo files from the sstate mirror and then see which ones are
> >> different, then run bitbake-diffsigs X Y to compare the two files.
> >
> > How do I find what to pull? I have (ssh) access to both machines. The
> > sstate-cache dir contains a bunch of two-digit directories and a gazillion files.
> >
> > I could just copy the whole thing to one machine, there's gigabit between
> > them, but then what do I do with these files?
> 
> mike:~/zynq_platform/build$ find sstate-cache -name 
> '*fpga-image-miami*.siginfo' | wc -l
> 480
> 
> Suppose I copy them. Where do I copy them to, and what do I do with these 480 
> files to tell me why the system insists on rebuilding this package?
> 
> Mike.
> 
> PS: I really really want to find out. Several of these FPGA recipes take 3.5 
> hours to build on the fastest i5 we could buy. So you can imagine we really 
> want to prevent having to build it more than once.

Let me explain the manual process you can follow for this. Its a pain to
walk through and its what -S attempts to automate but you should be able
to get an answer manually this way.

You have BUILDA and BUILDB, the two builds which should be reusing
sstate but are not. The fact they're on different machines is irrelevant
to this procedure. It would help if these two builds are just the result
of "bitbake fpga-image-miami" but that isn't essential, it will just
introduce more noise. It will also help if they are either both built
from an existing sstate cache or both not.

The first thing I'd do in each build:

"find tmp/stamps/ -type f | sort > stamplistA.txt"

I'd then so something like:

"meld stamplistA stamplistB"

the next steps depend upon how clear the differences are. Basically
there should be some degree of commonality between the two builds and at
some point there will be divergence. We need to pinpoint the point of
divergence. The divergence will be in fpga-image-miami itself or in one
of its dependencies.

The one thing which can confuse this view is if some things were reused
from sstate. You can tell since a task which runs looks like:

tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86

and a task which is from sstate looks like:

tmp/stamps/all-poky-linux/gstreamer1.0-meta-base/1.0-r0.do_populate_sysroot_setscene.a49fa811727c557c14ab6ce6f2973587.qemux86

The "_setscene" part tells you this. Machine specific tasks have the
machine name appended to them ".qemux86" in this case. The hash
representing the task is clear in the filename
("a49fa811727c557c14ab6ce6f2973587" in this case").

You'll have to filter out any "noise" from these changes you're not
interested in. If a task is "_setscene" its dependencies may be missing
from the list of files entirely (no install/compile/configure etc.).

So you take a guess at the divergence point and take note of the two
different hashes. You can then find the corresponding siginfo files in
sstate:

find sstate-cache/ -name *d90d4404368125acd109a2ac64ca688f* 
sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz
sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz.siginfo

which are the sstate files corresponding to my above x11-common task.
You will note that the first two letters of the hash are used as a
directory prefix. You can also find sigdata files in the stamp
directory:

$ find tmp/stamps/ -type f | grep d90d440
tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86
tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.sigdata.d90d4404368125acd109a2ac64ca688f

The sigdata and siginfo files are identical and equivalent. Once you
have the two sstate files, you can run:

bitbake-diffsigs A.siginfo B.siginfo

and this will tell you why their hashes are different. If you need help
deciphering that output, link to it and I can help.

If you didn't guess the divergence point correctly, it will tell you
that some prior task is actually different. In that case you would then
go and fetch the siginfo tasks for the previous task and compare those.
Either you find the difference or again, you have to trace it further
back.

Its a pain of a process to go through but it is deterministic and you
will eventually find the difference. Have a go and if you have any
issues let me know and I'll do what I can to help.

Cheers,

Richard

Re: How to find out why shared sstate is not being used for a recipe

[Richard Purdie]

On Tue, 2014-06-03 at 07:25 +0200, Mike Looijmans wrote:
> On 05/29/2014 01:12 AM, Richard Purdie wrote:
> >
> > bitbake -S just tries to automate that process if it can.
> >
> 
> bitbake -S usually crashes here.

Is there a bug report for that? It shouldn't crash...

Cheers,

Richard

Re: How to find out why shared sstate is not being used for a recipe

[Mike Looijmans]

On 06/03/2014 10:45 AM, Richard Purdie wrote:
> On Tue, 2014-06-03 at 07:35 +0200, Mike Looijmans wrote:
>> On 06/03/2014 07:25 AM, Mike Looijmans wrote:
>>>> Worst case, you can pull the siginfo files from one build and the
>>>> siginfo files from the sstate mirror and then see which ones are
>>>> different, then run bitbake-diffsigs X Y to compare the two files.
>>>
>>> How do I find what to pull? I have (ssh) access to both machines. The
>>> sstate-cache dir contains a bunch of two-digit directories and a gazillion files.
>>>
>>> I could just copy the whole thing to one machine, there's gigabit between
>>> them, but then what do I do with these files?
>>
>> mike:~/zynq_platform/build$ find sstate-cache -name
>> '*fpga-image-miami*.siginfo' | wc -l
>> 480
>>
>> Suppose I copy them. Where do I copy them to, and what do I do with these 480
>> files to tell me why the system insists on rebuilding this package?
>>
>> Mike.
>>
>> PS: I really really want to find out. Several of these FPGA recipes take 3.5
>> hours to build on the fastest i5 we could buy. So you can imagine we really
>> want to prevent having to build it more than once.
>
> Let me explain the manual process you can follow for this. Its a pain to
> walk through and its what -S attempts to automate but you should be able
> to get an answer manually this way.
>
> You have BUILDA and BUILDB, the two builds which should be reusing
> sstate but are not. The fact they're on different machines is irrelevant
> to this procedure. It would help if these two builds are just the result
> of "bitbake fpga-image-miami" but that isn't essential, it will just
> introduce more noise. It will also help if they are either both built
> from an existing sstate cache or both not.
>
> The first thing I'd do in each build:
>
> "find tmp/stamps/ -type f | sort > stamplistA.txt"
>
> I'd then so something like:
>
> "meld stamplistA stamplistB"
>
> the next steps depend upon how clear the differences are. Basically
> there should be some degree of commonality between the two builds and at
> some point there will be divergence. We need to pinpoint the point of
> divergence. The divergence will be in fpga-image-miami itself or in one
> of its dependencies.

ALL stamps for ALL tasks for fpga-image-miami are different. There's a few 
thousand lines in these files, with very few similarities.

> The one thing which can confuse this view is if some things were reused
> from sstate. You can tell since a task which runs looks like:
>
> tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86
>
> and a task which is from sstate looks like:
>
> tmp/stamps/all-poky-linux/gstreamer1.0-meta-base/1.0-r0.do_populate_sysroot_setscene.a49fa811727c557c14ab6ce6f2973587.qemux86
>
> The "_setscene" part tells you this. Machine specific tasks have the
> machine name appended to them ".qemux86" in this case. The hash
> representing the task is clear in the filename
> ("a49fa811727c557c14ab6ce6f2973587" in this case").
>
> You'll have to filter out any "noise" from these changes you're not
> interested in. If a task is "_setscene" its dependencies may be missing
> from the list of files entirely (no install/compile/configure etc.).
>
> So you take a guess at the divergence point and take note of the two
> different hashes. You can then find the corresponding siginfo files in
> sstate:
>
> find sstate-cache/ -name *d90d4404368125acd109a2ac64ca688f*
> sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz
 
sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz.siginfo

I'll just go for the "do_fetch" task then, since that's about the first to 
execute.

> which are the sstate files corresponding to my above x11-common task.
> You will note that the first two letters of the hash are used as a
> directory prefix. You can also find sigdata files in the stamp
> directory:
>
> $ find tmp/stamps/ -type f | grep d90d440
> tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86
> tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.sigdata.d90d4404368125acd109a2ac64ca688f
>
> The sigdata and siginfo files are identical and equivalent. Once you
> have the two sstate files, you can run:
>
> bitbake-diffsigs A.siginfo B.siginfo
>
> and this will tell you why their hashes are different. If you need help
> deciphering that output, link to it and I can help.
>
> If you didn't guess the divergence point correctly, it will tell you
> that some prior task is actually different. In that case you would then
> go and fetch the siginfo tasks for the previous task and compare those.
> Either you find the difference or again, you have to trace it further
> back.
>
> Its a pain of a process to go through but it is deterministic and you
> will eventually find the difference. Have a go and if you have any
> issues let me know and I'll do what I can to help.
>

Thank you very much!

So far, on one project (that is, NOT the fpga-image-miami) the SRC_URI is 
slightly different. The machines are on various sides of a router/firewall, so 
SRC_URI="git://${MY_SERVER}/blabla" evaluates differently.

How can I tell the system that the value of "MY_SERVER" is irrelevant to each 
and every build step in each and every recipe?


I'm still digging into the fpga recipes, but that's rather slow because each 
build takes about an hour, and that's for the "fast" ones.

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20
http://medteceurope.com/index.php?page=home-en

Re: How to find out why shared sstate is not being used for a recipe

[Richard Purdie]

On Tue, 2014-06-03 at 15:54 +0200, Mike Looijmans wrote:
> On 06/03/2014 10:45 AM, Richard Purdie wrote:
> > the next steps depend upon how clear the differences are. Basically
> > there should be some degree of commonality between the two builds and at
> > some point there will be divergence. We need to pinpoint the point of
> > divergence. The divergence will be in fpga-image-miami itself or in one
> > of its dependencies.
> 
> ALL stamps for ALL tasks for fpga-image-miami are different. There's a few 
> thousand lines in these files, with very few similarities.

Ok, this gives a very strong pointer then, see below.

> I'll just go for the "do_fetch" task then, since that's about the first to 
> execute.

This is a good move.

> So far, on one project (that is, NOT the fpga-image-miami) the SRC_URI is 
> slightly different. The machines are on various sides of a router/firewall, so 
> SRC_URI="git://${MY_SERVER}/blabla" evaluates differently.
> 
> How can I tell the system that the value of "MY_SERVER" is irrelevant to each 
> and every build step in each and every recipe?

The SRC_URI changing would indeed cause the tasks build separately since
bitbake thinks they're different things.

You have two ways of addressing this. Firstly, you could setup PREMIRROR
entries for these git urls which remaps them to the correct thing in
each case. There would then be one "canonical" url in the recipe and
you'd not need to change the hash generation.

The other way would be to either exclude the variable from the checksum
generation or give it a specific value. This would be something like:

do_fetch[vardepsexclude] += "SRC_URI"

you may also have to do this with other tasks that use SRC_URI such as
do_unpack and do_patch. The other option is:

SRC_URI[vardepvalue] = "git://canonical/url/for/recipe"

which gives it a specific value to use for checksum purposes.

Cheers,

Richard

Re: How to find out why shared sstate is not being used for a recipe

[Mike Looijmans]

On 06/03/2014 04:10 PM, Richard Purdie wrote:
> On Tue, 2014-06-03 at 15:54 +0200, Mike Looijmans wrote:
>> On 06/03/2014 10:45 AM, Richard Purdie wrote:
>>> the next steps depend upon how clear the differences are. Basically
>>> there should be some degree of commonality between the two builds and at
>>> some point there will be divergence. We need to pinpoint the point of
>>> divergence. The divergence will be in fpga-image-miami itself or in one
>>> of its dependencies.
>>
>> ALL stamps for ALL tasks for fpga-image-miami are different. There's a few
>> thousand lines in these files, with very few similarities.
>
> Ok, this gives a very strong pointer then, see below.
>
>> I'll just go for the "do_fetch" task then, since that's about the first to
>> execute.
>
> This is a good move.
>
>> So far, on one project (that is, NOT the fpga-image-miami) the SRC_URI is
>> slightly different. The machines are on various sides of a router/firewall, so
>> SRC_URI="git://${MY_SERVER}/blabla" evaluates differently.
>>
>> How can I tell the system that the value of "MY_SERVER" is irrelevant to each
>> and every build step in each and every recipe?
>
> The SRC_URI changing would indeed cause the tasks build separately since
> bitbake thinks they're different things.
>
> You have two ways of addressing this. Firstly, you could setup PREMIRROR
> entries for these git urls which remaps them to the correct thing in
> each case. There would then be one "canonical" url in the recipe and
> you'd not need to change the hash generation.
>
> The other way would be to either exclude the variable from the checksum
> generation or give it a specific value. This would be something like:
>
> do_fetch[vardepsexclude] += "SRC_URI"
>
> you may also have to do this with other tasks that use SRC_URI such as
> do_unpack and do_patch. The other option is:
>
> SRC_URI[vardepvalue] = "git://canonical/url/for/recipe"
>
> which gives it a specific value to use for checksum purposes.

In this case, a very useful syntax. The machines have different MY_SERVER 
values in their local.conf, so I've changed one into:

MY_SERVER = "git://some/local/path"
MY_SERVER[vardepvalue] = "git://common.server/path"

This generates the same SRC_URI signatures for the locations that look 
different but actually refer to the same location.

The PREMIRROR sounds like a generic solution to the underlying problem of 
accessing servers from different locations. I'll keep that in mind for the 
next project.

Next, I'll need to investigate why the fpga-image-miami generates different 
signatures, as both machines fetch it from the same URI.

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20
http://medteceurope.com/index.php?page=home-en

Re: How to find out why shared sstate is not being used for a recipe

[Mike Looijmans]

On 06/03/2014 10:45 AM, Richard Purdie wrote:
> On Tue, 2014-06-03 at 07:35 +0200, Mike Looijmans wrote:
>> On 06/03/2014 07:25 AM, Mike Looijmans wrote:
>>>> Worst case, you can pull the siginfo files from one build and the
>>>> siginfo files from the sstate mirror and then see which ones are
>>>> different, then run bitbake-diffsigs X Y to compare the two files.
>>>
>>> How do I find what to pull? I have (ssh) access to both machines. The
>>> sstate-cache dir contains a bunch of two-digit directories and a gazillion files.
>>>
>>> I could just copy the whole thing to one machine, there's gigabit between
>>> them, but then what do I do with these files?
>>
>> mike:~/zynq_platform/build$ find sstate-cache -name
>> '*fpga-image-miami*.siginfo' | wc -l
>> 480
>>
>> Suppose I copy them. Where do I copy them to, and what do I do with these 480
>> files to tell me why the system insists on rebuilding this package?
>>
>> Mike.
>>
>> PS: I really really want to find out. Several of these FPGA recipes take 3.5
>> hours to build on the fastest i5 we could buy. So you can imagine we really
>> want to prevent having to build it more than once.
>
> Let me explain the manual process you can follow for this. Its a pain to
> walk through and its what -S attempts to automate but you should be able
> to get an answer manually this way.
>
> You have BUILDA and BUILDB, the two builds which should be reusing
> sstate but are not. The fact they're on different machines is irrelevant
> to this procedure. It would help if these two builds are just the result
> of "bitbake fpga-image-miami" but that isn't essential, it will just
> introduce more noise. It will also help if they are either both built
> from an existing sstate cache or both not.
>
> The first thing I'd do in each build:
>
> "find tmp/stamps/ -type f | sort > stamplistA.txt"
>
> I'd then so something like:
>
> "meld stamplistA stamplistB"
>
> the next steps depend upon how clear the differences are. Basically
> there should be some degree of commonality between the two builds and at
> some point there will be divergence. We need to pinpoint the point of
> divergence. The divergence will be in fpga-image-miami itself or in one
> of its dependencies.
>
> The one thing which can confuse this view is if some things were reused
> from sstate. You can tell since a task which runs looks like:
>
> tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86
>
> and a task which is from sstate looks like:
>
> tmp/stamps/all-poky-linux/gstreamer1.0-meta-base/1.0-r0.do_populate_sysroot_setscene.a49fa811727c557c14ab6ce6f2973587.qemux86
>
> The "_setscene" part tells you this. Machine specific tasks have the
> machine name appended to them ".qemux86" in this case. The hash
> representing the task is clear in the filename
> ("a49fa811727c557c14ab6ce6f2973587" in this case").
>
> You'll have to filter out any "noise" from these changes you're not
> interested in. If a task is "_setscene" its dependencies may be missing
> from the list of files entirely (no install/compile/configure etc.).
>
> So you take a guess at the divergence point and take note of the two
> different hashes. You can then find the corresponding siginfo files in
> sstate:
>
> find sstate-cache/ -name *d90d4404368125acd109a2ac64ca688f*
> sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz
> sstate-cache/d9/sstate:x11-common:i586-poky-linux:0.1:r47:i586:3:d90d4404368125acd109a2ac64ca688f_populate_sysroot.tgz.siginfo
>
> which are the sstate files corresponding to my above x11-common task.
> You will note that the first two letters of the hash are used as a
> directory prefix. You can also find sigdata files in the stamp
> directory:
>
> $ find tmp/stamps/ -type f | grep d90d440
> tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.d90d4404368125acd109a2ac64ca688f.qemux86
> tmp/stamps/i586-poky-linux/x11-common/0.1-r47.do_populate_sysroot.sigdata.d90d4404368125acd109a2ac64ca688f
>
> The sigdata and siginfo files are identical and equivalent. Once you
> have the two sstate files, you can run:
>
> bitbake-diffsigs A.siginfo B.siginfo
>
> and this will tell you why their hashes are different. If you need help
> deciphering that output, link to it and I can help.

Dependency on task fpga-image-medical-demo.bb.do_patch was added with hash 
8fbb54a572a402e28d05daae0a238f8b
Dependency on task fpga-image-medical-demo.bb.do_patch was removed with hash 
44a252a24b20321e8dbec6d9d8b54c67

Now this means that I have to look at the "unpack" task now, right?

The weird thing is, both machines have identical do_patch hashes:
tmp-eglibc/stamps/zynq_miami_7030-oe-linux-gnueabi/fpga-image-medical-demo/0.AUTOINC-06349d0386-r0.do_patch.sigdata.44a252a24b20321e8dbec6d9d8b54c67

I cannot find that "8fbb54a572a402e28d05daae0a238f8b" hash anywhere.





Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Visit us at MEDTEC Europe 3-5 June, Messe Stuttgart, stand number 5D20
http://medteceurope.com/index.php?page=home-en

Re: How to find out why shared sstate is not being used for a recipe

[Mike Looijmans]

On 05/27/2014 08:58 AM, Mike Looijmans wrote:
> I have a deja-vu feeling about this question.
>
> I have this recipe:
>
> https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image-miami.bb
>
>
> Which includes this one:
> https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/fpga/fpga-image.inc
>
>
> I have a build server that exports its sstate-cache directory through HTTP,
> and a local host that attempts to use that sstate-cache. This works fine,
> except for the recipe above. Building this recipe takes about 1 hour, so i
> really really really want to share that state at any cost. As you can see,
> I've done a big shotgun blast of "vardepdsexclude" to get the recipe to be as
> common as possible. Still any host wants to build its own version.
>

I recently figured out the cause of this:

LIC_FILES_CHKSUM = 
"file://${META_ZYNQ_BASE}/COPYING;md5=751419260aa954499f7abaabaa882bbe"

The META_ZYNQ_BASE variable is set to the overlay path in the layer.conf file. 
This path varies on different machines.

I added the following line to layer.conf to prevent this from happening:
META_ZYNQ_BASE[vardepvalue] = "meta-zynq"

This triggered a rebuild of all pacakges that I've seen to have this sstate 
problem, thereby confirming that this really was the cause.

It DID surprise me that the LIC_FILES_CHKSUM full path filename somehow ended 
up in the hash for the package's contents. It would have been better to only 
include the license contents itself (or its checksum).

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Bezoek ons op 9 en 10 september tijdens Technology for Health Den Bosch (stand 53)
http://www.technologyforhealth.nl