Little sstate reuse when using uninative and yocto sstate cache

Little sstate reuse when using uninative and yocto sstate cache

[Jacob Kroon]

Hi,

I'm on latest dunfell and using uninative and have put the yocto servers 
in my SSTATE_MIRRORS:

my-distro.conf:
...
> SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/3.1.11/PATH;downloadfilename=PATH"
> require conf/distro/include/yocto-uninative.inc
> INHERIT += "uninative"
...

but I thought I'd be getting more sstate reuse than I am:

> bitbake m4-native
...
> Sstate summary: Wanted 12 Found 1 Missed 11 Current 0 (8% match, 0% complete)

Is the above reasonable, or should I be able to pull m4-native directly 
from yocto servers sstate ? If so, is there a way to debug what differs 
between my local siginfos and the ones on the sstate server ?

Regards Jacob

Re: [OE-core] Little sstate reuse when using uninative and yocto sstate cache

[Richard Purdie]

On Mon, 2021-10-18 at 09:16 +0200, Jacob Kroon wrote:
> I'm on latest dunfell and using uninative and have put the yocto servers 
> in my SSTATE_MIRRORS:
> 
> my-distro.conf:
> ...
> > SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/3.1.11/PATH;downloadfilename=PATH"
> > require conf/distro/include/yocto-uninative.inc
> > INHERIT += "uninative"
> ...
> 
> but I thought I'd be getting more sstate reuse than I am:
> 
> > bitbake m4-native
> ...
> > Sstate summary: Wanted 12 Found 1 Missed 11 Current 0 (8% match, 0% complete)
> 
> Is the above reasonable, or should I be able to pull m4-native directly 
> from yocto servers sstate ? If so, is there a way to debug what differs 
> between my local siginfos and the ones on the sstate server ?


In dunfell hash equivalence is enabled and this unfortunately means the sstate
doesn't work very well unless you also have the matching hash equivalence data.

We do now have a way to have a readonly hash equivalence server and we have made
one available from the autobuilder so this data is shared. Unfortunately there
isn't a way to configure dunfell to use one though :(. That patch is a feature
but could be something we consider backporting.

That said, I did a lot of testing/work with master recently as sstate reuse with
hash equivalence was not what we'd expected. We ended up making a lot of
changes, including a bit of a rewrite of the server side of the hash
equivalence. I mention this just to illustrate it isn't a simple problem and
that fixing this in dunfell may be tricky and would need backporting of some
invasive changes. I'm trying to at least get master/honister working right in
the first instance.

Cheers,

Richard

Re: [OE-core] Little sstate reuse when using uninative and yocto sstate cache

[Jacob Kroon]

On 10/18/21 12:15, Richard Purdie wrote:
> On Mon, 2021-10-18 at 09:16 +0200, Jacob Kroon wrote:
>> I'm on latest dunfell and using uninative and have put the yocto servers
>> in my SSTATE_MIRRORS:
>>
>> my-distro.conf:
>> ...
>>> SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/3.1.11/PATH;downloadfilename=PATH"
>>> require conf/distro/include/yocto-uninative.inc
>>> INHERIT += "uninative"
>> ...
>>
>> but I thought I'd be getting more sstate reuse than I am:
>>
>>> bitbake m4-native
>> ...
>>> Sstate summary: Wanted 12 Found 1 Missed 11 Current 0 (8% match, 0% complete)
>>
>> Is the above reasonable, or should I be able to pull m4-native directly
>> from yocto servers sstate ? If so, is there a way to debug what differs
>> between my local siginfos and the ones on the sstate server ?
> 
> 
> In dunfell hash equivalence is enabled and this unfortunately means the sstate
> doesn't work very well unless you also have the matching hash equivalence data.
> 
> We do now have a way to have a readonly hash equivalence server and we have made
> one available from the autobuilder so this data is shared. Unfortunately there
> isn't a way to configure dunfell to use one though :(. That patch is a feature
> but could be something we consider backporting.
> 
> That said, I did a lot of testing/work with master recently as sstate reuse with
> hash equivalence was not what we'd expected. We ended up making a lot of
> changes, including a bit of a rewrite of the server side of the hash
> equivalence. I mention this just to illustrate it isn't a simple problem and
> that fixing this in dunfell may be tricky and would need backporting of some
> invasive changes. I'm trying to at least get master/honister working right in
> the first instance.
> 
> Cheers,
> 
> Richard
> 
> 

Thanks for the detailed answer. I will focus on getting this to perform 
well with my master build then.

Regards
Jacob