sstate management

sstate management

[Gary Thomas]

I'm looking into using sstate more and in particular sharing
it amongst a number of builds.  I have a couple of questions
which I didn't find much info about.

* The sstate-cache for a given build/target seems to grow
   without bounds.  I have one build which I've been reusing
   since last November has grown to 62GB.  A very similar
   build which hasn't see quite so many 'bakes' is only 27GB.

   Is there some maintenance to be done on the sstate-cache?
   I'm thinking I want to set up a shared cache which might
   last for a long time and I would like to only keep the bits
   that are really needed.

* The second operational question I have is if I have a shared
   sstate cache and I make some sort of build, what is the best
   way (if any) to share any newly created objects so that my
   other builds can make use of them?

I've not actually tried to share any caches yet, so these
questions are just based on my rough understanding of the
use of sstate.  Please feel free to correct me if I've got
it [totally] wrong.

Thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate management

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1263 bytes --]

On Mon, Feb 2, 2015 at 9:42 AM, Gary Thomas <gary@mlbassoc.com> wrote:

> * The sstate-cache for a given build/target seems to grow
>   without bounds.  I have one build which I've been reusing
>   since last November has grown to 62GB.  A very similar
>   build which hasn't see quite so many 'bakes' is only 27GB.
>
>   Is there some maintenance to be done on the sstate-cache?
>   I'm thinking I want to set up a shared cache which might
>   last for a long time and I would like to only keep the bits
>   that are really needed.
>

In the past i've either used sstate-cache-management.sh or ensured that
SSTATE_DIR is on a mount with atime enabled and just periodically wiped
anything that hasn't been accessed in over a week.

* The second operational question I have is if I have a shared
>   sstate cache and I make some sort of build, what is the best
>   way (if any) to share any newly created objects so that my
>   other builds can make use of them?
>

I doubt there's a "best" on that. Personally I either use a shared
filesystem path (e.g. nfs) or hook up rsync.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1932 bytes --]

Re: sstate management

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 711 bytes --]

On 2 February 2015 at 16:48, Christopher Larson <clarson@kergoth.com> wrote:

>   Is there some maintenance to be done on the sstate-cache?
>>   I'm thinking I want to set up a shared cache which might
>>   last for a long time and I would like to only keep the bits
>>   that are really needed.
>>
>
> In the past i've either used sstate-cache-management.sh or ensured that
> SSTATE_DIR is on a mount with atime enabled and just periodically wiped
> anything that hasn't been accessed in over a week.
>

Seconded on this - after doing lots of builds in the last few weeks
involving five machines and new eglibc/gcc patches my sstate was 1.2TB.  A
quick find -atime -delete did the job.

Ross

[-- Attachment #2: Type: text/html, Size: 1241 bytes --]

Re: sstate management

[Paul Eggleton]

On Monday 02 February 2015 17:33:23 Burton, Ross wrote:
> On 2 February 2015 at 16:48, Christopher Larson <clarson@kergoth.com> wrote:
> >   Is there some maintenance to be done on the sstate-cache?
> >   
> >>   I'm thinking I want to set up a shared cache which might
> >>   last for a long time and I would like to only keep the bits
> >>   that are really needed.
> > 
> > In the past i've either used sstate-cache-management.sh or ensured that
> > SSTATE_DIR is on a mount with atime enabled and just periodically wiped
> > anything that hasn't been accessed in over a week.
> 
> Seconded on this - after doing lots of builds in the last few weeks
> involving five machines and new eglibc/gcc patches my sstate was 1.2TB.  A
> quick find -atime -delete did the job.

I don't suppose I can talk one or more of you into writing up some 
documentation for this to add to the manual? (As usual, something raw in the 
form of a wiki page that Scott can then adapt would be ideal.)

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: sstate management

[Rifenbark, Scott M]

You might want to quickly look through http://www.yoctoproject.org/docs/1.8/ref-manual/ref-manual.html#shared-state-cache in the ref-manual before supplying raw documentation information.  This section is our current discourse on sstate.

Scott

>-----Original Message-----
>From: yocto-bounces@yoctoproject.org [mailto:yocto-
>bounces@yoctoproject.org] On Behalf Of Paul Eggleton
>Sent: Monday, February 02, 2015 9:43 AM
>To: Burton, Ross; Christopher Larson; Gary Thomas
>Cc: yocto@yoctoproject.org
>Subject: Re: [yocto] sstate management
>
>On Monday 02 February 2015 17:33:23 Burton, Ross wrote:
>> On 2 February 2015 at 16:48, Christopher Larson <clarson@kergoth.com>
>wrote:
>> >   Is there some maintenance to be done on the sstate-cache?
>> >
>> >>   I'm thinking I want to set up a shared cache which might
>> >>   last for a long time and I would like to only keep the bits
>> >>   that are really needed.
>> >
>> > In the past i've either used sstate-cache-management.sh or ensured
>> > that SSTATE_DIR is on a mount with atime enabled and just
>> > periodically wiped anything that hasn't been accessed in over a week.
>>
>> Seconded on this - after doing lots of builds in the last few weeks
>> involving five machines and new eglibc/gcc patches my sstate was
>> 1.2TB.  A quick find -atime -delete did the job.
>
>I don't suppose I can talk one or more of you into writing up some
>documentation for this to add to the manual? (As usual, something raw in the
>form of a wiki page that Scott can then adapt would be ideal.)
>
>Cheers,
>Paul
>
>--
>
>Paul Eggleton
>Intel Open Source Technology Centre
>--
>_______________________________________________
>yocto mailing list
>yocto@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/yocto