[PATCH] dev-manual: how to purge duplicate sstate cache files

[PATCH] dev-manual: how to purge duplicate sstate cache files

[Michael Opdenacker]

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
---
 documentation/dev-manual/common-tasks.rst | 25 ++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
index 3eead147a3..37612c84b7 100644
--- a/documentation/dev-manual/common-tasks.rst
+++ b/documentation/dev-manual/common-tasks.rst
@@ -6242,8 +6242,11 @@ Changing the listed common targets is as easy as editing your version of
 ``conf-notes.txt`` in your custom template configuration directory and
 making sure you have ``TEMPLATECONF`` set to your directory.
 
+Conserving Disk Space
+=====================
+
 Conserving Disk Space During Builds
-===================================
+-----------------------------------
 
 To help conserve disk space during builds, you can add the following
 statement to your project's ``local.conf`` configuration file found in
@@ -6257,6 +6260,26 @@ building a recipe once the recipe is built. For more information on
 :ref:`rm_work <ref-classes-rm-work>` class in the
 Yocto Project Reference Manual.
 
+Purging Duplicate Shared State Cache Files
+-------------------------------------------
+
+After multiple build iterations, the Shared State (sstate) cache can contain
+duplicate cache files for a given package, while only the most recent one
+is likely to be reusable. The following command purges all but the
+newest sstate cache file for each package::
+
+   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
+
+This command will ask you to confirm the deletions it identifies.
+
+Note::
+
+   The duplicated sstate cache files of one package must have the same
+   architecture, which means that sstate cache files with multiple
+   architectures are not considered as duplicate.
+
+Run ``sstate-cache-management.sh`` for more details about this script.
+
 Working with Packages
 =====================
 
-- 
2.25.1

Re: [docs] [PATCH] dev-manual: how to purge duplicate sstate cache files

[Michael Opdenacker]

Greetings,

On 11/19/21 5:06 PM, Michael Opdenacker wrote:
> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> ---
>  documentation/dev-manual/common-tasks.rst | 25 ++++++++++++++++++++++-
>  1 file changed, 24 insertions(+), 1 deletion(-)
>
> diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
> index 3eead147a3..37612c84b7 100644
> --- a/documentation/dev-manual/common-tasks.rst
> +++ b/documentation/dev-manual/common-tasks.rst
> @@ -6242,8 +6242,11 @@ Changing the listed common targets is as easy as editing your version of
>  ``conf-notes.txt`` in your custom template configuration directory and
>  making sure you have ``TEMPLATECONF`` set to your directory.
>  
> +Conserving Disk Space
> +=====================
> +
>  Conserving Disk Space During Builds
> -===================================
> +-----------------------------------
>  
>  To help conserve disk space during builds, you can add the following
>  statement to your project's ``local.conf`` configuration file found in
> @@ -6257,6 +6260,26 @@ building a recipe once the recipe is built. For more information on
>  :ref:`rm_work <ref-classes-rm-work>` class in the
>  Yocto Project Reference Manual.
>  
> +Purging Duplicate Shared State Cache Files
> +-------------------------------------------
> +
> +After multiple build iterations, the Shared State (sstate) cache can contain
> +duplicate cache files for a given package, while only the most recent one
> +is likely to be reusable. The following command purges all but the
> +newest sstate cache file for each package::
> +
> +   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
> +
> +This command will ask you to confirm the deletions it identifies.
> +
> +Note::
> +
> +   The duplicated sstate cache files of one package must have the same
> +   architecture, which means that sstate cache files with multiple
> +   architectures are not considered as duplicate.
> +
> +Run ``sstate-cache-management.sh`` for more details about this script.
> +
>  Working with Packages
>  =====================
>  

Did anyone have time to have a look at what I wrote ? There may be
better ways to purge the sstate cache files.
Thanks in advance
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH] dev-manual: how to purge duplicate sstate cache files

[Vyacheslav Yurkov]

Hi Michael

On 23.11.2021 10:14, Michael Opdenacker wrote:
> Greetings,
>
> On 11/19/21 5:06 PM, Michael Opdenacker wrote:
>> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
>> ---
>>   documentation/dev-manual/common-tasks.rst | 25 ++++++++++++++++++++++-
>>   1 file changed, 24 insertions(+), 1 deletion(-)
>>
>> diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
>> index 3eead147a3..37612c84b7 100644
>> --- a/documentation/dev-manual/common-tasks.rst
>> +++ b/documentation/dev-manual/common-tasks.rst
>> @@ -6242,8 +6242,11 @@ Changing the listed common targets is as easy as editing your version of
>>   ``conf-notes.txt`` in your custom template configuration directory and
>>   making sure you have ``TEMPLATECONF`` set to your directory.
>>   
>> +Conserving Disk Space
>> +=====================
>> +
>>   Conserving Disk Space During Builds
>> -===================================
>> +-----------------------------------
>>   
>>   To help conserve disk space during builds, you can add the following
>>   statement to your project's ``local.conf`` configuration file found in
>> @@ -6257,6 +6260,26 @@ building a recipe once the recipe is built. For more information on
>>   :ref:`rm_work <ref-classes-rm-work>` class in the
>>   Yocto Project Reference Manual.
>>   
>> +Purging Duplicate Shared State Cache Files
>> +-------------------------------------------
>> +
>> +After multiple build iterations, the Shared State (sstate) cache can contain
>> +duplicate cache files for a given package, while only the most recent one
>> +is likely to be reusable. The following command purges all but the
>> +newest sstate cache file for each package::
>> +
>> +   sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
>> +
>> +This command will ask you to confirm the deletions it identifies.
>> +
>> +Note::
>> +
>> +   The duplicated sstate cache files of one package must have the same
>> +   architecture, which means that sstate cache files with multiple
>> +   architectures are not considered as duplicate.
>> +
>> +Run ``sstate-cache-management.sh`` for more details about this script.
>> +
>>   Working with Packages
>>   =====================
>>   
> Did anyone have time to have a look at what I wrote ? There may be
> better ways to purge the sstate cache files.
> Thanks in advance
> Michael.

The description looks good to me. By other ways you mean other usage 
scenarios of sstate-cache-management.sh script or by means of something 
else?

Vyacheslav

Re: [OE-core] [docs] [PATCH] dev-manual: how to purge duplicate sstate cache files

[Jacob Kroon]

Hi Michael,

On 11/26/21 10:13, Vyacheslav Yurkov wrote:
> Hi Michael
> 
> On 23.11.2021 10:14, Michael Opdenacker wrote:
>> Greetings,
>>
>> On 11/19/21 5:06 PM, Michael Opdenacker wrote:
>>> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
>>> ---
>>>   documentation/dev-manual/common-tasks.rst | 25 ++++++++++++++++++++++-
>>>   1 file changed, 24 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/documentation/dev-manual/common-tasks.rst
>>> b/documentation/dev-manual/common-tasks.rst
>>> index 3eead147a3..37612c84b7 100644
>>> --- a/documentation/dev-manual/common-tasks.rst
>>> +++ b/documentation/dev-manual/common-tasks.rst
>>> @@ -6242,8 +6242,11 @@ Changing the listed common targets is as easy
>>> as editing your version of
>>>   ``conf-notes.txt`` in your custom template configuration directory and
>>>   making sure you have ``TEMPLATECONF`` set to your directory.
>>>   +Conserving Disk Space
>>> +=====================
>>> +
>>>   Conserving Disk Space During Builds
>>> -===================================
>>> +-----------------------------------
>>>     To help conserve disk space during builds, you can add the following
>>>   statement to your project's ``local.conf`` configuration file found in
>>> @@ -6257,6 +6260,26 @@ building a recipe once the recipe is built.
>>> For more information on
>>>   :ref:`rm_work <ref-classes-rm-work>` class in the
>>>   Yocto Project Reference Manual.
>>>   +Purging Duplicate Shared State Cache Files
>>> +-------------------------------------------
>>> +
>>> +After multiple build iterations, the Shared State (sstate) cache can
>>> contain
>>> +duplicate cache files for a given package, while only the most
>>> recent one
>>> +is likely to be reusable. The following command purges all but the
>>> +newest sstate cache file for each package::
>>> +
>>> +   sstate-cache-management.sh --remove-duplicated
>>> --cache-dir=build/sstate-cache
>>> +
>>> +This command will ask you to confirm the deletions it identifies.
>>> +
>>> +Note::
>>> +
>>> +   The duplicated sstate cache files of one package must have the same
>>> +   architecture, which means that sstate cache files with multiple
>>> +   architectures are not considered as duplicate.
>>> +
>>> +Run ``sstate-cache-management.sh`` for more details about this script.
>>> +
>>>   Working with Packages
>>>   =====================
>>>   
>> Did anyone have time to have a look at what I wrote ? There may be
>> better ways to purge the sstate cache files.
>> Thanks in advance
>> Michael.
> 
> The description looks good to me. By other ways you mean other usage
> scenarios of sstate-cache-management.sh script or by means of something
> else?
> 

I also use the --stamps-dir method since that removes additional files.
You just gotta remember to do a build of everything you want to keep first.

/Jacob

Re: [OE-core] [docs] [PATCH] dev-manual: how to purge duplicate sstate cache files

[Michael Opdenacker]

Hi Jacob,

Thank you for the suggestion!

On 11/26/21 10:20 AM, Jacob Kroon wrote:
> I also use the --stamps-dir method since that removes additional files.
> You just gotta remember to do a build of everything you want to keep first.


I don't understand the value of this option in the simple case when you
have a single build directory.  Here's the description of this option:

  --stamps-dir=<dir1>,<dir2>...<dirn>
        Specify the build directory's stamps directories, the sstate
        cache file which IS USED by these build diretories will be KEPT,
        other sstate cache files in cache-dir will be removed. Use ","
        as the separator. For example:
        --stamps-dir=build1/tmp/stamps,build2/tmp/stamps

What am I missing?

Thanks again

Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [docs] [PATCH] dev-manual: how to purge duplicate sstate cache files

[Jacob Kroon]

On 11/26/21 18:23, Michael Opdenacker wrote:
> Hi Jacob,
> 
> Thank you for the suggestion!
> 
> On 11/26/21 10:20 AM, Jacob Kroon wrote:
>> I also use the --stamps-dir method since that removes additional files.
>> You just gotta remember to do a build of everything you want to keep first.
> 
> 
> I don't understand the value of this option in the simple case when you
> have a single build directory.  Here's the description of this option:
> 
>   --stamps-dir=<dir1>,<dir2>...<dirn>
>         Specify the build directory's stamps directories, the sstate
>         cache file which IS USED by these build diretories will be KEPT,
>         other sstate cache files in cache-dir will be removed. Use ","
>         as the separator. For example:
>         --stamps-dir=build1/tmp/stamps,build2/tmp/stamps
> 
> What am I missing?
> 

--stamps-dir=${TMPDIR}/stamps

in a default build dir I think that would be "build/tmp-glibc/stamps"

Re: [OE-core] [docs] [PATCH] dev-manual: how to purge duplicate sstate cache files

[Michael Opdenacker]

Hi Slava,

On 11/26/21 10:13 AM, Vyacheslav Yurkov wrote:Did anyone have time to
have a look at what I wrote ? There may be
>> better ways to purge the sstate cache files.
>> Thanks in advance
>> Michael.
>
> The description looks good to me. By other ways you mean other usage
> scenarios of sstate-cache-management.sh script or by means of
> something else?


Thanks for the review. I was interested in any advise from real life
experience, with or without this script, to get rid of obsolete sstate
cache files.
Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com