[PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Yannick Gicquel]

ccache directories are limited to 1G by default.
This patch enables the configuration of their limits, and as default
location is TMPDIR, it proposes a size limit to "0" (unlimited).

The setup can be overloaded in local.conf by setting
CCACHE_MAX_SIZE to a custom value if needed.

Signed-off-by: Yannick Gicquel <yannick.gicquel@iot.bzh>
---
 meta/classes/ccache.bbclass | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
index 93fcaca..b6643a1 100644
--- a/meta/classes/ccache.bbclass
+++ b/meta/classes/ccache.bbclass
@@ -1,6 +1,14 @@
 CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
 export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
 CCACHE_DISABLE[unexport] = "1"
+CCACHE_MAX_SIZE ?= "0"
 
 do_configure[dirs] =+ "${CCACHE_DIR}"
 do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
+
+ccache_init() {
+    if [ -n "${CCACHE}" ]; then
+        ${CCACHE} -M ${CCACHE_MAX_SIZE}
+    fi
+}
+do_configure[postfuncs] += "ccache_init"
-- 
1.9.1

[PATCH v2 2/3] ccache.bbclass: add trimming tasks

[Yannick Gicquel]

These can be use to trim ccache directories to default cache size (1G)
without deleting the whole dirs.

Signed-off-by: Yannick Gicquel <yannick.gicquel@iot.bzh>
---
 meta/classes/ccache.bbclass | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
index b6643a1..0bad4d4 100644
--- a/meta/classes/ccache.bbclass
+++ b/meta/classes/ccache.bbclass
@@ -12,3 +12,20 @@ ccache_init() {
     fi
 }
 do_configure[postfuncs] += "ccache_init"
+
+addtask ccache_trim
+do_ccache_trim[nostamp] = "1"
+do_ccache_trim() {
+    if [ -n "${CCACHE}" -a -d "${CCACHE_DIR}" ]; then
+        ${CCACHE} -M 1G
+        ${CCACHE} -c
+    fi
+}
+
+addtask ccache_trimall after do_ccache_trim
+do_ccache_trimall[recrdeptask] = "do_ccache_trimall do_ccache_trim"
+do_ccache_trimall[recideptask] = "do_${BB_DEFAULT_TASK}"
+do_ccache_trimall[nostamp] = "1"
+do_ccache_trimall() {
+        :
+}
-- 
1.9.1

[PATCH v2 3/3] cmake.bbclass: enable usage of ccache

[Yannick Gicquel]

This allows ccache usage for recipes which inherit cmake.bbclass.
Since cmake v3.4, it can be enabled using some "-D" options.

Here below are some metrics while using it for webkitgtk recipe.
(machine is a 4x core i7 @ 3.4GHz)

$ bitbake -c fetchall webkitgtk
$ time bitbake webkitgtk
<snip>
real	56m25.191s
user	359m32.003s
sys	34m49.356s

$ bitbake -c clean webkitgtk
$ bitbake -c cleansstate webkitgtk
$ time bitbake webkitgtk
<snip>
real	25m19.298s
user	17m57.861s
sys	6m20.157s

Signed-off-by: Yannick Gicquel <yannick.gicquel@iot.bzh>
---
 meta/classes/cmake.bbclass | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index 9e74599..737a4e0 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -4,12 +4,9 @@ OECMAKE_SOURCEPATH ?= "${S}"
 DEPENDS_prepend = "cmake-native "
 B = "${WORKDIR}/build"
 
-# We need to unset CCACHE otherwise cmake gets too confused
-CCACHE = ""
-
 # C/C++ Compiler (without cpu arch/tune arguments)
-OECMAKE_C_COMPILER ?= "`echo ${CC} | sed 's/^\([^ ]*\).*/\1/'`"
-OECMAKE_CXX_COMPILER ?= "`echo ${CXX} | sed 's/^\([^ ]*\).*/\1/'`"
+OECMAKE_C_COMPILER ?= "${@'${CC}'.replace('${CCACHE}','',1).split(' ')[0]}"
+OECMAKE_CXX_COMPILER ?= "${@'${CXX}'.replace('${CCACHE}','',1).split(' ')[0]}"
 OECMAKE_AR ?= "${AR}"
 
 # Compiler flags
@@ -108,9 +105,16 @@ cmake_do_configure() {
 		OECMAKE_SITEFILE=""
 	fi
 
+	if [ -n "${CCACHE}" ]; then
+		OECMAKE_CCACHE="-DCMAKE_C_COMPILER_LAUNCHER=${CCACHE} -DCMAKE_CXX_COMPILER_LAUNCHER=${CCACHE}"
+	else
+		OECMAKE_CCACHE=""
+	fi
+
 	cmake \
 	  ${OECMAKE_SITEFILE} \
 	  ${OECMAKE_SOURCEPATH} \
+	  ${OECMAKE_CCACHE} \
 	  -DCMAKE_INSTALL_PREFIX:PATH=${prefix} \
 	  -DCMAKE_INSTALL_BINDIR:PATH=${@os.path.relpath(d.getVar('bindir'), d.getVar('prefix'))} \
 	  -DCMAKE_INSTALL_SBINDIR:PATH=${@os.path.relpath(d.getVar('sbindir'), d.getVar('prefix'))} \
-- 
1.9.1

Re: [PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Andre McCurdy]

On Thu, Jan 19, 2017 at 7:01 AM, Yannick Gicquel
<yannick.gicquel@iot.bzh> wrote:
> ccache directories are limited to 1G by default.

1G has been the default for a long time, so maybe it's time to propose
an upstream change to increase, e.g. to 10G, giving the size of the
webkitgtk build as justification?

Since we build ccache-native within oe-core, any such patch could be
applied to our version of ccache right now (ie we don't need to wait
for it to be accepted and merged upstream).

> This patch enables the configuration of their limits, and as default
> location is TMPDIR, it proposes a size limit to "0" (unlimited).
>
> The setup can be overloaded in local.conf by setting
> CCACHE_MAX_SIZE to a custom value if needed.
>
> Signed-off-by: Yannick Gicquel <yannick.gicquel@iot.bzh>
> ---
>  meta/classes/ccache.bbclass | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
> index 93fcaca..b6643a1 100644
> --- a/meta/classes/ccache.bbclass
> +++ b/meta/classes/ccache.bbclass
> @@ -1,6 +1,14 @@
>  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
>  CCACHE_DISABLE[unexport] = "1"
> +CCACHE_MAX_SIZE ?= "0"
>
>  do_configure[dirs] =+ "${CCACHE_DIR}"
>  do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
> +
> +ccache_init() {
> +    if [ -n "${CCACHE}" ]; then
> +        ${CCACHE} -M ${CCACHE_MAX_SIZE}
> +    fi
> +}
> +do_configure[postfuncs] += "ccache_init"
> --
> 1.9.1
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Yannick GICQUEL]

Hi,

Some comments inline below (with oe-list this time),

Regards,

2017-01-20 17:23 GMT+01:00 Yannick GICQUEL <yannick.gicquel@iot.bzh>:
> 2017-01-19 20:02 GMT+01:00 Andre McCurdy <armccurdy@gmail.com>:
>> On Thu, Jan 19, 2017 at 7:01 AM, Yannick Gicquel
>> <yannick.gicquel@iot.bzh> wrote:
>>> ccache directories are limited to 1G by default.
>>
>> 1G has been the default for a long time, so maybe it's time to propose
>> an upstream change to increase, e.g. to 10G, giving the size of the
>> webkitgtk build as justification?
>
> You are right, 1GB was the default cache size limit in old versions
> and it's up to 5GB since release 3.2 [1].
>
>>
>> Since we build ccache-native within oe-core, any such patch could be
>> applied to our version of ccache right now (ie we don't need to wait
>> for it to be accepted and merged upstream).
>
> Current ccache.bbclass implementation is using the host ccache, so the
> default limit can be different depending on the version deployed on
> host distro. The point is that even if we use ccache-native, their
> still will be a default size limit set.
> Even if 5GB (or 10GB) limit should fit the webkitgtk cache size
> requirements, some other recipes can require more space to gain full
> benefit of the ccache (e.g. chromium from meta-browser) and for those
> it can be useful to tune the size.
>
> Does it make sense?
>
> Yannick
>
> [1]: https://ccache.samba.org/releasenotes.html#_ccache_3_2
>
>>
>>> This patch enables the configuration of their limits, and as default
>>> location is TMPDIR, it proposes a size limit to "0" (unlimited).
>>>
>>> The setup can be overloaded in local.conf by setting
>>> CCACHE_MAX_SIZE to a custom value if needed.
>>>
>>> Signed-off-by: Yannick Gicquel <yannick.gicquel@iot.bzh>
>>> ---
>>>  meta/classes/ccache.bbclass | 8 ++++++++
>>>  1 file changed, 8 insertions(+)
>>>
>>> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
>>> index 93fcaca..b6643a1 100644
>>> --- a/meta/classes/ccache.bbclass
>>> +++ b/meta/classes/ccache.bbclass
>>> @@ -1,6 +1,14 @@
>>>  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>>>  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
>>>  CCACHE_DISABLE[unexport] = "1"
>>> +CCACHE_MAX_SIZE ?= "0"
>>>
>>>  do_configure[dirs] =+ "${CCACHE_DIR}"
>>>  do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
>>> +
>>> +ccache_init() {
>>> +    if [ -n "${CCACHE}" ]; then
>>> +        ${CCACHE} -M ${CCACHE_MAX_SIZE}
>>> +    fi
>>> +}
>>> +do_configure[postfuncs] += "ccache_init"
>>> --
>>> 1.9.1
>>>
>>> --
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Andre McCurdy]

On Fri, Jan 20, 2017 at 8:47 AM, Yannick GICQUEL
<yannick.gicquel@iot.bzh> wrote:
> 2017-01-20 17:23 GMT+01:00 Yannick GICQUEL <yannick.gicquel@iot.bzh>:
>> 2017-01-19 20:02 GMT+01:00 Andre McCurdy <armccurdy@gmail.com>:
>>> On Thu, Jan 19, 2017 at 7:01 AM, Yannick Gicquel
>>> <yannick.gicquel@iot.bzh> wrote:
>>>> ccache directories are limited to 1G by default.
>>>
>>> 1G has been the default for a long time, so maybe it's time to propose
>>> an upstream change to increase, e.g. to 10G, giving the size of the
>>> webkitgtk build as justification?
>>
>> You are right, 1GB was the default cache size limit in old versions
>> and it's up to 5GB since release 3.2 [1].
>>
>>> Since we build ccache-native within oe-core, any such patch could be
>>> applied to our version of ccache right now (ie we don't need to wait
>>> for it to be accepted and merged upstream).
>>
>> Current ccache.bbclass implementation is using the host ccache, so the
>> default limit can be different depending on the version deployed on
>> host distro. The point is that even if we use ccache-native, their
>> still will be a default size limit set.
>> Even if 5GB (or 10GB) limit should fit the webkitgtk cache size
>> requirements, some other recipes can require more space to gain full
>> benefit of the ccache (e.g. chromium from meta-browser) and for those
>> it can be useful to tune the size.
>>
>> Does it make sense?

Yes, it does. If 5GB isn't enough, then enabling compression of files
in ccache ( https://ccache.samba.org/manual.html#_cache_compression )
may be a better solution than removing the 5GB limit, but I do see the
argument that since we don't place arbitrary limits on downloads or
sstate directory sizes, why limit ccache?

However, calling "ccache -M XXX" after every recipe's do_configure()
still doesn't seem ideal. Firstly because the command will run many
more times than necessary but also because, for users who over-ride
CCACHE_DIR to try to share ccache files between builds, it will
silently and permanently over-write any existing "max_size" option
which they may have manually configured.

As an alternative, does simply exporting CCACHE_MAXSIZE=0 from
ccache.bbclass work?

(In general, I'm all in favour of improving ccache support in oe-core
so it's good to see these changes being proposed and discussed).

>> Yannick
>>
>> [1]: https://ccache.samba.org/releasenotes.html#_ccache_3_2
>>
>>>
>>>> This patch enables the configuration of their limits, and as default
>>>> location is TMPDIR, it proposes a size limit to "0" (unlimited).
>>>>
>>>> The setup can be overloaded in local.conf by setting
>>>> CCACHE_MAX_SIZE to a custom value if needed.
>>>>
>>>> Signed-off-by: Yannick Gicquel <yannick.gicquel@iot.bzh>
>>>> ---
>>>>  meta/classes/ccache.bbclass | 8 ++++++++
>>>>  1 file changed, 8 insertions(+)
>>>>
>>>> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
>>>> index 93fcaca..b6643a1 100644
>>>> --- a/meta/classes/ccache.bbclass
>>>> +++ b/meta/classes/ccache.bbclass
>>>> @@ -1,6 +1,14 @@
>>>>  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>>>>  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
>>>>  CCACHE_DISABLE[unexport] = "1"
>>>> +CCACHE_MAX_SIZE ?= "0"
>>>>
>>>>  do_configure[dirs] =+ "${CCACHE_DIR}"
>>>>  do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
>>>> +
>>>> +ccache_init() {
>>>> +    if [ -n "${CCACHE}" ]; then
>>>> +        ${CCACHE} -M ${CCACHE_MAX_SIZE}
>>>> +    fi
>>>> +}
>>>> +do_configure[postfuncs] += "ccache_init"
>>>> --
>>>> 1.9.1
>>>>

Re: [PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Yannick GICQUEL]

2017-01-24 0:20 GMT+01:00 Andre McCurdy <armccurdy@gmail.com>:
> On Fri, Jan 20, 2017 at 8:47 AM, Yannick GICQUEL
> <yannick.gicquel@iot.bzh> wrote:
>> 2017-01-20 17:23 GMT+01:00 Yannick GICQUEL <yannick.gicquel@iot.bzh>:
>>> 2017-01-19 20:02 GMT+01:00 Andre McCurdy <armccurdy@gmail.com>:
>>>> On Thu, Jan 19, 2017 at 7:01 AM, Yannick Gicquel
>>>> <yannick.gicquel@iot.bzh> wrote:
>>>>> ccache directories are limited to 1G by default.
>>>>
>>>> 1G has been the default for a long time, so maybe it's time to propose
>>>> an upstream change to increase, e.g. to 10G, giving the size of the
>>>> webkitgtk build as justification?
>>>
>>> You are right, 1GB was the default cache size limit in old versions
>>> and it's up to 5GB since release 3.2 [1].
>>>
>>>> Since we build ccache-native within oe-core, any such patch could be
>>>> applied to our version of ccache right now (ie we don't need to wait
>>>> for it to be accepted and merged upstream).
>>>
>>> Current ccache.bbclass implementation is using the host ccache, so the
>>> default limit can be different depending on the version deployed on
>>> host distro. The point is that even if we use ccache-native, their
>>> still will be a default size limit set.
>>> Even if 5GB (or 10GB) limit should fit the webkitgtk cache size
>>> requirements, some other recipes can require more space to gain full
>>> benefit of the ccache (e.g. chromium from meta-browser) and for those
>>> it can be useful to tune the size.
>>>
>>> Does it make sense?
>
> Yes, it does. If 5GB isn't enough, then enabling compression of files
> in ccache ( https://ccache.samba.org/manual.html#_cache_compression )
> may be a better solution than removing the 5GB limit, but I do see the
> argument that since we don't place arbitrary limits on downloads or
> sstate directory sizes, why limit ccache?
>
> However, calling "ccache -M XXX" after every recipe's do_configure()
> still doesn't seem ideal. Firstly because the command will run many
> more times than necessary but also because, for users who over-ride

This add a call to ccache binary once per recipe build. In term of
execution time, IMHO it looks quite reasonable compared to the
drawback of a rebuild with limited cache size, but...

> CCACHE_DIR to try to share ccache files between builds, it will
> silently and permanently over-write any existing "max_size" option
> which they may have manually configured.

... I agree : the default behavior proposed can silently change some
user's custom setup, which may not be a good thing. (Same remark can
be done on the activation of compression, which may be kept to the
user's preferences)

>
> As an alternative, does simply exporting CCACHE_MAXSIZE=0 from
> ccache.bbclass work?

With recent ccache release, yes it does (v3.3.3 at least). But
depending on the host distro, it may not.
I had some experiment about using "ccache-native" binary, and I also
agree it can provides more flexibility : most (all?) of the
configuration variable can be tuned by exporting relevant environment
variable.

Updating ccache.bbclass to use ccache-native is not an easy task :
basically we could add a DEPENDS on "ccache-native" to all recipe
which inherit this class, but it can bring some dependency loops on
ccache recipe itself, which I do not know how to fix in a generic way.

About this patch serie, I propose to split it and keep this "max_size"
tune feature on a separate scope.

>
> (In general, I'm all in favour of improving ccache support in oe-core
> so it's good to see these changes being proposed and discussed).

You are welcome, thanks for your feedbacks.

>
>>> Yannick
>>>
>>> [1]: https://ccache.samba.org/releasenotes.html#_ccache_3_2
>>>
>>>>
>>>>> This patch enables the configuration of their limits, and as default
>>>>> location is TMPDIR, it proposes a size limit to "0" (unlimited).
>>>>>
>>>>> The setup can be overloaded in local.conf by setting
>>>>> CCACHE_MAX_SIZE to a custom value if needed.
>>>>>
>>>>> Signed-off-by: Yannick Gicquel <yannick.gicquel@iot.bzh>
>>>>> ---
>>>>>  meta/classes/ccache.bbclass | 8 ++++++++
>>>>>  1 file changed, 8 insertions(+)
>>>>>
>>>>> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
>>>>> index 93fcaca..b6643a1 100644
>>>>> --- a/meta/classes/ccache.bbclass
>>>>> +++ b/meta/classes/ccache.bbclass
>>>>> @@ -1,6 +1,14 @@
>>>>>  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>>>>>  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
>>>>>  CCACHE_DISABLE[unexport] = "1"
>>>>> +CCACHE_MAX_SIZE ?= "0"
>>>>>
>>>>>  do_configure[dirs] =+ "${CCACHE_DIR}"
>>>>>  do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
>>>>> +
>>>>> +ccache_init() {
>>>>> +    if [ -n "${CCACHE}" ]; then
>>>>> +        ${CCACHE} -M ${CCACHE_MAX_SIZE}
>>>>> +    fi
>>>>> +}
>>>>> +do_configure[postfuncs] += "ccache_init"
>>>>> --
>>>>> 1.9.1
>>>>>