[PATCH 0/3] ccache: Fixes for 4.1

[PATCH 0/3] ccache: Fixes for 4.1

[Robert Yang]

The following changes since commit 23cb39a5fa2a55681e7bc2605f435135cec9173b:

  diffstat: point the license checksum at the license (2021-01-05 13:48:07 +0000)

are available in the Git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/ccache
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/ccache

Robert Yang (3):
  ccache: Extend to nativesdk
  buildtools-tarball: Add nativesdk-ccache
  ccache.bbclass: Set CCACHE_TEMPDIR

 meta/classes/ccache.bbclass                  | 4 ++++
 meta/recipes-core/meta/buildtools-tarball.bb | 1 +
 meta/recipes-devtools/ccache/ccache_4.1.bb   | 2 +-
 3 files changed, 6 insertions(+), 1 deletion(-)

-- 
2.17.1

[PATCH 1/3] ccache: Extend to nativesdk

[Robert Yang]

Now we have to use host's ccache as described by:
f5b29367af ccache.bbclass: use ccache from host distribution

So extend it to nativesdk and will add it to buildtools-tarball.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/recipes-devtools/ccache/ccache_4.1.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/ccache/ccache_4.1.bb b/meta/recipes-devtools/ccache/ccache_4.1.bb
index 96254a38753..1205f4996cb 100644
--- a/meta/recipes-devtools/ccache/ccache_4.1.bb
+++ b/meta/recipes-devtools/ccache/ccache_4.1.bb
@@ -23,4 +23,4 @@ inherit cmake
 
 PATCHTOOL = "patch"
 
-BBCLASSEXTEND = "native"
+BBCLASSEXTEND = "native nativesdk"
-- 
2.17.1

[PATCH 2/3] buildtools-tarball: Add nativesdk-ccache

[Robert Yang]

Add it to buildtools-tarball so that there will be a unify version of ccache,
which can help avoid various compile errors.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/recipes-core/meta/buildtools-tarball.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/meta/buildtools-tarball.bb b/meta/recipes-core/meta/buildtools-tarball.bb
index 9da81d55235..2ffdd7c7253 100644
--- a/meta/recipes-core/meta/buildtools-tarball.bb
+++ b/meta/recipes-core/meta/buildtools-tarball.bb
@@ -29,6 +29,7 @@ TOOLCHAIN_HOST_TASK ?= "\
     nativesdk-rpcsvc-proto \
     nativesdk-patch \
     nativesdk-mtools \
+    nativesdk-ccache \
     "
 
 MULTIMACH_TARGET_SYS = "${SDK_ARCH}-nativesdk${SDK_VENDOR}-${SDK_OS}"
-- 
2.17.1

[PATCH 3/3] ccache.bbclass: Set CCACHE_TEMPDIR

[Robert Yang]

Fixed when build with buildtools-tarball:

$ bitbake linux-libc-headers
  HOSTCC  arch/x86/tools/relocs_common.o
  ccache: error: Failed to create directory /run/user/0/ccache-tmp: Permission denied

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/ccache.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
index 11a3f1cb52e..f00fafc292a 100644
--- a/meta/classes/ccache.bbclass
+++ b/meta/classes/ccache.bbclass
@@ -35,6 +35,10 @@ export CCACHE_CONFIGPATH ?= "${COREBASE}/meta/conf/ccache.conf"
 
 export CCACHE_DIR ?= "${CCACHE_TOP_DIR}/${MULTIMACH_TARGET_SYS}/${PN}"
 
+# Fixed errors:
+# ccache: error: Failed to create directory /run/user/0/ccache-tmp: Permission denied
+export CCACHE_TEMPDIR ?= "${CCACHE_DIR}/tmp"
+
 # We need to stop ccache considering the current directory or the
 # debug-prefix-map target directory to be significant when calculating
 # its hash. Without this the cache would be invalidated every time
-- 
2.17.1

Re: [OE-core] [PATCH 2/3] buildtools-tarball: Add nativesdk-ccache

[Richard Purdie]

On Wed, 2021-01-06 at 04:09 -0800, Robert Yang wrote:
> Add it to buildtools-tarball so that there will be a unify version of ccache,
> which can help avoid various compile errors.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  meta/recipes-core/meta/buildtools-tarball.bb | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/recipes-core/meta/buildtools-tarball.bb b/meta/recipes-core/meta/buildtools-tarball.bb
> index 9da81d55235..2ffdd7c7253 100644
> --- a/meta/recipes-core/meta/buildtools-tarball.bb
> +++ b/meta/recipes-core/meta/buildtools-tarball.bb
> @@ -29,6 +29,7 @@ TOOLCHAIN_HOST_TASK ?= "\
>      nativesdk-rpcsvc-proto \
>      nativesdk-patch \
>      nativesdk-mtools \
> +    nativesdk-ccache \
>      "

My view is we should not start to ship ccache with buildtools by
default. Its something that the user can install if they want/need it
but its not essential, required or helps much in standard builds (which
would reuse from sstate if built previously).

Cheers,

Richard

Re: [OE-core] [PATCH 2/3] buildtools-tarball: Add nativesdk-ccache

[Robert Yang]

Hi RP,

On 1/6/21 9:56 PM, Richard Purdie wrote:
> On Wed, 2021-01-06 at 04:09 -0800, Robert Yang wrote:
>> Add it to buildtools-tarball so that there will be a unify version of ccache,
>> which can help avoid various compile errors.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   meta/recipes-core/meta/buildtools-tarball.bb | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/meta/recipes-core/meta/buildtools-tarball.bb b/meta/recipes-core/meta/buildtools-tarball.bb
>> index 9da81d55235..2ffdd7c7253 100644
>> --- a/meta/recipes-core/meta/buildtools-tarball.bb
>> +++ b/meta/recipes-core/meta/buildtools-tarball.bb
>> @@ -29,6 +29,7 @@ TOOLCHAIN_HOST_TASK ?= "\
>>       nativesdk-rpcsvc-proto \
>>       nativesdk-patch \
>>       nativesdk-mtools \
>> +    nativesdk-ccache \
>>       "
> 
> My view is we should not start to ship ccache with buildtools by
> default. Its something that the user can install if they want/need it


The problem is that user installed ccache may not work with our recipes, for
example, we found the following recipes failed to build with ccache on different
hosts:

cdrtools-native
fribidi
rocksdb


So have a fixed version of ccache can make sure the recipes work well, it also 
helps us to reproduce the errors easily when user reported a bug.

// Robert

> but its not essential, required or helps much in standard builds (which
> would reuse from sstate if built previously).
> 
> Cheers,
> 
> Richard
> 

Re: [OE-core] [PATCH 2/3] buildtools-tarball: Add nativesdk-ccache

[Richard Purdie]

On Thu, 2021-01-07 at 10:44 +0800, Robert Yang wrote:
> Hi RP,
> 
> On 1/6/21 9:56 PM, Richard Purdie wrote:
> > On Wed, 2021-01-06 at 04:09 -0800, Robert Yang wrote:
> > > Add it to buildtools-tarball so that there will be a unify version of ccache,
> > > which can help avoid various compile errors.
> > > 
> > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> > > ---
> > >   meta/recipes-core/meta/buildtools-tarball.bb | 1 +
> > >   1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/meta/recipes-core/meta/buildtools-tarball.bb b/meta/recipes-core/meta/buildtools-tarball.bb
> > > index 9da81d55235..2ffdd7c7253 100644
> > > --- a/meta/recipes-core/meta/buildtools-tarball.bb
> > > +++ b/meta/recipes-core/meta/buildtools-tarball.bb
> > > @@ -29,6 +29,7 @@ TOOLCHAIN_HOST_TASK ?= "\
> > >       nativesdk-rpcsvc-proto \
> > >       nativesdk-patch \
> > >       nativesdk-mtools \
> > > +    nativesdk-ccache \
> > >       "
> > 
> > My view is we should not start to ship ccache with buildtools by
> > default. Its something that the user can install if they want/need it
> 
> 
> The problem is that user installed ccache may not work with our recipes, for
> example, we found the following recipes failed to build with ccache on different
> hosts:
> 
> cdrtools-native
> fribidi
> rocksdb
> 
> 
> So have a fixed version of ccache can make sure the recipes work well, it also 
> helps us to reproduce the errors easily when user reported a bug.

I really do not want ccache in buildtools. If this is such a problem,
we need to work out how to resolve the circular dependency issues so
ccache-native can work again.

Cheers,

Richard

Re: [OE-core] [PATCH 2/3] buildtools-tarball: Add nativesdk-ccache

[Robert Yang]

Hi RP,

On 1/7/21 6:40 PM, Richard Purdie wrote:
> On Thu, 2021-01-07 at 10:44 +0800, Robert Yang wrote:
>> Hi RP,
>>
>> On 1/6/21 9:56 PM, Richard Purdie wrote:
>>> On Wed, 2021-01-06 at 04:09 -0800, Robert Yang wrote:
>>>> Add it to buildtools-tarball so that there will be a unify version of ccache,
>>>> which can help avoid various compile errors.
>>>>
>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>>> ---
>>>>    meta/recipes-core/meta/buildtools-tarball.bb | 1 +
>>>>    1 file changed, 1 insertion(+)
>>>>
>>>> diff --git a/meta/recipes-core/meta/buildtools-tarball.bb b/meta/recipes-core/meta/buildtools-tarball.bb
>>>> index 9da81d55235..2ffdd7c7253 100644
>>>> --- a/meta/recipes-core/meta/buildtools-tarball.bb
>>>> +++ b/meta/recipes-core/meta/buildtools-tarball.bb
>>>> @@ -29,6 +29,7 @@ TOOLCHAIN_HOST_TASK ?= "\
>>>>        nativesdk-rpcsvc-proto \
>>>>        nativesdk-patch \
>>>>        nativesdk-mtools \
>>>> +    nativesdk-ccache \
>>>>        "
>>>
>>> My view is we should not start to ship ccache with buildtools by
>>> default. Its something that the user can install if they want/need it
>>
>>
>> The problem is that user installed ccache may not work with our recipes, for
>> example, we found the following recipes failed to build with ccache on different
>> hosts:
>>
>> cdrtools-native
>> fribidi
>> rocksdb
>>
>>
>> So have a fixed version of ccache can make sure the recipes work well, it also
>> helps us to reproduce the errors easily when user reported a bug.
> 
> I really do not want ccache in buildtools. If this is such a problem,
> we need to work out how to resolve the circular dependency issues so
> ccache-native can work again.

There are two problems to use cache-native now:
* ccache-native depends on cmake-native and zstd-native which makes it requires
   more time to build, so it might be a bottleneck for the build

* We need disable ccache-native for about 20 native recipes to resolve the 
circular dependency.

It seems that there isn't a good way to fix the problem, or we have to disable
ccache for native recipes, only make other recipes (cross, target and nativesdk)
use ccache-native.

Host's ccache isn't reliable from the history.

// Robert

> 
> Cheers,
> 
> Richard
> 
> 
> 

Re: [OE-core] [PATCH 2/3] buildtools-tarball: Add nativesdk-ccache

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 2798 bytes --]

On Thu, Jan 7, 2021, 5:00 AM Robert Yang <liezhi.yang@windriver.com> wrote:

> Hi RP,
>
> On 1/7/21 6:40 PM, Richard Purdie wrote:
> > On Thu, 2021-01-07 at 10:44 +0800, Robert Yang wrote:
> >> Hi RP,
> >>
> >> On 1/6/21 9:56 PM, Richard Purdie wrote:
> >>> On Wed, 2021-01-06 at 04:09 -0800, Robert Yang wrote:
> >>>> Add it to buildtools-tarball so that there will be a unify version of
> ccache,
> >>>> which can help avoid various compile errors.
> >>>>
> >>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> >>>> ---
> >>>>    meta/recipes-core/meta/buildtools-tarball.bb | 1 +
> >>>>    1 file changed, 1 insertion(+)
> >>>>
> >>>> diff --git a/meta/recipes-core/meta/buildtools-tarball.bb
> b/meta/recipes-core/meta/buildtools-tarball.bb
> >>>> index 9da81d55235..2ffdd7c7253 100644
> >>>> --- a/meta/recipes-core/meta/buildtools-tarball.bb
> >>>> +++ b/meta/recipes-core/meta/buildtools-tarball.bb
> >>>> @@ -29,6 +29,7 @@ TOOLCHAIN_HOST_TASK ?= "\
> >>>>        nativesdk-rpcsvc-proto \
> >>>>        nativesdk-patch \
> >>>>        nativesdk-mtools \
> >>>> +    nativesdk-ccache \
> >>>>        "
> >>>
> >>> My view is we should not start to ship ccache with buildtools by
> >>> default. Its something that the user can install if they want/need it
> >>
> >>
> >> The problem is that user installed ccache may not work with our
> recipes, for
> >> example, we found the following recipes failed to build with ccache on
> different
> >> hosts:
> >>
> >> cdrtools-native
> >> fribidi
> >> rocksdb
> >>
> >>
> >> So have a fixed version of ccache can make sure the recipes work well,
> it also
> >> helps us to reproduce the errors easily when user reported a bug.
> >
> > I really do not want ccache in buildtools. If this is such a problem,
> > we need to work out how to resolve the circular dependency issues so
> > ccache-native can work again.
>
> There are two problems to use cache-native now:
> * ccache-native depends on cmake-native and zstd-native which makes it
> requires
>    more time to build, so it might be a bottleneck for the build
>
> * We need disable ccache-native for about 20 native recipes to resolve the
> circular dependency.
>
> It seems that there isn't a good way to fix the problem, or we have to
> disable
> ccache for native recipes, only make other recipes (cross, target and
> nativesdk)
> use ccache-native.
>
> Host's ccache isn't reliable from the history.
>

We had the same problem with icecream and solved it by building in a docker
container with the version of ice cream that we wanted FWIW.

Alternatively, I think it's possible to roll your own buildtools that has
ccache?


> // Robert
>
> >
> > Cheers,
> >
> > Richard
> >
> >
> >
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 4557 bytes --]

Re: [OE-core] [PATCH 2/3] buildtools-tarball: Add nativesdk-ccache

[Richard Purdie]

On Thu, 2021-01-07 at 18:59 +0800, Robert Yang wrote:
> There are two problems to use cache-native now:
> * ccache-native depends on cmake-native and zstd-native which makes it requires
>    more time to build, so it might be a bottleneck for the build
> 
> * We need disable ccache-native for about 20 native recipes to resolve the 
> circular dependency.
> 
> It seems that there isn't a good way to fix the problem, or we have to disable
> ccache for native recipes, only make other recipes (cross, target and nativesdk)
> use ccache-native.
> 
> Host's ccache isn't reliable from the history.

Thanks for the analysis. I think I'd be inclined to disable ccache for
native recipes then. If anyone does want to use it for native recipes,
they can build a buildtools tarball of their own that contains it but I
don't want to default to that.

Cheers,

Richard

Re: [OE-core] [PATCH 2/3] buildtools-tarball: Add nativesdk-ccache

[Mikko Rapeli]

On Thu, Jan 07, 2021 at 10:12:54AM -0600, Joshua Watt wrote:
> We had the same problem with icecream and solved it by building in a docker
> container with the version of ice cream that we wanted FWIW.
> 
> Alternatively, I think it's possible to roll your own buildtools that has
> ccache?

Yea, it's a long standing best practice that an LXC or docker container
or buildtools tarball must be used to get reproducible builds in
CI systems and on developers machines with random Linux distros and
versions. Host contamination issues have reduced considerably
in recent years, thanks for all fixes, but I still can't recommend
mixing builds from different Linux distros when developing yocto
based products.

ccache version is far from the only host tool which has incompatibilities
between versions and distros.

Cheers,

-Mikko

Re: [OE-core] [PATCH 2/3] buildtools-tarball: Add nativesdk-ccache

[Richard Purdie]

On Fri, 2021-01-08 at 07:47 +0000, Mikko.Rapeli@bmw.de wrote:
> On Thu, Jan 07, 2021 at 10:12:54AM -0600, Joshua Watt wrote:
> > We had the same problem with icecream and solved it by building in a docker
> > container with the version of ice cream that we wanted FWIW.
> > 
> > Alternatively, I think it's possible to roll your own buildtools that has
> > ccache?
> 
> Yea, it's a long standing best practice that an LXC or docker container
> or buildtools tarball must be used to get reproducible builds in
> CI systems and on developers machines with random Linux distros and
> versions. Host contamination issues have reduced considerably
> in recent years, thanks for all fixes, but I still can't recommend
> mixing builds from different Linux distros when developing yocto
> based products.

Whilst this has been an issue I am quite proud of the work we've done
with the autobuilder to detect and resolve reproducibility issues which
has now been extended to world builds, i.e. apart from a small subset
of known issues (~65 packages), OE-Core is fully reproducible over all
out tested distros.

Whilst containers are a way of avoiding such issues, I think some of
the recent issues would even have caused differences between containers
(though kernel config or the geo of nameservers).

I appreciate that it hasn't been extended beyond OE-Core as yet but
think it does show what is possible and that there aren't that many
issues, certainly the core and tools are in a strong position.

Cheers,

Richard

Re: [OE-core] [PATCH 2/3] buildtools-tarball: Add nativesdk-ccache

[Robert Yang]

On 1/8/21 1:10 AM, Richard Purdie wrote:
> On Thu, 2021-01-07 at 18:59 +0800, Robert Yang wrote:
>> There are two problems to use cache-native now:
>> * ccache-native depends on cmake-native and zstd-native which makes it requires
>>     more time to build, so it might be a bottleneck for the build
>>
>> * We need disable ccache-native for about 20 native recipes to resolve the
>> circular dependency.
>>
>> It seems that there isn't a good way to fix the problem, or we have to disable
>> ccache for native recipes, only make other recipes (cross, target and nativesdk)
>> use ccache-native.
>>
>> Host's ccache isn't reliable from the history.
> 
> Thanks for the analysis. I think I'd be inclined to disable ccache for
> native recipes then. If anyone does want to use it for native recipes,
> they can build a buildtools tarball of their own that contains it but I
> don't want to default to that.

Yes, make sense, I will send a patch to disable ccache for native recipes and 
enable ccache-native for other recipes.

// Robert

> 
> Cheers,
> 
> Richard
> 

Re: [OE-core] [PATCH 2/3] buildtools-tarball: Add nativesdk-ccache

[Robert Yang]

On 1/8/21 12:12 AM, Joshua Watt wrote:
> 
> 
> On Thu, Jan 7, 2021, 5:00 AM Robert Yang <liezhi.yang@windriver.com 
> <mailto:liezhi.yang@windriver.com>> wrote:
> 
>     Hi RP,
> 
>     On 1/7/21 6:40 PM, Richard Purdie wrote:
>      > On Thu, 2021-01-07 at 10:44 +0800, Robert Yang wrote:
>      >> Hi RP,
>      >>
>      >> On 1/6/21 9:56 PM, Richard Purdie wrote:
>      >>> On Wed, 2021-01-06 at 04:09 -0800, Robert Yang wrote:
>      >>>> Add it to buildtools-tarball so that there will be a unify version of
>     ccache,
>      >>>> which can help avoid various compile errors.
>      >>>>
>      >>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com
>     <mailto:liezhi.yang@windriver.com>>
>      >>>> ---
>      >>>>    meta/recipes-core/meta/buildtools-tarball.bb
>     <http://buildtools-tarball.bb> | 1 +
>      >>>>    1 file changed, 1 insertion(+)
>      >>>>
>      >>>> diff --git a/meta/recipes-core/meta/buildtools-tarball.bb
>     <http://buildtools-tarball.bb>
>     b/meta/recipes-core/meta/buildtools-tarball.bb <http://buildtools-tarball.bb>
>      >>>> index 9da81d55235..2ffdd7c7253 100644
>      >>>> --- a/meta/recipes-core/meta/buildtools-tarball.bb
>     <http://buildtools-tarball.bb>
>      >>>> +++ b/meta/recipes-core/meta/buildtools-tarball.bb
>     <http://buildtools-tarball.bb>
>      >>>> @@ -29,6 +29,7 @@ TOOLCHAIN_HOST_TASK ?= "\
>      >>>>        nativesdk-rpcsvc-proto \
>      >>>>        nativesdk-patch \
>      >>>>        nativesdk-mtools \
>      >>>> +    nativesdk-ccache \
>      >>>>        "
>      >>>
>      >>> My view is we should not start to ship ccache with buildtools by
>      >>> default. Its something that the user can install if they want/need it
>      >>
>      >>
>      >> The problem is that user installed ccache may not work with our recipes, for
>      >> example, we found the following recipes failed to build with ccache on
>     different
>      >> hosts:
>      >>
>      >> cdrtools-native
>      >> fribidi
>      >> rocksdb
>      >>
>      >>
>      >> So have a fixed version of ccache can make sure the recipes work well,
>     it also
>      >> helps us to reproduce the errors easily when user reported a bug.
>      >
>      > I really do not want ccache in buildtools. If this is such a problem,
>      > we need to work out how to resolve the circular dependency issues so
>      > ccache-native can work again.
> 
>     There are two problems to use cache-native now:
>     * ccache-native depends on cmake-native and zstd-native which makes it requires
>         more time to build, so it might be a bottleneck for the build
> 
>     * We need disable ccache-native for about 20 native recipes to resolve the
>     circular dependency.
> 
>     It seems that there isn't a good way to fix the problem, or we have to disable
>     ccache for native recipes, only make other recipes (cross, target and nativesdk)
>     use ccache-native.
> 
>     Host's ccache isn't reliable from the history.
> 
> 
> We had the same problem with icecream and solved it by building in a docker 
> container with the version of ice cream that we wanted FWIW.
> 
> Alternatively, I think it's possible to roll your own buildtools that has ccache?

Yes, thanks, it's a good suggestion.

// Robert

> 
> 
>     // Robert
> 
>      >
>      > Cheers,
>      >
>      > Richard
>      >
>      >
>      >
> 
>     
>