[Buildroot] [Bug 10536] New: Finding non-relative paths in the ccache

[Buildroot] [Bug 10536] New: Finding non-relative paths in the ccache

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10536

            Bug ID: 10536
           Summary: Finding non-relative paths in the ccache
           Product: buildroot
           Version: unspecified
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: Other
          Assignee: unassigned at buildroot.uclibc.org
          Reporter: mhirota at impinj.com
                CC: buildroot at uclibc.org
  Target Milestone: ---

It appears that non-relative paths are leaking into the buildroot ccache
despite the default BR2_CCACHE_USE_BASEDIR=y.

Here are my steps to demonstrate this:

    $ git clone git://git.buildroot.net/buildroot
    $ cd buildroot
    $ make zynq_zybo_defconfig
    $ make menuconfig
    $ # Build options -->
    $ #   [*] Enable compiler cache
    $ #   < Save > and < Exit > back to prompt
    $ make

At this point, I grep the ccache for anything that starts with "/home":

    $ grep -r --include "*.d" "^\/home" $HOME/.buildroot-ccache/ | wc -l
    647

I was expecting this to be zero.

Just to confirm some assumptions:

    $ grep -r --include "*.d" "^\.\." $HOME/.buildroot-ccache/ | wc -l
    60840
    $ grep CCACHE .config
    BR2_CCACHE=y
    BR2_CCACHE_DIR="$(HOME)/.buildroot-ccache"
    BR2_CCACHE_INITIAL_SETUP=""
    BR2_CCACHE_USE_BASEDIR=y
    # BR2_PACKAGE_CCACHE is not set

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10536] Finding non-relative paths in the ccache

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #1 from Trent Piepho <tpiepho@gmail.com> ---
Examining the build of objects that have non-relative paths into the buildroot
output directory in ccache shows how they are getting created.  Here's an
example from host-uboot-tools:

    /work/build-system/output/r700/host/bin/ccache /usr/lib64/ccache/gcc
-Wp,-MD,tools/.mkimage.o.d -O2 -I/work/build-system/output/r700/host/include  
-include ./include/libfdt_env.h -idirafterinclude -idirafter./arch//include
-I./lib/libfdt -I./tools -DUSE_HOSTCC -D__KERNEL_STRICT_NAMES -D_GNU_SOURCE  -c
-o tools/mkimage.o tools/mkimage.c

Since this is building a host binary it doesn't not call the buildroot
toolchain wrapper.  The toolchain wrapper is what sets basedir for ccache and
so ccache will not make absolute paths into the buildroot output directory tree
relative.

I think in order to see this one needs to build a host binary, have the
package's build system for the host binary use buildroot's cccache, have it
produce an auto-dependency file (-MD, etc), and have the code use a header that
in the buildroot output tree, i.e. some part of a host library package.

Perhaps this could be fixed if CCACHE_BASEDIR was injected in a way that also
managed to apply it to host compilations?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10536] Finding non-relative paths in the ccache

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #2 from Trent Piepho <tpiepho@gmail.com> ---
This ends up being a big problem if multiple buildroot output directories are
used.  For example:

    make O=first some_defconfig ; make O=first all
    rm -rf first
    make O=second some_defconfig ; make O=second all

The build in second fails because ccache will use cached dependency files from
the first build, and these cached files reference headers with a "*/first/*"
path and those headers no longer exist.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10536] Finding non-relative paths in the ccache

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10536

Thomas Petazzoni <thomas.petazzoni@free-electrons.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|unassigned at buildroot.uclibc |arnout at mind.be
                   |.org                        |

--- Comment #3 from Thomas Petazzoni <thomas.petazzoni@free-electrons.com> ---
Arnout, since you're looking after all the ccache related topics, could you
have a look at this? Thanks!

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10536] Finding non-relative paths in the ccache

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #4 from Arnout Vandecappelle <arnout@mind.be> ---
Does the build really fail in that second example? Since the first/ directory
is in the command-line arguments during the build and it doesn't match
BR2_CCACHE_DIR, ccache should just treat it as a cache miss. So indeed, ccache
doesn't work for host packages unless the output directory is the same.

We could actually easily export CCACHE_BASEDIR from make, so that it works also
for host packages. We could perhaps even remove it from the toolchain wrapper
then, though it's probably convenient that the same basedir is used when
building outside of Buildroot.

That said, this problem will still exist when a BR2_HOST_DIR is specified that
points outside of the output directory. In that case we'd need two different
'base directories', which is obviously not possible...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10536] Finding non-relative paths in the ccache

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #5 from Mark Hirota <mhirota@impinj.com> ---
Arnout: yes, we have seen a failure as Trent describes, but with 2 similar
configs: r700 and r700_rescue.

>>> uboot-tools 2017.09 Building
PATH="/mnt/work/output/r700/host/bin:/mnt/work/output/r700/host/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
/usr/bin/make -j5 -C /mnt/work/output/r700/build/uboot-tools-2017.09
CROSS_COMPILE="/mnt/work/output/r700/host/bin/arm-buildroot-linux-gnueabihf-"
CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O3 
-fstack-protector-strong" LDFLAGS=""
STRIP=/mnt/work/output/r700/host/bin/arm-buildroot-linux-gnueabihf-strip
CROSS_BUILD_TOOLS=y tools-only
  CHK     include/config/uboot.release
  CHK     include/generated/timestamp_autogenerated.h
  HOSTCC  scripts/basic/fixdep
  UPD     include/generated/timestamp_autogenerated.h
  UPD     include/config/uboot.release
  CHK     include/generated/version_autogenerated.h
  UPD     include/generated/version_autogenerated.h
  HOSTCC  tools/mkenvimage.o
  WRAP    tools/lib/crc32.c
  HOSTCC  tools/os_support.o
  HOSTCC  tools/aisimage.o
  HOSTCC  tools/atmelimage.o
  WRAP    tools/common/bootm.c
  HOSTCC  tools/default_image.o
fixdep: error opening config file:
/mnt/work/output/r700_rescue/host/lib/gcc/arm-buildroot-linux-gnueabihf/7.2.0/include/stddef.h:
No such file or directory

Our build automation performs clean builds, but shares a buildroot-ccache
across them.  This type of failure forces us to build the projects on separate
build hosts to avoid hitting it.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10536] Finding non-relative paths in the ccache

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #6 from Arnout Vandecappelle <arnout@mind.be> ---
Hm, this build failure doesn't look like it will be solved with CCACHE_BASEDIR.
Indeed, it is the *target* ccache'd compiler that generated a *.d file that
apparently still has a full path in it. I don't see how this could happen...

Possibly the issue is that this is a path that is embedded in the specs file,
and ccache will only substitute paths that get passed on the command line
(through -I etc.).

Needs more debugging...

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10536] Finding non-relative paths in the ccache

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #7 from Mark Hirota <mhirota@impinj.com> ---
Arnout: I believe the issue is with ccache 3.3.4.  I see in the 3.3.5 release
notes (https://ccache.samba.org/releasenotes.html#_ccache_3_3_5) the following:

> Fixed a bug related to erroneously storing a dependency file with absolute paths in the cache on a preprocessed hit.

Should the ccache version in buildroot be updated to 3.3.5?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10536] Finding non-relative paths in the ccache

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #8 from Arnout Vandecappelle <arnout@mind.be> ---
Good catch! Can you try updating ccache, check if that fixes things for you,
and if so send the patch to the list with a reference to this bug in the commit
log?

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10536] Finding non-relative paths in the ccache

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10536

Peter Korsgaard <jacmet@uclibc.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #9 from Peter Korsgaard <jacmet@uclibc.org> ---
Fixed in git by bumping ccache to 3.3.5

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[Buildroot] [Bug 10536] Finding non-relative paths in the ccache

[bugzilla at busybox.net]

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #10 from Trent Piepho <tpiepho@gmail.com> ---
I seems like Arnout should be right about host compiles in different
directories not sharing ccache.  Since CCACHE_BASEDIR is not set, the paths
should be absolute and different directories would be a cache miss.

But here's some testing with host-uboot-tools.  First a build from a cleared
and reset ccache:
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                            58
cache hit rate                      0.00 %
called for link                        8
cleanups performed                     0
files in cache                       171

As expected, all misses. 58 files.

Building again from the same output dir:
cache hit (direct)                    53
cache hit (preprocessed)               5
cache miss                            58
cache hit rate                     50.00 %
called for link                       16
cleanups performed                     0
files in cache                       176

Exactly 58 new hits, as expected, everything in cache.  But 5 new files, odd.

Now the tricky one, building from a different output dir:
cache hit (direct)                    53
cache hit (preprocessed)              15
cache miss                           106
cache hit rate                     39.08 %
called for link                       24
cleanups performed                     0
files in cache                       331

These should be all misses, right?  But there were 10 hits in preprocessed
mode!  Which is fair, if the preprocessed code ends up being exactly the same. 
But just because the c code is the same doesn't mean the auto-dependency file
is the same (absolute paths in the .d file differ!).  If ccache tries to use
the cached .d file with absolute paths on this supposed preprocessed hit, it
will fail.

-- 
You are receiving this mail because:
You are on the CC list for the bug.