[GIT PULL 1/2] Kbuild updates for v4.15

[GIT PULL 1/2] Kbuild updates for v4.15

[Masahiro Yamada]

Hi Linus,

Here are Kbuild updates for v4.15.  Please pull!



The following changes since commit 9e66317d3c92ddaab330c125dfe9d06eee268aff:

  Linux 4.14-rc3 (2017-10-01 14:54:54 -0700)

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
tags/kbuild-v4.15

for you to fetch changes up to 7f855fc805cd9c29867aed56cc20f818b36a7b7b:

  kbuild: move coccicheck help from scripts/Makefile.help to top
Makefile (2017-11-17 00:33:09 +0900)

----------------------------------------------------------------
Kbuild updates for v4.15

One of the most remarkable improvements in this cycle is, Kbuild is
now able to cache the result of shell commands.  Some variables are
expensive to compute, for example, $(call cc-option,...) invokes the
compiler.  It is not efficient to redo this computation every time,
even when we are not actually building anything.  Kbuild creates a
hidden file ".cache.mk" that contains invoked shell commands and
their results.  The speed-up should be noticeable.

Summary:

- Fix arch build issues (hexagon, sh)

- Clean up various Makefiles and scripts

- Fix wrong usage of {CFLAGS,LDFLAGS}_MODULE in arch Makefiles

- Cache variables that are expensive to compute

- Improve cc-ldopton and ld-option for Clang

- Optimize output directory creation

----------------------------------------------------------------
Cao jin (4):
      kbuild: comments cleanup in Makefile.lib
      arm/kbuild: replace {C, LD}FLAGS_MODULE with KBUILD_{C, LD}FLAGS_MODULE
      c6x/kbuild: replace CFLAGS_MODULE with KBUILD_CFLAGS_MODULE
      hexagon/kbuild: replace CFLAGS_MODULE with KBUILD_CFLAGS_MODULE

Douglas Anderson (2):
      kbuild: Add a cache for generated variables
      kbuild: Cache a few more calls to the compiler

Masahiro Yamada (22):
      hexagon: get rid of #include <generated/compile.h>
      kbuild: link-vmlinux.sh: simplify .version increment
      kbuild: mkcompile_h: do not create .version
      kbuild: replace $(hdr-arch) with $(SRCARCH)
      kbuild: move "_all" target out of $(KBUILD_SRC) conditional
      kbuild: re-order the code to not parse unnecessary variables
      kbuild: remove KBUILD_SUBDIR_ASFLAGS and KBUILD_SUBDIR_CCFLAGS
      kbuild: add forward declaration of default target to Makefile.asm-generic
      kbuild: do not call cc-option before KBUILD_CFLAGS initialization
      kbuild: shrink .cache.mk when it exceeds 1000 lines
      sh: select KBUILD_DEFCONFIG depending on ARCH
      kbuild: create directory for make cache only when necessary
      kbuild: remove redundant $(wildcard ...) for cmd_files calculation
      kbuild: filter-out PHONY targets from "targets"
      kbuild: create object directories simpler and faster
      kbuild: optimize object directory creation for incremental build
      kbuild: remove redundant mkdir from ./Kbuild
      kbuild: specify FORCE in Makefile.headersinst as .PHONY target
      selinux: remove unnecessary assignment to subdir-
      frv: .gitignore: ignore vmlinux.lds
      sh: decompressor: add shipped files to .gitignore
      kbuild: move coccicheck help from scripts/Makefile.help to top Makefile

Nick Desaulniers (2):
      kbuild: clang: remove crufty HOSTCFLAGS
      kbuild: fix linker feature test macros when cross compiling with Clang

 Kbuild                             |   2 -
 Makefile                           | 300 ++++++++++++++--------------
 arch/arm/Makefile                  |   6 +-
 arch/c6x/Makefile                  |   2 +-
 arch/frv/kernel/.gitignore         |   1 +
 arch/hexagon/Makefile              |   6 +-
 arch/hexagon/kernel/ptrace.c       |   4 +-
 arch/sh/Makefile                   |   8 +-
 arch/sh/boot/compressed/.gitignore |   5 +
 arch/x86/entry/vdso/Makefile       |   4 -
 scripts/Kbuild.include             | 106 ++++++++--
 scripts/Makefile.asm-generic       |   3 +
 scripts/Makefile.build             |  25 +--
 scripts/Makefile.headersinst       |  10 +-
 scripts/Makefile.help              |   3 -
 scripts/Makefile.host              |  12 --
 scripts/Makefile.lib               |  34 +---
 scripts/Makefile.modpost           |   3 +-
 scripts/link-vmlinux.sh            |  15 +-
 scripts/mkcompile_h                |   7 +-
 scripts/selinux/Makefile           |   1 -
 21 files changed, 297 insertions(+), 260 deletions(-)
 create mode 100644 arch/frv/kernel/.gitignore
 delete mode 100644 scripts/Makefile.help


-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Linus Torvalds]

On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
>
> Here are Kbuild updates for v4.15.  Please pull!

Mind checking that I got the CFLAGS_KCOV behavior right. The conflict
looked pretty simple, but somebody should check the ordering of the

   export CFLAGS_KCOV

vs the actual setting (that is now in scripts/Makefile.kcov.

It probably doesn't matter, but our makefiles are black magic.

              Linus

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Linus Torvalds]

Oh, and I forgot to ask..

On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
>
> One of the most remarkable improvements in this cycle is, Kbuild is
> now able to cache the result of shell commands.

I see the "limit it to 500 lines", but I don't see any real coherency.

So I take it that if you upgrade your gcc version, you may need to
blow this cache away manually?

Or is there something subtle going on that I've missed?

FWIW, I still think we should probably make the compiler versions etc
available to the configuration management rather than necessarily
cache them.

                Linus

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Masahiro Yamada]

Hi Linus,


2017-11-18 10:51 GMT+09:00 Linus Torvalds <torvalds@linux-foundation.org>:
> On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>>
>> Here are Kbuild updates for v4.15.  Please pull!
>
> Mind checking that I got the CFLAGS_KCOV behavior right. The conflict
> looked pretty simple, but somebody should check the ordering of the
>
>    export CFLAGS_KCOV
>
> vs the actual setting (that is now in scripts/Makefile.kcov.
>
> It probably doesn't matter, but our makefiles are black magic.
>
>               Linus


Strictly speaking, there is a case where the placement of "export"
make a difference.


-----(example1)-----
BAR := bar

FOO += $(BAR)
export FOO

BAR := bar2
---------------------


-----(example2)-----
BAR := bar

export FOO
FOO += $(BAR)

BAR := bar2
---------------------


In example1, the exported value for FOO is "bar2"
whereas, in example2, "bar".

The difference probably comes "export" implies
":=" flavor initialization if not explicitly initialized yet.
To avoid confusion, we should explicitly use either ":=" or "=", though.


Talking about the kcov case, CFLAGS_KCOV is initialized by ":="
in scripts/Malefile.kcov, so it should work as we expect.

(In my personal preference, I'd like collect
the kcov stuff into scripts/Makefile.kcof, but
it is just a matter of slight taste.)



-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Masahiro Yamada]

Hi Linus


2017-11-18 11:01 GMT+09:00 Linus Torvalds <torvalds@linux-foundation.org>:
> Oh, and I forgot to ask..
>
> On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>>
>> One of the most remarkable improvements in this cycle is, Kbuild is
>> now able to cache the result of shell commands.
>
> I see the "limit it to 500 lines", but I don't see any real coherency.

The limit is 1000 lines.
If your cache file exceeds 1000 lines,
it will be cut down to 500 lines at the next invocation of build.

I used two values 1000 and 500
so that the cache shrink operation is not triggered every time.


> So I take it that if you upgrade your gcc version, you may need to
> blow this cache away manually?

Right.  This is a limitation of this feature.
But, this limitation has existed since before.

When you upgrade your gcc,
you need to do "make clean" anyway to blow all *.o files
so that all objects are re-compiled by the new gcc.

Kbuild stores build commands in .*.cmd files,
but it cannot notice the compiler upgrade.



> Or is there something subtle going on that I've missed?
>
> FWIW, I still think we should probably make the compiler versions etc
> available to the configuration management rather than necessarily
> cache them.


Do you mean something like this?

https://lkml.org/lkml/2016/12/9/577



At first, I thought it was allowed to use a different compiler
for external modules than the one compiled the kernel.
But, Greg said we do not support that case.

Then, the runtime test of compiler capabilities is pointless,
so I think it is a possible solution.
CONFIG_CC_STACKPROTECTOR_AUTO will even more mess up the top Makefile.
https://patchwork.kernel.org/patch/9981173/


-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Linus Torvalds]

On Sun, Nov 19, 2017 at 2:40 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
>>
>> FWIW, I still think we should probably make the compiler versions etc
>> available to the configuration management rather than necessarily
>> cache them.
>
> Do you mean something like this?
>
> https://lkml.org/lkml/2016/12/9/577

Yes. With the important part not being so much that particular syntax,
but the concept of "get some config options from automation".

And the most obvious thing to do would be to just initialize a "bool"
variable by running a script where the end result is either success or
failure.

Now, the actual example:

   config COMPILER_SUPPORTS_XYZ
       bool
       option shell="gcc -XYZ"

is obviously too simplified to be realistic, because it would have to
account for the actual compiler setup, so what you really need is not
"just execute a shell command" but the equivalent of the Makefile
"cc-option" function. Aes, for "known buggy versions" you might want
to also get the actual compiler version into a config option.



Even if "cc-option" is the _only_ thing you can do (and not some kind
of "generic shell escape"), I think that would be very useful.
Wouldn't it be nice to be able to have all those Makefile things as
Kcconfig scripts - and be able to very naturally take them into
account when offering people some Kconfig options?

So you could do all the logic of not only testing what flags the
compiler supports, but then use the Kconfig language to _combine_ that
knowledge with the build options. Both in the sense of "this kernel
config option depends on the compiler supporting flag XYZ" but also in
the sense of "the use flag ABC depends on not only compiler support,
but also on whether the kernel was configured for profiling" or
whatever.

Then the actual Makefile parts would be things like

    CFLAGS-$(CONFIG_CC_SPLIT_DWARF) += -gsplit_dwarf

because the Kconfig phase would already have all the logic for whether
(a) gcc actually supports -gsplit-dwarf at all and (b) whether the
user actually asked for split debug info.

And we'd not have the performance issues that made that whole caching
thing be an issue, because the actual gcc support testing would only
happen at Kconfig time.

.. it would also make the caching rules be obvious. It's just a "if
you upgraded or changed compilers enough to be noticeable, re-do your
config ('make oldconfig')".

               Linus

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Masahiro Yamada]

Hi Linus,


2017-11-20 3:02 GMT+09:00 Linus Torvalds <torvalds@linux-foundation.org>:
> On Sun, Nov 19, 2017 at 2:40 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>>>
>>> FWIW, I still think we should probably make the compiler versions etc
>>> available to the configuration management rather than necessarily
>>> cache them.
>>
>> Do you mean something like this?
>>
>> https://lkml.org/lkml/2016/12/9/577
>
> Yes. With the important part not being so much that particular syntax,
> but the concept of "get some config options from automation".
>
> And the most obvious thing to do would be to just initialize a "bool"
> variable by running a script where the end result is either success or
> failure.
>
> Now, the actual example:
>
>    config COMPILER_SUPPORTS_XYZ
>        bool
>        option shell="gcc -XYZ"
>
> is obviously too simplified to be realistic, because it would have to
> account for the actual compiler setup, so what you really need is not
> "just execute a shell command" but the equivalent of the Makefile
> "cc-option" function. Aes, for "known buggy versions" you might want
> to also get the actual compiler version into a config option.
>
>
>
> Even if "cc-option" is the _only_ thing you can do (and not some kind
> of "generic shell escape"), I think that would be very useful.
> Wouldn't it be nice to be able to have all those Makefile things as
> Kcconfig scripts - and be able to very naturally take them into
> account when offering people some Kconfig options?
>
> So you could do all the logic of not only testing what flags the
> compiler supports, but then use the Kconfig language to _combine_ that
> knowledge with the build options. Both in the sense of "this kernel
> config option depends on the compiler supporting flag XYZ" but also in
> the sense of "the use flag ABC depends on not only compiler support,
> but also on whether the kernel was configured for profiling" or
> whatever.
>
> Then the actual Makefile parts would be things like
>
>     CFLAGS-$(CONFIG_CC_SPLIT_DWARF) += -gsplit_dwarf
>
> because the Kconfig phase would already have all the logic for whether
> (a) gcc actually supports -gsplit-dwarf at all and (b) whether the
> user actually asked for split debug info.
>
> And we'd not have the performance issues that made that whole caching
> thing be an issue, because the actual gcc support testing would only
> happen at Kconfig time.
>
> .. it would also make the caching rules be obvious. It's just a "if
> you upgraded or changed compilers enough to be noticeable, re-do your
> config ('make oldconfig')".
>
>                Linus

OK, probably this is the right direction.

Embedded folks need to agree to give CROSS_COMPILE to the Kconfig phase.





-- 
Best Regards
Masahiro Yamada