[RFC PATCH 0/2] build: Pass CC, CFLAGS, ... to tools build from main Makefile

[RFC PATCH 0/2] build: Pass CC, CFLAGS, ... to tools build from main Makefile

[Douglas Anderson]

While trying to enable CONFIG_STACK_VALIDATION in the Chrome OS build
system, Guenter Roeck noticed that the compilation of objtool was
failing in a weird way.  He tracked this down to the fact that,
although the kernel build ignores CFLAGS as of the ancient commit
69ee0b352242 ("kbuild: do not pick up CFLAGS from the environment"),
those the tools builds generally don't ignore CFLAGS.

For the most part it's OK that the tools builds don't ignore CFLAGS.
One could argue that the tools builds really _should_ honor CFLAGS
because, unlike the kernel, they don't need to be special snowflakes.
In general we're just building normal userspace programs and CFLAGS
from the environment can make sense.

...but... When a tools build is invoked from the kernel build then
CFLAGS probably won't make sense.  Specifically the kernel build
expects that "CC" is the _target_ compiler, not the _host_ compiler.
Presumably if CFLAGS is set in the current environment it will match
"CC".  That means it won't make a lot of sense to use when building
tools.

Technically we could just "fix" this by blanking out CFLAGS in the
call to the tools build, just like we've always done for LDFLAGS.
That would fix the problem that Guenter saw.  ...but we probably want
to go a step further.

In general it seems non-ideal that, even if someone overrides HOSTCC
in the kernel, that the tools build will still go back to a hardcoded
"gcc".  It seems like the compiler provided to the kernel as HOSTCC
really ought to be the compiler used in the tools build.  That seems
like it will be important for clang work.  The patches in this series
attempt to do that.

NOTE: I personally am not spending lots of time on clangification of
the kernel (though many of my colleagues are), so I haven't confirmed
that this is actually useful to clang.  It just seemed like a good
fix.  I also continue to assert that I'm not a kernel build expert, so
I've marked this as RFC in the hopes that people will not scream too
loudly if this is a stupid thing to do.  ;-)


Douglas Anderson (2):
  kbuild: Pass HOSTCC and similar to tools Makefiles
  tools build: Take CC, AS, and LD from the command line

 Makefile                       | 30 ++++++++++++++++++++++++++++--
 tools/scripts/Makefile.include | 22 ++++++++++++++++++++--
 2 files changed, 48 insertions(+), 4 deletions(-)

-- 
2.14.2.920.gcf0c67979c-goog

[RFC PATCH 1/2] kbuild: Pass HOSTCC and similar to tools Makefiles

[Douglas Anderson]

The main Linux Makefiles and the tools sub-Makefiles have different
conventions for passing in CC / CFLAGS.

Here's brief summary for the kernel:
* CC: target C compiler (must be passed as an argument to make to
  override)
* HOSTCC: host C compiler (must be passed as an argument to make to
  override)
* CFLAGS: ignored (kernel manages its own CFLAGS)
* KCFLAGS: extra cflags for the target (expected as an env var)
* HOSTCFLAGS: host C compiler flags (not modifiable by the caller of
  make)

Here's a brief summary for the tools:
* CC: host C compiler (must be passed as an argument to make to
  override)
* CFLAGS: base arguments for the host C compiler which are added to by
  sub builds (expected as an env var)

When the main kernel Makefile calls into the tools Makefile, it should
adapt things from its syntax to the tools Makefile syntax.

If we don't do this, we have a few issues:
* If someone wants to user another compiler (like clang, maybe) for
  hostcc then the tools will still be compiled with gcc.
* If you happen to be building and you left ${CFLAGS} set to something
  random (maybe it matches ${CC}, the _target_ C compiler, not the
  _host_ C compiler) then this random value will be used to compile
  the tools.

In any case, it seems like it makes sense to pass CC, CFLAGS, and
similar properly into the tools Makefile.

NOTE: in order to do this properly, we need to add some new
definitions of HOSTAS and HOSTLD into the main kernel Makefile.  If we
don't do this and someone overrides "AS" or "LD" on the command line
then those (which were intended for the target) will be used to build
host side tools.  We also make up some imaginary "HOSTASFLAGS" and
"HOSTLDFLAGS".  Someone could specify these, but if they don't at
least these blank variables will properly clobber ASFLAGS and LDFLAGS
from the kernel.

This was discovered in the Chrome OS build system where CFLAGS (an
environment variable) was accidentally left pointing to flags that
would be an appropriate match to CC.  The kernel didn't use these
CFLAGS so it was never an issue.  Turning on STACKVALIDATION, however,
suddenly invoked a tools build that blew up.

Signed-off-by: Douglas Anderson <dianders@chromium.org>
---

 Makefile | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index cf007a31d575..0d3af0677d88 100644
--- a/Makefile
+++ b/Makefile
@@ -298,7 +298,9 @@ HOST_LFS_CFLAGS := $(shell getconf LFS_CFLAGS)
 HOST_LFS_LDFLAGS := $(shell getconf LFS_LDFLAGS)
 HOST_LFS_LIBS := $(shell getconf LFS_LIBS)
 
+HOSTAS       = as
 HOSTCC       = gcc
+HOSTLD       = ld
 HOSTCXX      = g++
 HOSTCFLAGS   := -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 \
 		-fomit-frame-pointer -std=gnu89 $(HOST_LFS_CFLAGS)
@@ -1616,11 +1618,35 @@ image_name:
 # Clear a bunch of variables before executing the submake
 tools/: FORCE
 	$(Q)mkdir -p $(objtree)/tools
-	$(Q)$(MAKE) LDFLAGS= MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" O=$(abspath $(objtree)) subdir=tools -C $(src)/tools/
+	$(Q)ASFLAGS="$(HOSTASFLAGS)"\
+		LDFLAGS="$(HOSTLDFLAGS)" \
+		CFLAGS="$(HOSTCFLAGS)" \
+		CXXFLAGS="$(HOSTCXXFLAGS)" \
+		$(MAKE) \
+		AS="$(HOSTAS)" \
+		CC="$(HOSTCC)" \
+		CXX="$(HOSTCXX)" \
+		LD="$(HOSTLD)" \
+		MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" \
+		O=$(abspath $(objtree)) \
+		subdir=tools \
+		-C $(src)/tools/
 
 tools/%: FORCE
 	$(Q)mkdir -p $(objtree)/tools
-	$(Q)$(MAKE) LDFLAGS= MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" O=$(abspath $(objtree)) subdir=tools -C $(src)/tools/ $*
+	$(Q)ASFLAGS="$(HOSTASFLAGS)"\
+		LDFLAGS="$(HOSTLDFLAGS)" \
+		CFLAGS="$(HOSTCFLAGS)" \
+		CXXFLAGS="$(HOSTCXXFLAGS)" \
+		$(MAKE) \
+		AS="$(HOSTAS)" \
+		CC="$(HOSTCC)" \
+		CXX="$(HOSTCXX)" \
+		LD="$(HOSTLD)" \
+		MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" \
+		O=$(abspath $(objtree)) \
+		subdir=tools \
+		-C $(src)/tools/ $*
 
 # Single targets
 # ---------------------------------------------------------------------------
-- 
2.14.2.920.gcf0c67979c-goog

[RFC PATCH 2/2] tools build: Take CC, AS, and LD from the command line

[Douglas Anderson]

If the top-level tools build is called to build a tool and is passed
CC, AS, or LD on the command line then someone wants to use a
different compiler, assembler, or linker.  Let's honor that.

This together with the change ("kbuild: Pass HOSTCC and similar to
tools Makefiles") allows us to properly get the HOST C Compiler used
when the main kernel Makefile calls into the tools.

Signed-off-by: Douglas Anderson <dianders@chromium.org>
---

 tools/scripts/Makefile.include | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
index 9dc8f078a83c..00261848ec54 100644
--- a/tools/scripts/Makefile.include
+++ b/tools/scripts/Makefile.include
@@ -10,6 +10,20 @@ endif
 endif
 endif
 
+SUBMAKE_ARGS :=
+
+# If CC, LD, or AR were passed on the command line, pass them through to the
+# command line of the sub-tools.
+ifeq ($(origin CC), command line)
+	SUBMAKE_ARGS += "CC=$(CC)"
+endif
+ifeq ($(origin LD), command line)
+	SUBMAKE_ARGS += "LD=$(LD)"
+endif
+ifeq ($(origin AR), command line)
+	SUBMAKE_ARGS += "AR=$(ar)"
+endif
+
 # check that the output directory actually exists
 ifneq ($(OUTPUT),)
 OUTDIR := $(realpath $(OUTPUT))
@@ -71,7 +85,9 @@ endif
 #
 descend = \
 	+mkdir -p $(OUTPUT)$(1) && \
-	$(MAKE) $(COMMAND_O) subdir=$(if $(subdir),$(subdir)/$(1),$(1)) $(PRINT_DIR) -C $(1) $(2)
+	$(MAKE) $(SUBMAKE_ARGS) $(COMMAND_O) \
+		subdir=$(if $(subdir),$(subdir)/$(1),$(1)) \
+		$(PRINT_DIR) -C $(1) $(2)
 
 QUIET_SUBDIR0  = +$(MAKE) $(COMMAND_O) -C # space to separate -C and subdir
 QUIET_SUBDIR1  =
@@ -94,7 +110,9 @@ ifneq ($(silent),1)
 	descend = \
 		+@echo	       '  DESCEND  '$(1); \
 		mkdir -p $(OUTPUT)$(1) && \
-		$(MAKE) $(COMMAND_O) subdir=$(if $(subdir),$(subdir)/$(1),$(1)) $(PRINT_DIR) -C $(1) $(2)
+		$(MAKE) $(SUBMAKE_ARGS) $(COMMAND_O) \
+			subdir=$(if $(subdir),$(subdir)/$(1),$(1)) \
+			$(PRINT_DIR) -C $(1) $(2)
 
 	QUIET_CLEAN    = @printf '  CLEAN    %s\n' $1;
 	QUIET_INSTALL  = @printf '  INSTALL  %s\n' $1;
-- 
2.14.2.920.gcf0c67979c-goog

Re: [RFC PATCH 1/2] kbuild: Pass HOSTCC and similar to tools Makefiles

[Guenter Roeck]

On Tue, Oct 3, 2017 at 1:48 PM, Douglas Anderson <dianders@chromium.org> wrote:
> The main Linux Makefiles and the tools sub-Makefiles have different
> conventions for passing in CC / CFLAGS.
>
> Here's brief summary for the kernel:
> * CC: target C compiler (must be passed as an argument to make to
>   override)
> * HOSTCC: host C compiler (must be passed as an argument to make to
>   override)
> * CFLAGS: ignored (kernel manages its own CFLAGS)
> * KCFLAGS: extra cflags for the target (expected as an env var)
> * HOSTCFLAGS: host C compiler flags (not modifiable by the caller of
>   make)
>
> Here's a brief summary for the tools:
> * CC: host C compiler (must be passed as an argument to make to
>   override)
> * CFLAGS: base arguments for the host C compiler which are added to by
>   sub builds (expected as an env var)
>
> When the main kernel Makefile calls into the tools Makefile, it should
> adapt things from its syntax to the tools Makefile syntax.
>
> If we don't do this, we have a few issues:
> * If someone wants to user another compiler (like clang, maybe) for
>   hostcc then the tools will still be compiled with gcc.
> * If you happen to be building and you left ${CFLAGS} set to something
>   random (maybe it matches ${CC}, the _target_ C compiler, not the
>   _host_ C compiler) then this random value will be used to compile
>   the tools.
>
> In any case, it seems like it makes sense to pass CC, CFLAGS, and
> similar properly into the tools Makefile.
>
> NOTE: in order to do this properly, we need to add some new
> definitions of HOSTAS and HOSTLD into the main kernel Makefile.  If we
> don't do this and someone overrides "AS" or "LD" on the command line
> then those (which were intended for the target) will be used to build
> host side tools.  We also make up some imaginary "HOSTASFLAGS" and
> "HOSTLDFLAGS".  Someone could specify these, but if they don't at
> least these blank variables will properly clobber ASFLAGS and LDFLAGS
> from the kernel.
>
> This was discovered in the Chrome OS build system where CFLAGS (an
> environment variable) was accidentally left pointing to flags that
> would be an appropriate match to CC.  The kernel didn't use these
> CFLAGS so it was never an issue.  Turning on STACKVALIDATION, however,
> suddenly invoked a tools build that blew up.
>
> Signed-off-by: Douglas Anderson <dianders@chromium.org>

Still not sure if I'd want to go that far. Just clearing out CFLAGS
would have fixed my problem, and I start to realize that I am more of
a minimalist when it comes to kernel changes.

Anyway, not that it means much,

Reviewed-by: Guenter Roeck <groeck@chromium.org>
Tested-by: Guenter Roeck <groeck@chromium.org>

[ I didn't test all possible flag combinations, only a few of them,
but I did make sure that "make allmodconfig; CFLAGS=-junk make" no
longer fails. ]

Guenter

> ---
>
>  Makefile | 30 ++++++++++++++++++++++++++++--
>  1 file changed, 28 insertions(+), 2 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index cf007a31d575..0d3af0677d88 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -298,7 +298,9 @@ HOST_LFS_CFLAGS := $(shell getconf LFS_CFLAGS)
>  HOST_LFS_LDFLAGS := $(shell getconf LFS_LDFLAGS)
>  HOST_LFS_LIBS := $(shell getconf LFS_LIBS)
>
> +HOSTAS       = as
>  HOSTCC       = gcc
> +HOSTLD       = ld
>  HOSTCXX      = g++
>  HOSTCFLAGS   := -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 \
>                 -fomit-frame-pointer -std=gnu89 $(HOST_LFS_CFLAGS)
> @@ -1616,11 +1618,35 @@ image_name:
>  # Clear a bunch of variables before executing the submake
>  tools/: FORCE
>         $(Q)mkdir -p $(objtree)/tools
> -       $(Q)$(MAKE) LDFLAGS= MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" O=$(abspath $(objtree)) subdir=tools -C $(src)/tools/
> +       $(Q)ASFLAGS="$(HOSTASFLAGS)"\
> +               LDFLAGS="$(HOSTLDFLAGS)" \
> +               CFLAGS="$(HOSTCFLAGS)" \
> +               CXXFLAGS="$(HOSTCXXFLAGS)" \
> +               $(MAKE) \
> +               AS="$(HOSTAS)" \
> +               CC="$(HOSTCC)" \
> +               CXX="$(HOSTCXX)" \
> +               LD="$(HOSTLD)" \
> +               MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" \
> +               O=$(abspath $(objtree)) \
> +               subdir=tools \
> +               -C $(src)/tools/
>
>  tools/%: FORCE
>         $(Q)mkdir -p $(objtree)/tools
> -       $(Q)$(MAKE) LDFLAGS= MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" O=$(abspath $(objtree)) subdir=tools -C $(src)/tools/ $*
> +       $(Q)ASFLAGS="$(HOSTASFLAGS)"\
> +               LDFLAGS="$(HOSTLDFLAGS)" \
> +               CFLAGS="$(HOSTCFLAGS)" \
> +               CXXFLAGS="$(HOSTCXXFLAGS)" \
> +               $(MAKE) \
> +               AS="$(HOSTAS)" \
> +               CC="$(HOSTCC)" \
> +               CXX="$(HOSTCXX)" \
> +               LD="$(HOSTLD)" \
> +               MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" \
> +               O=$(abspath $(objtree)) \
> +               subdir=tools \
> +               -C $(src)/tools/ $*
>
>  # Single targets
>  # ---------------------------------------------------------------------------
> --
> 2.14.2.920.gcf0c67979c-goog
>

Re: [RFC PATCH 2/2] tools build: Take CC, AS, and LD from the command line

[Guenter Roeck]

On Tue, Oct 3, 2017 at 1:48 PM, Douglas Anderson <dianders@chromium.org> wrote:
> If the top-level tools build is called to build a tool and is passed
> CC, AS, or LD on the command line then someone wants to use a
> different compiler, assembler, or linker.  Let's honor that.
>
> This together with the change ("kbuild: Pass HOSTCC and similar to
> tools Makefiles") allows us to properly get the HOST C Compiler used
> when the main kernel Makefile calls into the tools.
>
> Signed-off-by: Douglas Anderson <dianders@chromium.org>
> ---
>
>  tools/scripts/Makefile.include | 22 ++++++++++++++++++++--
>  1 file changed, 20 insertions(+), 2 deletions(-)
>
> diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
> index 9dc8f078a83c..00261848ec54 100644
> --- a/tools/scripts/Makefile.include
> +++ b/tools/scripts/Makefile.include
> @@ -10,6 +10,20 @@ endif
>  endif
>  endif
>
> +SUBMAKE_ARGS :=
> +
> +# If CC, LD, or AR were passed on the command line, pass them through to the
> +# command line of the sub-tools.
> +ifeq ($(origin CC), command line)
> +       SUBMAKE_ARGS += "CC=$(CC)"
> +endif
> +ifeq ($(origin LD), command line)
> +       SUBMAKE_ARGS += "LD=$(LD)"
> +endif
> +ifeq ($(origin AR), command line)
> +       SUBMAKE_ARGS += "AR=$(ar)"

s/ar/AR/ ?

> +endif
> +
>  # check that the output directory actually exists
>  ifneq ($(OUTPUT),)
>  OUTDIR := $(realpath $(OUTPUT))
> @@ -71,7 +85,9 @@ endif
>  #
>  descend = \
>         +mkdir -p $(OUTPUT)$(1) && \
> -       $(MAKE) $(COMMAND_O) subdir=$(if $(subdir),$(subdir)/$(1),$(1)) $(PRINT_DIR) -C $(1) $(2)
> +       $(MAKE) $(SUBMAKE_ARGS) $(COMMAND_O) \
> +               subdir=$(if $(subdir),$(subdir)/$(1),$(1)) \
> +               $(PRINT_DIR) -C $(1) $(2)
>
>  QUIET_SUBDIR0  = +$(MAKE) $(COMMAND_O) -C # space to separate -C and subdir
>  QUIET_SUBDIR1  =
> @@ -94,7 +110,9 @@ ifneq ($(silent),1)
>         descend = \
>                 +@echo         '  DESCEND  '$(1); \
>                 mkdir -p $(OUTPUT)$(1) && \
> -               $(MAKE) $(COMMAND_O) subdir=$(if $(subdir),$(subdir)/$(1),$(1)) $(PRINT_DIR) -C $(1) $(2)
> +               $(MAKE) $(SUBMAKE_ARGS) $(COMMAND_O) \
> +                       subdir=$(if $(subdir),$(subdir)/$(1),$(1)) \
> +                       $(PRINT_DIR) -C $(1) $(2)
>
>         QUIET_CLEAN    = @printf '  CLEAN    %s\n' $1;
>         QUIET_INSTALL  = @printf '  INSTALL  %s\n' $1;
> --
> 2.14.2.920.gcf0c67979c-goog
>

Re: [RFC PATCH 1/2] kbuild: Pass HOSTCC and similar to tools Makefiles

[Masahiro Yamada]

2017-10-04 5:48 GMT+09:00 Douglas Anderson <dianders@chromium.org>:
> The main Linux Makefiles and the tools sub-Makefiles have different
> conventions for passing in CC / CFLAGS.
>
> Here's brief summary for the kernel:
> * CC: target C compiler (must be passed as an argument to make to
>   override)
> * HOSTCC: host C compiler (must be passed as an argument to make to
>   override)
> * CFLAGS: ignored (kernel manages its own CFLAGS)
> * KCFLAGS: extra cflags for the target (expected as an env var)
> * HOSTCFLAGS: host C compiler flags (not modifiable by the caller of
>   make)
>
> Here's a brief summary for the tools:
> * CC: host C compiler (must be passed as an argument to make to
>   override)
> * CFLAGS: base arguments for the host C compiler which are added to by
>   sub builds (expected as an env var)
>
> When the main kernel Makefile calls into the tools Makefile, it should
> adapt things from its syntax to the tools Makefile syntax.
>
> If we don't do this, we have a few issues:
> * If someone wants to user another compiler (like clang, maybe) for
>   hostcc then the tools will still be compiled with gcc.
> * If you happen to be building and you left ${CFLAGS} set to something
>   random (maybe it matches ${CC}, the _target_ C compiler, not the
>   _host_ C compiler) then this random value will be used to compile
>   the tools.
>
> In any case, it seems like it makes sense to pass CC, CFLAGS, and
> similar properly into the tools Makefile.
>
> NOTE: in order to do this properly, we need to add some new
> definitions of HOSTAS and HOSTLD into the main kernel Makefile.  If we
> don't do this and someone overrides "AS" or "LD" on the command line
> then those (which were intended for the target) will be used to build
> host side tools.  We also make up some imaginary "HOSTASFLAGS" and
> "HOSTLDFLAGS".  Someone could specify these, but if they don't at
> least these blank variables will properly clobber ASFLAGS and LDFLAGS
> from the kernel.
>
> This was discovered in the Chrome OS build system where CFLAGS (an
> environment variable) was accidentally left pointing to flags that
> would be an appropriate match to CC.  The kernel didn't use these
> CFLAGS so it was never an issue.  Turning on STACKVALIDATION, however,
> suddenly invoked a tools build that blew up.
>
> Signed-off-by: Douglas Anderson <dianders@chromium.org>
> ---
>
>  Makefile | 30 ++++++++++++++++++++++++++++--
>  1 file changed, 28 insertions(+), 2 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index cf007a31d575..0d3af0677d88 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -298,7 +298,9 @@ HOST_LFS_CFLAGS := $(shell getconf LFS_CFLAGS)
>  HOST_LFS_LDFLAGS := $(shell getconf LFS_LDFLAGS)
>  HOST_LFS_LIBS := $(shell getconf LFS_LIBS)
>
> +HOSTAS       = as
>  HOSTCC       = gcc
> +HOSTLD       = ld
>  HOSTCXX      = g++
>  HOSTCFLAGS   := -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 \
>                 -fomit-frame-pointer -std=gnu89 $(HOST_LFS_CFLAGS)
> @@ -1616,11 +1618,35 @@ image_name:
>  # Clear a bunch of variables before executing the submake
>  tools/: FORCE
>         $(Q)mkdir -p $(objtree)/tools
> -       $(Q)$(MAKE) LDFLAGS= MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" O=$(abspath $(objtree)) subdir=tools -C $(src)/tools/
> +       $(Q)ASFLAGS="$(HOSTASFLAGS)"\
> +               LDFLAGS="$(HOSTLDFLAGS)" \
> +               CFLAGS="$(HOSTCFLAGS)" \
> +               CXXFLAGS="$(HOSTCXXFLAGS)" \
> +               $(MAKE) \
> +               AS="$(HOSTAS)" \
> +               CC="$(HOSTCC)" \
> +               CXX="$(HOSTCXX)" \
> +               LD="$(HOSTLD)" \
> +               MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" \
> +               O=$(abspath $(objtree)) \
> +               subdir=tools \
> +               -C $(src)/tools/
>
>  tools/%: FORCE
>         $(Q)mkdir -p $(objtree)/tools
> -       $(Q)$(MAKE) LDFLAGS= MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" O=$(abspath $(objtree)) subdir=tools -C $(src)/tools/ $*
> +       $(Q)ASFLAGS="$(HOSTASFLAGS)"\
> +               LDFLAGS="$(HOSTLDFLAGS)" \
> +               CFLAGS="$(HOSTCFLAGS)" \
> +               CXXFLAGS="$(HOSTCXXFLAGS)" \
> +               $(MAKE) \
> +               AS="$(HOSTAS)" \
> +               CC="$(HOSTCC)" \
> +               CXX="$(HOSTCXX)" \
> +               LD="$(HOSTLD)" \
> +               MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" \
> +               O=$(abspath $(objtree)) \
> +               subdir=tools \
> +               -C $(src)/tools/ $*
>

Please do not do this.



CC:     for building tools that run on the target machine
HOSTCC: for building tools that run on the build machine


IMHO, this should be consistent to avoid confusion.


Grepping CROSS_COMPILE under tools/ directory,
I see many Makefile expect CC for the target system.

For ex, tools/croup/Makefile

CC = $(CROSS_COMPILE)gcc



Why don't you fix tools/objtool/Makefile ?





-- 
Best Regards
Masahiro Yamada

Re: [RFC PATCH 1/2] kbuild: Pass HOSTCC and similar to tools Makefiles

[Guenter Roeck]

On Tue, Oct 3, 2017 at 8:20 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2017-10-04 5:48 GMT+09:00 Douglas Anderson <dianders@chromium.org>:
>> The main Linux Makefiles and the tools sub-Makefiles have different
>> conventions for passing in CC / CFLAGS.
>>
>> Here's brief summary for the kernel:
>> * CC: target C compiler (must be passed as an argument to make to
>>   override)
>> * HOSTCC: host C compiler (must be passed as an argument to make to
>>   override)
>> * CFLAGS: ignored (kernel manages its own CFLAGS)
>> * KCFLAGS: extra cflags for the target (expected as an env var)
>> * HOSTCFLAGS: host C compiler flags (not modifiable by the caller of
>>   make)
>>
>> Here's a brief summary for the tools:
>> * CC: host C compiler (must be passed as an argument to make to
>>   override)
>> * CFLAGS: base arguments for the host C compiler which are added to by
>>   sub builds (expected as an env var)
>>
>> When the main kernel Makefile calls into the tools Makefile, it should
>> adapt things from its syntax to the tools Makefile syntax.
>>
>> If we don't do this, we have a few issues:
>> * If someone wants to user another compiler (like clang, maybe) for
>>   hostcc then the tools will still be compiled with gcc.
>> * If you happen to be building and you left ${CFLAGS} set to something
>>   random (maybe it matches ${CC}, the _target_ C compiler, not the
>>   _host_ C compiler) then this random value will be used to compile
>>   the tools.
>>
>> In any case, it seems like it makes sense to pass CC, CFLAGS, and
>> similar properly into the tools Makefile.
>>
>> NOTE: in order to do this properly, we need to add some new
>> definitions of HOSTAS and HOSTLD into the main kernel Makefile.  If we
>> don't do this and someone overrides "AS" or "LD" on the command line
>> then those (which were intended for the target) will be used to build
>> host side tools.  We also make up some imaginary "HOSTASFLAGS" and
>> "HOSTLDFLAGS".  Someone could specify these, but if they don't at
>> least these blank variables will properly clobber ASFLAGS and LDFLAGS
>> from the kernel.
>>
>> This was discovered in the Chrome OS build system where CFLAGS (an
>> environment variable) was accidentally left pointing to flags that
>> would be an appropriate match to CC.  The kernel didn't use these
>> CFLAGS so it was never an issue.  Turning on STACKVALIDATION, however,
>> suddenly invoked a tools build that blew up.
>>
>> Signed-off-by: Douglas Anderson <dianders@chromium.org>
>> ---
>>
>>  Makefile | 30 ++++++++++++++++++++++++++++--
>>  1 file changed, 28 insertions(+), 2 deletions(-)
>>
>> diff --git a/Makefile b/Makefile
>> index cf007a31d575..0d3af0677d88 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -298,7 +298,9 @@ HOST_LFS_CFLAGS := $(shell getconf LFS_CFLAGS)
>>  HOST_LFS_LDFLAGS := $(shell getconf LFS_LDFLAGS)
>>  HOST_LFS_LIBS := $(shell getconf LFS_LIBS)
>>
>> +HOSTAS       = as
>>  HOSTCC       = gcc
>> +HOSTLD       = ld
>>  HOSTCXX      = g++
>>  HOSTCFLAGS   := -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 \
>>                 -fomit-frame-pointer -std=gnu89 $(HOST_LFS_CFLAGS)
>> @@ -1616,11 +1618,35 @@ image_name:
>>  # Clear a bunch of variables before executing the submake
>>  tools/: FORCE
>>         $(Q)mkdir -p $(objtree)/tools
>> -       $(Q)$(MAKE) LDFLAGS= MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" O=$(abspath $(objtree)) subdir=tools -C $(src)/tools/
>> +       $(Q)ASFLAGS="$(HOSTASFLAGS)"\
>> +               LDFLAGS="$(HOSTLDFLAGS)" \
>> +               CFLAGS="$(HOSTCFLAGS)" \
>> +               CXXFLAGS="$(HOSTCXXFLAGS)" \
>> +               $(MAKE) \
>> +               AS="$(HOSTAS)" \
>> +               CC="$(HOSTCC)" \
>> +               CXX="$(HOSTCXX)" \
>> +               LD="$(HOSTLD)" \
>> +               MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" \
>> +               O=$(abspath $(objtree)) \
>> +               subdir=tools \
>> +               -C $(src)/tools/
>>
>>  tools/%: FORCE
>>         $(Q)mkdir -p $(objtree)/tools
>> -       $(Q)$(MAKE) LDFLAGS= MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" O=$(abspath $(objtree)) subdir=tools -C $(src)/tools/ $*
>> +       $(Q)ASFLAGS="$(HOSTASFLAGS)"\
>> +               LDFLAGS="$(HOSTLDFLAGS)" \
>> +               CFLAGS="$(HOSTCFLAGS)" \
>> +               CXXFLAGS="$(HOSTCXXFLAGS)" \
>> +               $(MAKE) \
>> +               AS="$(HOSTAS)" \
>> +               CC="$(HOSTCC)" \
>> +               CXX="$(HOSTCXX)" \
>> +               LD="$(HOSTLD)" \
>> +               MAKEFLAGS="$(tools_silent) $(filter --j% -j,$(MAKEFLAGS))" \
>> +               O=$(abspath $(objtree)) \
>> +               subdir=tools \
>> +               -C $(src)/tools/ $*
>>
>
> Please do not do this.
>
>
>
> CC:     for building tools that run on the target machine
> HOSTCC: for building tools that run on the build machine
>
>
> IMHO, this should be consistent to avoid confusion.
>
>
> Grepping CROSS_COMPILE under tools/ directory,
> I see many Makefile expect CC for the target system.
>
> For ex, tools/croup/Makefile
>
> CC = $(CROSS_COMPILE)gcc
>
>
>
> Why don't you fix tools/objtool/Makefile ?
>
>

Interesting question. You are right, most tools are target tools, not
host tools. Maybe it _would_ make sense to use HOSTCC/HOSTCFLAGS to
build it. Copying Josh for input.

Guenter

Re: [RFC PATCH 2/2] tools build: Take CC, AS, and LD from the command line

[Doug Anderson]

Hi

On Tue, Oct 3, 2017 at 8:06 PM, Guenter Roeck <groeck@google.com> wrote:
> On Tue, Oct 3, 2017 at 1:48 PM, Douglas Anderson <dianders@chromium.org> wrote:
>> If the top-level tools build is called to build a tool and is passed
>> CC, AS, or LD on the command line then someone wants to use a
>> different compiler, assembler, or linker.  Let's honor that.
>>
>> This together with the change ("kbuild: Pass HOSTCC and similar to
>> tools Makefiles") allows us to properly get the HOST C Compiler used
>> when the main kernel Makefile calls into the tools.
>>
>> Signed-off-by: Douglas Anderson <dianders@chromium.org>
>> ---
>>
>>  tools/scripts/Makefile.include | 22 ++++++++++++++++++++--
>>  1 file changed, 20 insertions(+), 2 deletions(-)
>>
>> diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
>> index 9dc8f078a83c..00261848ec54 100644
>> --- a/tools/scripts/Makefile.include
>> +++ b/tools/scripts/Makefile.include
>> @@ -10,6 +10,20 @@ endif
>>  endif
>>  endif
>>
>> +SUBMAKE_ARGS :=
>> +
>> +# If CC, LD, or AR were passed on the command line, pass them through to the
>> +# command line of the sub-tools.
>> +ifeq ($(origin CC), command line)
>> +       SUBMAKE_ARGS += "CC=$(CC)"
>> +endif
>> +ifeq ($(origin LD), command line)
>> +       SUBMAKE_ARGS += "LD=$(LD)"
>> +endif
>> +ifeq ($(origin AR), command line)
>> +       SUBMAKE_ARGS += "AR=$(ar)"
>
> s/ar/AR/ ?

Oops!  I guess you can tell that I didn't test overriding AR, just CC and LD.

...it sounds as if my patch series was overall misguided anyway as per
<http://patchwork.kernel.org/patch/9983535>, so I won't plan to
re-spin.  Thank you for the review, though!

-Doug

Re: [RFC PATCH 1/2] kbuild: Pass HOSTCC and similar to tools Makefiles

[Doug Anderson]

Hi,

On Tue, Oct 3, 2017 at 9:07 PM, Guenter Roeck <groeck@google.com> wrote:
> On Tue, Oct 3, 2017 at 8:20 PM, Masahiro Yamada
>> CC:     for building tools that run on the target machine
>> HOSTCC: for building tools that run on the build machine
>>
>>
>> IMHO, this should be consistent to avoid confusion.
>>
>>
>> Grepping CROSS_COMPILE under tools/ directory,
>> I see many Makefile expect CC for the target system.
>>
>> For ex, tools/croup/Makefile
>>
>> CC = $(CROSS_COMPILE)gcc
>>
>>
>>
>> Why don't you fix tools/objtool/Makefile ?
>>
>>
>
> Interesting question. You are right, most tools are target tools, not
> host tools. Maybe it _would_ make sense to use HOSTCC/HOSTCFLAGS to
> build it. Copying Josh for input.

Ah, good point.  I'm not terribly familiar with the things under
tools, so when I saw "objtool" I assumed it was following a standard
pattern for tools.

I can submit a patch for objtool's Makefile.  ...or, if it makes more
sense, I can submit a patch to move "objtool" to the scripts
directory.  ...or I can butt out of it.  :)  Which would folks prefer?

-Doug

Re: [RFC PATCH 1/2] kbuild: Pass HOSTCC and similar to tools Makefiles

[Josh Poimboeuf]

On Wed, Oct 04, 2017 at 09:09:51AM -0700, Doug Anderson wrote:
> Hi,
> 
> On Tue, Oct 3, 2017 at 9:07 PM, Guenter Roeck <groeck@google.com> wrote:
> > On Tue, Oct 3, 2017 at 8:20 PM, Masahiro Yamada
> >> CC:     for building tools that run on the target machine
> >> HOSTCC: for building tools that run on the build machine
> >>
> >>
> >> IMHO, this should be consistent to avoid confusion.
> >>
> >>
> >> Grepping CROSS_COMPILE under tools/ directory,
> >> I see many Makefile expect CC for the target system.
> >>
> >> For ex, tools/croup/Makefile
> >>
> >> CC = $(CROSS_COMPILE)gcc
> >>
> >>
> >>
> >> Why don't you fix tools/objtool/Makefile ?
> >>
> >>
> >
> > Interesting question. You are right, most tools are target tools, not
> > host tools. Maybe it _would_ make sense to use HOSTCC/HOSTCFLAGS to
> > build it. Copying Josh for input.

Yes, converting objtool to use HOSTCC/HOSTCFLAGS etc would be a good
improvement.  It has a dependency on libsubcmd, so
tools/lib/subcmd/Makefile will also need to be changed (and be careful
not to break perf in the process).

> Ah, good point.  I'm not terribly familiar with the things under
> tools, so when I saw "objtool" I assumed it was following a standard
> pattern for tools.
> 
> I can submit a patch for objtool's Makefile.  ...or, if it makes more
> sense, I can submit a patch to move "objtool" to the scripts
> directory.  ...or I can butt out of it.  :)  Which would folks prefer?

In some ways objtool would be a more natural fit in the scripts dir.
However, we decided to put it in tools because it might be useful
outside of the kernel and it's much easier to extract it from the kernel
that way.  Also, it has a shared dependency on libsubcmd with perf,
which is also in tools.

-- 
Josh