[PATCH 0/2] objtool: fix libelf detection in non-standard locations

[PATCH 0/2] objtool: fix libelf detection in non-standard locations

[Robin Jarry]

When libelf headers and libs are not in standard locations, the check
fails and host flags are ignored.

I have encountered this issue when building linux into the buildroot
framework. host-* versions of libs may be compiled and installed in a
host staging dir removing the need to install them on the build system.

Comments are welcome.

Regards,

Robin Jarry (2):
  kbuild: introduce HOST_EXTRALDFLAGS
  objtool: use host flags to detect if libelf is missing

 Makefile | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

-- 
2.11.0

[PATCH 1/2] kbuild: introduce HOST_EXTRALDFLAGS

[Robin Jarry]

Add a new user specifiable variable to provide specific linker options
(-L, -rpath, ...) for host programs build.

Append these options to HOSTLDFLAGS.

Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Ingo Molnar <mingo@kernel.org>
---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index d9cf3a40eda9..e9609319d2b8 100644
--- a/Makefile
+++ b/Makefile
@@ -365,6 +365,7 @@ HOSTCFLAGS   := -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 \
 		-fomit-frame-pointer -std=gnu89 $(HOST_LFS_CFLAGS)
 HOSTCXXFLAGS := -O2 $(HOST_LFS_CFLAGS)
 HOSTLDFLAGS  := $(HOST_LFS_LDFLAGS)
+HOSTLDFLAGS  += $(HOST_EXTRALDFLAGS)
 HOST_LOADLIBES := $(HOST_LFS_LIBS)
 
 # Make variables (CC, etc...)
-- 
2.11.0

[PATCH 2/2] objtool: use host flags to detect if libelf is missing

[Robin Jarry]

When libelf headers and libs are not found and CONFIG_UNWINDER_ORC is
selected, there is a make error introduced by
commit 3dd40cb320fe ("objtool: Upgrade libelf-devel warning to error for
CONFIG_ORC_UNWINDER").

Host headers and libs may be in a non-standard location and the check is
performed without host flags. Make sure to use host flags for the check
(they will be used later when building objtool anyway).

Fixes: 3b27a0c85d70 ("objtool: Detect and warn if libelf is missing and don't break the build")
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Ingo Molnar <mingo@kernel.org>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index e9609319d2b8..4e8c667b9f3f 100644
--- a/Makefile
+++ b/Makefile
@@ -955,7 +955,7 @@ export mod_sign_cmd
 
 ifdef CONFIG_STACK_VALIDATION
   has_libelf := $(call try-run,\
-		echo "int main() {}" | $(HOSTCC) -xc -o /dev/null -lelf -,1,0)
+		echo "int main() {}" | $(HOSTCC) $(host_clfags) $(HOSTLDFLAGS) -xc -o /dev/null -lelf -,1,0)
   ifeq ($(has_libelf),1)
     objtool_target := tools/objtool FORCE
   else
-- 
2.11.0

Re: [PATCH 2/2] objtool: use host flags to detect if libelf is missing

[Josh Poimboeuf]

On Fri, Feb 23, 2018 at 03:22:15PM +0100, Robin Jarry wrote:
> When libelf headers and libs are not found and CONFIG_UNWINDER_ORC is
> selected, there is a make error introduced by
> commit 3dd40cb320fe ("objtool: Upgrade libelf-devel warning to error for
> CONFIG_ORC_UNWINDER").
> 
> Host headers and libs may be in a non-standard location and the check is
> performed without host flags. Make sure to use host flags for the check
> (they will be used later when building objtool anyway).
> 
> Fixes: 3b27a0c85d70 ("objtool: Detect and warn if libelf is missing and don't break the build")
> Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Cc: Ingo Molnar <mingo@kernel.org>
> ---
>  Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index e9609319d2b8..4e8c667b9f3f 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -955,7 +955,7 @@ export mod_sign_cmd
>  
>  ifdef CONFIG_STACK_VALIDATION
>    has_libelf := $(call try-run,\
> -		echo "int main() {}" | $(HOSTCC) -xc -o /dev/null -lelf -,1,0)
> +		echo "int main() {}" | $(HOSTCC) $(host_clfags) $(HOSTLDFLAGS) -xc -o /dev/null -lelf -,1,0)
>    ifeq ($(has_libelf),1)
>      objtool_target := tools/objtool FORCE
>    else
> -- 
> 2.11.0
> 

Hi Robin,

Did you test this?  I see some issues:

a) I don't see where the host_cflags variable comes from (and btw, it's
   misspelled...)

b) I don't see where objtool uses HOSTLDFLAGS.

-- 
Josh

Re: [PATCH 2/2] objtool: use host flags to detect if libelf is missing

[Robin Jarry]

2018-02-23, Josh Poimboeuf:
> Hi Robin,
> 
> Did you test this?  I see some issues:
> 
> a) I don't see where the host_cflags variable comes from (and btw, it's
>    misspelled...)

Hi Josh,

Shame on me... Last minute fix. I had tested this first by forcing
HOSTCC="gcc <CFLAGS> <LDFLAGS>" but I found it rather dirty. As it turns
out, my patch is not better.

> b) I don't see where objtool uses HOSTLDFLAGS.

Indeed, I also missed this (see above).

It looks like objtool does not use HOSTCFLAGS nor HOSTLDFLAGS (defined
in top Makefile). Do you see a reason why? Maybe it would be a good idea
to share these flags (adding some specific ones for objtool, if needed).

What do you think?

-- 
Robin

Re: [PATCH 2/2] objtool: use host flags to detect if libelf is missing

[Josh Poimboeuf]

On Fri, Feb 23, 2018 at 05:22:55PM +0100, Robin Jarry wrote:
> 2018-02-23, Josh Poimboeuf:
> > Hi Robin,
> > 
> > Did you test this?  I see some issues:
> > 
> > a) I don't see where the host_cflags variable comes from (and btw, it's
> >    misspelled...)
> 
> Hi Josh,
> 
> Shame on me... Last minute fix. I had tested this first by forcing
> HOSTCC="gcc <CFLAGS> <LDFLAGS>" but I found it rather dirty. As it turns
> out, my patch is not better.
> 
> > b) I don't see where objtool uses HOSTLDFLAGS.
> 
> Indeed, I also missed this (see above).
> 
> It looks like objtool does not use HOSTCFLAGS nor HOSTLDFLAGS (defined
> in top Makefile). Do you see a reason why? Maybe it would be a good idea
> to share these flags (adding some specific ones for objtool, if needed).
> 
> What do you think?

Yeah, I think it would be a good idea for objtool to use HOSTCC,
HOSTCFLAGS, and HOSTLDFLAGS.

-- 
Josh

[PATCH v2 0/3] kbuild: fix host progs build with libs in non standard locations

[Robin Jarry]

This patchset allows to build host programs that depend on external libs
installed in non standard locations (i.e. not in /usr/include, /usr/lib,
etc.).  For now, the only way is to force HOSTCC to include both the
path to the host compiler and the build flags.

I have encountered this issue when building linux into the buildroot
framework. host-* versions of libs may be compiled and installed in a
host staging dir removing the need to install them on the build system.

I'm not really satisfied with the new HOST_{C,LD}FLAGS variables.  They
are too similar to HOST{C,LD}FLAGS and I find them confusing.  However,
HOST_EXTRA*FLAGS are already reserved for local use in makefiles (see
Documentation/kbuild/makefiles.txt).  And I didn't want to have even
longer USER_HOST_*FLAGS.  If someone has a better proposition, I'll
happily make a v3.

Comments are welcome.

Changes since v1:
 - Fix typos.
 - Do not use HOST_EXTRA*FLAGS. Add new user specifiable variables.
 - Pass HOSTLDFLAGS to build single .c programs.
 - Build objtool with host flags.

Robin Jarry (3):
  kbuild: introduce HOST_{C,LD}FLAGS
  kbuild: use HOSTLDFLAGS for single .c executables
  objtool: use global host flags for compilation

 Documentation/admin-guide/README.rst | 16 ++++++++++++++++
 Makefile                             |  9 +++++----
 scripts/Makefile.host                |  2 +-
 tools/objtool/Makefile               |  4 ++--
 4 files changed, 24 insertions(+), 7 deletions(-)

-- 
2.11.0

[PATCH v2 1/3] kbuild: introduce HOST_{C,LD}FLAGS

[Robin Jarry]

Add user-specifiable compiler/linker flags for host programs. Append
these to already existing HOST{C,CXX,LD}FLAGS.

Update documentation accordingly.

Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Ingo Molnar <mingo@kernel.org>
---
 Documentation/admin-guide/README.rst | 16 ++++++++++++++++
 Makefile                             |  7 ++++---
 2 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/Documentation/admin-guide/README.rst b/Documentation/admin-guide/README.rst
index af5a437198d0..22ba3140e85f 100644
--- a/Documentation/admin-guide/README.rst
+++ b/Documentation/admin-guide/README.rst
@@ -271,6 +271,22 @@ Compiling the kernel
    To have the build system also tell the reason for the rebuild of each
    target, use ``V=2``.  The default is ``V=0``.
 
+ - Host libs in non standard locations:
+
+   Some host programs are linked with external libraries, such as openssl
+   or libelf.  If these libs are installed in non standard locations on
+   your build system, you may specify it via the ``HOST_CFLAGS`` and
+   ``HOST_LDFLAGS`` parameters on the ``make`` command line.
+
+   For example, if you have installed host libs in ``/foo/staging``, you
+   may use the following command::
+
+     make HOST_CFLAGS="-I/foo/staging/include" \
+          HOST_LDFLAGS="-L/foo/staging/lib -Wl,-rpath=/foo/staging/lib"
+
+   Please note: If the ``HOST_CFLAGS`` or ``HOST_LDFLAGS`` options are
+   used, then they must be used for all invocations of make.
+
  - Keep a backup kernel handy in case something goes wrong.  This is
    especially true for the development releases, since each new release
    contains new code which has not been debugged.  Make sure you keep a
diff --git a/Makefile b/Makefile
index 659a7780aeb3..f7c543cebfc6 100644
--- a/Makefile
+++ b/Makefile
@@ -362,9 +362,10 @@ HOST_LFS_LIBS := $(shell getconf LFS_LIBS)
 HOSTCC       = gcc
 HOSTCXX      = g++
 HOSTCFLAGS   := -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 \
-		-fomit-frame-pointer -std=gnu89 $(HOST_LFS_CFLAGS)
-HOSTCXXFLAGS := -O2 $(HOST_LFS_CFLAGS)
-HOSTLDFLAGS  := $(HOST_LFS_LDFLAGS)
+		-fomit-frame-pointer -std=gnu89 $(HOST_LFS_CFLAGS) \
+		$(HOST_CFLAGS)
+HOSTCXXFLAGS := -O2 $(HOST_LFS_CFLAGS) $(HOST_CFLAGS)
+HOSTLDFLAGS  := $(HOST_LFS_LDFLAGS) $(HOST_LDFLAGS)
 HOST_LOADLIBES := $(HOST_LFS_LIBS)
 
 # Make variables (CC, etc...)
-- 
2.11.0

[PATCH v2 2/3] kbuild: use HOSTLDFLAGS for single .c executables

[Robin Jarry]

When compiling executables from a single .c file, the linker is also
invoked. Pass the HOSTLDFLAGS like for other linker commands.

Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Ingo Molnar <mingo@kernel.org>
---
 scripts/Makefile.host | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.host b/scripts/Makefile.host
index e6dc6ae2d7c4..aa971cc3f339 100644
--- a/scripts/Makefile.host
+++ b/scripts/Makefile.host
@@ -84,7 +84,7 @@ hostcxx_flags  = -Wp,-MD,$(depfile) $(__hostcxx_flags)
 # Create executable from a single .c file
 # host-csingle -> Executable
 quiet_cmd_host-csingle 	= HOSTCC  $@
-      cmd_host-csingle	= $(HOSTCC) $(hostc_flags) -o $@ $< \
+      cmd_host-csingle	= $(HOSTCC) $(hostc_flags) $(HOSTLDFLAGS) -o $@ $< \
 	  	$(HOST_LOADLIBES) $(HOSTLOADLIBES_$(@F))
 $(host-csingle): $(obj)/%: $(src)/%.c FORCE
 	$(call if_changed_dep,host-csingle)
-- 
2.11.0

Re: [PATCH v2 2/3] kbuild: use HOSTLDFLAGS for single .c executables

[Masahiro Yamada]

2018-02-27 3:41 GMT+09:00 Robin Jarry <robin.jarry@6wind.com>:
> When compiling executables from a single .c file, the linker is also
> invoked. Pass the HOSTLDFLAGS like for other linker commands.
>
> Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Cc: Ingo Molnar <mingo@kernel.org>
> ---
>  scripts/Makefile.host | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.host b/scripts/Makefile.host
> index e6dc6ae2d7c4..aa971cc3f339 100644
> --- a/scripts/Makefile.host
> +++ b/scripts/Makefile.host
> @@ -84,7 +84,7 @@ hostcxx_flags  = -Wp,-MD,$(depfile) $(__hostcxx_flags)
>  # Create executable from a single .c file
>  # host-csingle -> Executable
>  quiet_cmd_host-csingle         = HOSTCC  $@
> -      cmd_host-csingle = $(HOSTCC) $(hostc_flags) -o $@ $< \
> +      cmd_host-csingle = $(HOSTCC) $(hostc_flags) $(HOSTLDFLAGS) -o $@ $< \
>                 $(HOST_LOADLIBES) $(HOSTLOADLIBES_$(@F))
>  $(host-csingle): $(obj)/%: $(src)/%.c FORCE
>         $(call if_changed_dep,host-csingle)


This one is correct anyway.

I applied to linux-kbuild.



-- 
Best Regards
Masahiro Yamada

[PATCH v2 3/3] objtool: use global host flags for compilation

[Robin Jarry]

When libelf headers and libs are not found and CONFIG_UNWINDER_ORC is
selected, there is a make error introduced by
commit 3dd40cb320fe ("objtool: Upgrade libelf-devel warning to error for
CONFIG_ORC_UNWINDER").

Host headers and libs may be in a non-standard location and the check is
performed without host flags. Make sure to use host flags for the check
and for building objtool. Remove duplicate flags.

Fixes: 3b27a0c85d70 ("objtool: Detect and warn if libelf is missing and don't break the build")
Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Ingo Molnar <mingo@kernel.org>
---
 Makefile               | 2 +-
 tools/objtool/Makefile | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index f7c543cebfc6..c9dc35fe0e48 100644
--- a/Makefile
+++ b/Makefile
@@ -955,7 +955,7 @@ export mod_sign_cmd
 
 ifdef CONFIG_STACK_VALIDATION
   has_libelf := $(call try-run,\
-		echo "int main() {}" | $(HOSTCC) -xc -o /dev/null -lelf -,1,0)
+	echo "int main() {}" | $(HOSTCC) $(HOSTLDFLAGS) -xc -o /dev/null -lelf -,1,0)
   ifeq ($(has_libelf),1)
     objtool_target := tools/objtool FORCE
   else
diff --git a/tools/objtool/Makefile b/tools/objtool/Makefile
index e6acc281dd37..4d4b810313e2 100644
--- a/tools/objtool/Makefile
+++ b/tools/objtool/Makefile
@@ -31,8 +31,8 @@ INCLUDES := -I$(srctree)/tools/include \
 	    -I$(srctree)/tools/arch/$(HOSTARCH)/include/uapi \
 	    -I$(srctree)/tools/objtool/arch/$(ARCH)/include
 WARNINGS := $(EXTRA_WARNINGS) -Wno-switch-default -Wno-switch-enum -Wno-packed
-CFLAGS   += -Wall -Werror $(WARNINGS) -fomit-frame-pointer -O2 -g $(INCLUDES)
-LDFLAGS  += -lelf $(LIBSUBCMD)
+CFLAGS   += $(HOSTCFLAGS) -Werror $(WARNINGS) -g $(INCLUDES)
+LDFLAGS  += $(HOSTLDFLAGS) -lelf $(LIBSUBCMD)
 
 # Allow old libelf to be used:
 elfshdr := $(shell echo '\#include <libelf.h>' | $(CC) $(CFLAGS) -x c -E - | grep elf_getshdr)
-- 
2.11.0

Re: [PATCH v2 3/3] objtool: use global host flags for compilation

[Josh Poimboeuf]

On Mon, Feb 26, 2018 at 07:41:48PM +0100, Robin Jarry wrote:
> When libelf headers and libs are not found and CONFIG_UNWINDER_ORC is
> selected, there is a make error introduced by
> commit 3dd40cb320fe ("objtool: Upgrade libelf-devel warning to error for
> CONFIG_ORC_UNWINDER").
> 
> Host headers and libs may be in a non-standard location and the check is
> performed without host flags. Make sure to use host flags for the check
> and for building objtool. Remove duplicate flags.
> 
> Fixes: 3b27a0c85d70 ("objtool: Detect and warn if libelf is missing and don't break the build")
> Signed-off-by: Robin Jarry <robin.jarry@6wind.com>
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Cc: Ingo Molnar <mingo@kernel.org>
> ---
>  Makefile               | 2 +-
>  tools/objtool/Makefile | 4 ++--
>  2 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index f7c543cebfc6..c9dc35fe0e48 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -955,7 +955,7 @@ export mod_sign_cmd
>  
>  ifdef CONFIG_STACK_VALIDATION
>    has_libelf := $(call try-run,\
> -		echo "int main() {}" | $(HOSTCC) -xc -o /dev/null -lelf -,1,0)
> +	echo "int main() {}" | $(HOSTCC) $(HOSTLDFLAGS) -xc -o /dev/null -lelf -,1,0)

This should probably also have HOSTCFLAGS, since the objtool makefile
uses it as well.

-- 
Josh

Re: [PATCH v2 3/3] objtool: use global host flags for compilation

[Robin Jarry]

2018-02-27, Josh Poimboeuf:
> On Mon, Feb 26, 2018 at 07:41:48PM +0100, Robin Jarry wrote:
[snip]
> >  ifdef CONFIG_STACK_VALIDATION
> >    has_libelf := $(call try-run,\
> > -		echo "int main() {}" | $(HOSTCC) -xc -o /dev/null -lelf -,1,0)
> > +	echo "int main() {}" | $(HOSTCC) $(HOSTLDFLAGS) -xc -o /dev/null -lelf -,1,0)
> 
> This should probably also have HOSTCFLAGS, since the objtool makefile
> uses it as well.

I had intentionally omitted HOSTCFLAGS since the compiled code does not
contain any #include statements (only -lelf). But adding them cannot
hurt.

-- 
Robin

Re: [PATCH v2 3/3] objtool: use global host flags for compilation

[Josh Poimboeuf]

On Tue, Feb 27, 2018 at 09:55:23PM +0100, Robin Jarry wrote:
> 2018-02-27, Josh Poimboeuf:
> > On Mon, Feb 26, 2018 at 07:41:48PM +0100, Robin Jarry wrote:
> [snip]
> > >  ifdef CONFIG_STACK_VALIDATION
> > >    has_libelf := $(call try-run,\
> > > -		echo "int main() {}" | $(HOSTCC) -xc -o /dev/null -lelf -,1,0)
> > > +	echo "int main() {}" | $(HOSTCC) $(HOSTLDFLAGS) -xc -o /dev/null -lelf -,1,0)
> > 
> > This should probably also have HOSTCFLAGS, since the objtool makefile
> > uses it as well.
> 
> I had intentionally omitted HOSTCFLAGS since the compiled code does not
> contain any #include statements (only -lelf). But adding them cannot
> hurt.

Ah, right.  The point of this check is to see if it can find libelf, so
I guess using HOSTCFLAGS wouldn't have a point here.  So it's fine the
way it is.

-- 
Josh

Re: [PATCH v2 0/3] kbuild: fix host progs build with libs in non standard locations

[Josh Poimboeuf]

On Mon, Feb 26, 2018 at 07:41:45PM +0100, Robin Jarry wrote:
> This patchset allows to build host programs that depend on external libs
> installed in non standard locations (i.e. not in /usr/include, /usr/lib,
> etc.).  For now, the only way is to force HOSTCC to include both the
> path to the host compiler and the build flags.
> 
> I have encountered this issue when building linux into the buildroot
> framework. host-* versions of libs may be compiled and installed in a
> host staging dir removing the need to install them on the build system.
> 
> I'm not really satisfied with the new HOST_{C,LD}FLAGS variables.  They
> are too similar to HOST{C,LD}FLAGS and I find them confusing.  However,
> HOST_EXTRA*FLAGS are already reserved for local use in makefiles (see
> Documentation/kbuild/makefiles.txt).  And I didn't want to have even
> longer USER_HOST_*FLAGS.  If someone has a better proposition, I'll
> happily make a v3.

In Documentation/kbuild/kbuild.txt, we have the following environment
variables:

  KCFLAGS
  --------------------------------------------------
  Additional options to the C compiler (for built-in and modules).
  
  CFLAGS_KERNEL
  --------------------------------------------------
  Additional options for $(CC) when used to compile
  code that is compiled as built-in.
  
  CFLAGS_MODULE
  --------------------------------------------------
  Additional module specific options to use for $(CC).
  
  LDFLAGS_MODULE
  --------------------------------------------------
  Additional options used for $(LD) when linking modules.
  
  LDFLAGS_vmlinux
  --------------------------------------------------
  Additional options passed to final link of vmlinux.

So instead of

  HOST_CFLAGS
  HOST_LDFLAGS

maybe it would be more consistent to call them

  CFLAGS_HOST
  LDFLAGS_HOST

?

Also, the new environment variables should be documented in the above
file.

-- 
Josh

Re: [PATCH v2 0/3] kbuild: fix host progs build with libs in non standard locations

[Robin Jarry]

2018-02-27, Josh Poimboeuf:
> In Documentation/kbuild/kbuild.txt, we have the following environment
> variables:
> 
>   KCFLAGS
>   --------------------------------------------------
>   Additional options to the C compiler (for built-in and modules).
>   
>   CFLAGS_KERNEL
>   --------------------------------------------------
>   Additional options for $(CC) when used to compile
>   code that is compiled as built-in.
>   
>   CFLAGS_MODULE
>   --------------------------------------------------
>   Additional module specific options to use for $(CC).
>   
>   LDFLAGS_MODULE
>   --------------------------------------------------
>   Additional options used for $(LD) when linking modules.
>   
>   LDFLAGS_vmlinux
>   --------------------------------------------------
>   Additional options passed to final link of vmlinux.
> 
> So instead of
> 
>   HOST_CFLAGS
>   HOST_LDFLAGS
> 
> maybe it would be more consistent to call them
> 
>   CFLAGS_HOST
>   LDFLAGS_HOST
> 
> ?

I had missed this file. Indeed {C,LD}FLAGS_HOST are much less confusing!

> Also, the new environment variables should be documented in the above
> file.

Sure, I'll do that. However, I feel like I should also leave the
paragraph in Documentation/admin-guide/README.txt.

What do you think?

-- 
Robin

Re: [PATCH v2 0/3] kbuild: fix host progs build with libs in non standard locations

[Josh Poimboeuf]

On Tue, Feb 27, 2018 at 09:52:31PM +0100, Robin Jarry wrote:
> 2018-02-27, Josh Poimboeuf:
> > In Documentation/kbuild/kbuild.txt, we have the following environment
> > variables:
> > 
> >   KCFLAGS
> >   --------------------------------------------------
> >   Additional options to the C compiler (for built-in and modules).
> >   
> >   CFLAGS_KERNEL
> >   --------------------------------------------------
> >   Additional options for $(CC) when used to compile
> >   code that is compiled as built-in.
> >   
> >   CFLAGS_MODULE
> >   --------------------------------------------------
> >   Additional module specific options to use for $(CC).
> >   
> >   LDFLAGS_MODULE
> >   --------------------------------------------------
> >   Additional options used for $(LD) when linking modules.
> >   
> >   LDFLAGS_vmlinux
> >   --------------------------------------------------
> >   Additional options passed to final link of vmlinux.
> > 
> > So instead of
> > 
> >   HOST_CFLAGS
> >   HOST_LDFLAGS
> > 
> > maybe it would be more consistent to call them
> > 
> >   CFLAGS_HOST
> >   LDFLAGS_HOST
> > 
> > ?
> 
> I had missed this file. Indeed {C,LD}FLAGS_HOST are much less confusing!
> 
> > Also, the new environment variables should be documented in the above
> > file.
> 
> Sure, I'll do that. However, I feel like I should also leave the
> paragraph in Documentation/admin-guide/README.txt.
> 
> What do you think?

[ Adding Jon Corbet to CC in case he has an opinion. ]

I think that paragraph in the admin guide seems a bit out of place,
because that file only seems to cover the most common cases.

No other env variables are described in that file (and some of the
previously existing env variables are much more likely to be used than
these new ones anyway).

It might be a good idea to instead just add a reference to kbuild.txt,
suggesting the user refer to that file if they want to learn more about
what env variables can be used to customize the build.

-- 
Josh

Re: [PATCH v2 0/3] kbuild: fix host progs build with libs in non standard locations

[Randy Dunlap]

On 02/27/2018 01:38 PM, Josh Poimboeuf wrote:
> On Tue, Feb 27, 2018 at 09:52:31PM +0100, Robin Jarry wrote:
>> 2018-02-27, Josh Poimboeuf:
>>> In Documentation/kbuild/kbuild.txt, we have the following environment
>>> variables:
>>>
>>>   KCFLAGS
>>>   --------------------------------------------------
>>>   Additional options to the C compiler (for built-in and modules).
>>>   
>>>   CFLAGS_KERNEL
>>>   --------------------------------------------------
>>>   Additional options for $(CC) when used to compile
>>>   code that is compiled as built-in.
>>>   
>>>   CFLAGS_MODULE
>>>   --------------------------------------------------
>>>   Additional module specific options to use for $(CC).
>>>   
>>>   LDFLAGS_MODULE
>>>   --------------------------------------------------
>>>   Additional options used for $(LD) when linking modules.
>>>   
>>>   LDFLAGS_vmlinux
>>>   --------------------------------------------------
>>>   Additional options passed to final link of vmlinux.
>>>
>>> So instead of
>>>
>>>   HOST_CFLAGS
>>>   HOST_LDFLAGS
>>>
>>> maybe it would be more consistent to call them
>>>
>>>   CFLAGS_HOST
>>>   LDFLAGS_HOST
>>>
>>> ?
>>
>> I had missed this file. Indeed {C,LD}FLAGS_HOST are much less confusing!
>>
>>> Also, the new environment variables should be documented in the above
>>> file.
>>
>> Sure, I'll do that. However, I feel like I should also leave the
>> paragraph in Documentation/admin-guide/README.txt.
>>
>> What do you think?
> 
> [ Adding Jon Corbet to CC in case he has an opinion. ]
> 
> I think that paragraph in the admin guide seems a bit out of place,
> because that file only seems to cover the most common cases.
> 
> No other env variables are described in that file (and some of the
> previously existing env variables are much more likely to be used than
> these new ones anyway).
> 
> It might be a good idea to instead just add a reference to kbuild.txt,
> suggesting the user refer to that file if they want to learn more about
> what env variables can be used to customize the build.

(IMHO :)  I agree, the addition to admin-guide/README.rst (not .txt) is in
the wrong place, although it could contain a pointer to kbuild/ like Josh says.

And if you do move that text to kbuild/, please change "non standard" to
"non-standard" in 2 places.

thanks,
-- 
~Randy

Re: [PATCH v2 0/3] kbuild: fix host progs build with libs in non standard locations

[Masahiro Yamada]

2018-02-28 2:45 GMT+09:00 Josh Poimboeuf <jpoimboe@redhat.com>:
> On Mon, Feb 26, 2018 at 07:41:45PM +0100, Robin Jarry wrote:
>> This patchset allows to build host programs that depend on external libs
>> installed in non standard locations (i.e. not in /usr/include, /usr/lib,
>> etc.).  For now, the only way is to force HOSTCC to include both the
>> path to the host compiler and the build flags.
>>
>> I have encountered this issue when building linux into the buildroot
>> framework. host-* versions of libs may be compiled and installed in a
>> host staging dir removing the need to install them on the build system.
>>
>> I'm not really satisfied with the new HOST_{C,LD}FLAGS variables.  They
>> are too similar to HOST{C,LD}FLAGS and I find them confusing.  However,
>> HOST_EXTRA*FLAGS are already reserved for local use in makefiles (see
>> Documentation/kbuild/makefiles.txt).  And I didn't want to have even
>> longer USER_HOST_*FLAGS.  If someone has a better proposition, I'll
>> happily make a v3.
>
> In Documentation/kbuild/kbuild.txt, we have the following environment
> variables:
>
>   KCFLAGS
>   --------------------------------------------------
>   Additional options to the C compiler (for built-in and modules).
>
>   CFLAGS_KERNEL
>   --------------------------------------------------
>   Additional options for $(CC) when used to compile
>   code that is compiled as built-in.
>
>   CFLAGS_MODULE
>   --------------------------------------------------
>   Additional module specific options to use for $(CC).
>
>   LDFLAGS_MODULE
>   --------------------------------------------------
>   Additional options used for $(LD) when linking modules.
>
>   LDFLAGS_vmlinux
>   --------------------------------------------------
>   Additional options passed to final link of vmlinux.
>
> So instead of
>
>   HOST_CFLAGS
>   HOST_LDFLAGS
>
> maybe it would be more consistent to call them
>
>   CFLAGS_HOST
>   LDFLAGS_HOST
>
> ?
>
> Also, the new environment variables should be documented in the above
> file.
>
> --
> Josh



A generic rule I see is almost like this:


[1] "KBUILD_" + (Executable Name) + "FLAGS" = (Internal-use Variable)

[2] (Executable Name)  + "FLAGS" = (User Interface via Command Line)


They also derive

[3] (Internal-use Variable)  =
                      "KBUILD_" + (User Interface via Command Line)


The following is the current situation:


[1] Flags for $(CC)

           Internal use                  User interface via command line
---------------------------------------------------------------------------
(common)   KBUILD_CFLAGS                     KCFLAGS
(builtin)  KBUILD_CFLAGS_KERNEL              CFLAGS_KERNEL
(module)   KBUILD_CFLAGS_MODULE              CFLAGS_MODULE



[2] Flags for $(AS)

           Internal use                   User interface via command line
--------------------------------------------------------------------------
(common)   KBUILD_AFLAGS                      KAFLAGS
(builtin)  KBUILD_AFLAGS_KERNEL               AFLAGS_KERNEL
(module)   KBUILD_AFLAGS_MODULE               AFLAGS_MODULE


[3] Flags for $(CPP)

           Internal use                     User interface via command line
--------------------------------------------------------------------------
(common)   KBUILD_CPPFLAGS                    KCPPFLAGS



[4] Flags for $(LD)

           Internal use                     User interface via command line
--------------------------------------------------------------------------

(builtin)   [None]  #1                         [None]
(module)   KBUILD_LDFLAGS_MODULE              LDFLAGS_MODULE


  #1 (LDFLAGS_vmlinux is used for internal variable for builtin)


[5] Flags for HOSTCC, HOSTCXX

            Internal use                      User interface via command line
-----------------------------------------------------------------------------
C             HOSTCFLAGS                          [None]
C++           HOSTCXXFLAGS                        [None]
Link          HOSTLDFLAGS                         [None]
Library       HOST_LOADLIBES                      [None]



KCFLAGS, KAFLAGS, KCPPFLAGS are exceptions.
[5] is also an exception.




A consistent way could be

[5] Flags for HOSTCC, HOSTCXX

            Internal use                      User interface via command line
-----------------------------------------------------------------------------
C           KBUILD_HOSTCFLAGS                     HOSTCFLAGS
C++         KBUILD_HOSTCXXFLAGS                   HOSTCXXFLAGS
Link        KBUILD_HOSTLDFLAGS                    HOSTLDFLAGS
Lib         KBUILD_HOSTLDLIBS                     HOSTLDLIBS




Documentation/kbuild/kbuild.txt lists user interface parameters.
It is difficult to change them.  (probably, not allowed)



Documentation/kbuild/makefiles.txt lists Make variables
used in kernel sources.  So, it is easier to rename them.

Only the problem would be external modules that compile
their own host-programs with own flags. It is rare.
Anyway, external modules are often broken
when the kernel version is updated.


So, my idea is to rename existing

 HOSTCFLAGS     ->  KBUILD_HOSTCFLAGS
 HOSTCXXFLAGS   ->  KBUILD_HOSTCXXFLAGS
 HOSTLDFLAGS    ->  KBUILD_HOSTLDFLAGS
 HOST_LOADLIBES ->  KBUILD_HOSTLDLIBS


("LOADLIBES" is too long, so rename it to "LDLIBS")


Then, re-add no-prefix ones as user interface.



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 0/3] kbuild: fix host progs build with libs in non standard locations

[Robin Jarry]

2018-02-28, Masahiro Yamada:
[snip]
> So, my idea is to rename existing
> 
>  HOSTCFLAGS     ->  KBUILD_HOSTCFLAGS
>  HOSTCXXFLAGS   ->  KBUILD_HOSTCXXFLAGS
>  HOSTLDFLAGS    ->  KBUILD_HOSTLDFLAGS
>  HOST_LOADLIBES ->  KBUILD_HOSTLDLIBS
> 
> 
> ("LOADLIBES" is too long, so rename it to "LDLIBS")
> 
> 
> Then, re-add no-prefix ones as user interface.

Hi Masahiro,

Ok, I'll add the rename patch in a v3 of my series.

-- 
Robin