[PATCH v2 00/21] kconfig: move compiler capability tests to Kconfig

[PATCH v2 00/21] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

Here is v2 to start to move compiler capability tests to Kconfig.

V1:
https://lkml.org/lkml/2018/2/16/610

I brushed up the implementation even more.

Major changes for v2:
[1] Environments and functions are expanded in the lexer (zconf.l)
    The parser (zconf.y) receives expanded tokens.  This simplifies
    the implementation.

[2] Implement only one built-in function 'shell'.  This returns
    stdout from the command.  We need something to return 'y' or 'n'.
    This is now implemented as a macro 'success'.

[3] Macros (user-defined function) are defined by 'macro' keyword
    instead of string type symbol.

    ex1) macro success $(shell ($(1) && echo y) || echo n)
    ex2) macro cc-option $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)

My plan for v3
--------------

The MW is approaching, but I'd like to make it more lovely if
I have some time.

While I was writing these patches, I noticed a kind of similarity
between Makefile and Kconfig.

Makefile can define variables and functions to do text processing
during the parse phase.  After parsing, it builds up dependency
graph, and moves on to the execution phase.  Then, it generates
objects as needed by checking time-stamps.

[Makefile]

  OBJ := foo.o
  SRC := foo.c
  CC := gcc

  $(OBJ): $(SRC)
            $(CC) -c -o $(OBJ) $(SRC)

[Makefile after text processing]

  foo.o: foo.c
          gcc -c -o foo.o foo.c

Now, we are adding tools for text processing to Kconfig.  Functions
are statically expanded by the lexer.  After all source files are
parsed, Kconfig moves on to the evaluation phase.  Then, users can
tweak symbol values from menus.

So, we should be able to describe Kconfig like follows:

[Kconfig]

   MY_SYMBOL := CC_STACKPROTECTOR
   MY_PROMPT := "Stackprotector"
   success = $(shell ($(1) && echo y) || echo n)
   cc-option = $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)

   config $(MY_SYMBOL)
           bool $(MY_PROMPT)
           depends on $(cc-option -fstack-protector)

[Kconfig after text processing]

   config CC_STACKPROTECTOR
           bool "Stackptector"
           depends on y

So, I think it is better to use '=' assignment for defining functions.
I will remove the 'macro' keyword.
Also, support variables.  A variable is a function with no argument.
So, it will be easy to implement it as a sub-set of function feature.
It is better to support two flavors of assignments, '=' and ':=' as well.

Masahiro Yamada (21):
  kbuild: remove kbuild cache
  kbuild: remove CONFIG_CROSS_COMPILE support
  kconfig: move and rename sym_expand_string_value()
  kconfig: reference environments directly and remove 'option env='
    syntax
  kconfig: remove string expansion in file_lookup()
  kconfig: remove string expansion for mainmenu after yyparse()
  kconfig: add function support and implement 'shell' function
  kconfig: replace $UNAME_RELEASE with function call
  kconfig: add 'macro' keyword to support user-defined function
  kconfig: add 'success' and 'cc-option' macros
  stack-protector: test compiler capability in Kconfig and drop AUTO
    mode
  kconfig: show compiler version text in the top comment
  kconfig: add CC_IS_GCC and GCC_VERSION
  kconfig: add CC_IS_CLANG and CLANG_VERSION
  gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT
  kcov: imply GCC_PLUGINS and GCC_PLUGIN_SANCOV instead of select'ing
    them
  gcc-plugins: always build plugins with C++
  gcc-plugins: move GCC version check for PowerPC to Kconfig
  gcc-plugins: test GCC plugin support in Kconfig
  gcc-plugins: enable GCC_PLUGINS for COMPILE_TEST
  arm64: move GCC version check for ARCH_SUPPORTS_INT128 to Kconfig

 Documentation/kbuild/kconfig-language.txt |   8 --
 Kconfig                                   |   4 +-
 Makefile                                  | 103 ++------------
 arch/Kconfig                              |  37 ++---
 arch/arm64/Kconfig                        |   1 +
 arch/arm64/Makefile                       |   2 -
 arch/powerpc/Kconfig                      |   2 +-
 arch/sh/Kconfig                           |   4 +-
 arch/sparc/Kconfig                        |   4 +-
 arch/tile/Kconfig                         |   2 +-
 arch/um/Kconfig.common                    |   4 -
 arch/x86/Kconfig                          |  12 +-
 arch/x86/um/Kconfig                       |   4 +-
 init/Kconfig                              |  44 +++---
 kernel/gcov/Kconfig                       |  17 +--
 kernel/gcov/Makefile                      |   2 -
 lib/Kconfig.debug                         |   7 +-
 scripts/Kbuild.include                    | 101 ++------------
 scripts/Makefile.gcc-plugins              |  95 ++++---------
 scripts/clang-version.sh                  |  24 ++--
 scripts/gcc-plugin.sh                     |  37 +----
 scripts/gcc-plugins/Makefile              |  15 +-
 scripts/gcc-x86_32-has-stack-protector.sh |   7 +-
 scripts/gcc-x86_64-has-stack-protector.sh |   5 -
 scripts/kconfig/confdata.c                |  31 +---
 scripts/kconfig/env.c                     |  95 +++++++++++++
 scripts/kconfig/function.c                | 225 ++++++++++++++++++++++++++++++
 scripts/kconfig/kconf_id.c                |   1 -
 scripts/kconfig/lkc.h                     |   9 +-
 scripts/kconfig/lkc_proto.h               |   7 +-
 scripts/kconfig/menu.c                    |   3 -
 scripts/kconfig/symbol.c                  | 109 ---------------
 scripts/kconfig/util.c                    |  96 ++++++++++---
 scripts/kconfig/zconf.l                   |  51 ++++++-
 scripts/kconfig/zconf.y                   |  35 ++---
 35 files changed, 608 insertions(+), 595 deletions(-)
 create mode 100644 scripts/kconfig/env.c
 create mode 100644 scripts/kconfig/function.c

-- 
2.7.4

[PATCH v2 01/21] kbuild: remove kbuild cache

[Masahiro Yamada]

The kbuild cache was introduced to remember the result of shell
commands, some of which are expensive to compute, such as
$(call cc-option,...).

However, this turned out not so clever as I had first expected.
Actually, it is problematic.  For example, "$(CC) -print-file-name"
is cached.  If the compiler is updated, the stale search path causes
build error, which is difficult to figure out.  Another problem
scenario is cache files could be touched while install targets are
running under the root permission.  We can patch them if desired,
but the build infrastructure is getting uglier and uglier.

Now, we are going to move compiler flag tests to the configuration
phase.  If this is completed, the result of compiler tests will be
naturally cached in the .config file.  We will not have performance
issues of incremental building since this testing only happens at
Kconfig time.

To start this work with a cleaner code base, remove the kbuild
cache first.

Revert the following commits:
Commit 9a234a2e3843 ("kbuild: create directory for make cache only when necessary")
Commit e17c400ae194 ("kbuild: shrink .cache.mk when it exceeds 1000 lines")
Commit 4e56207130ed ("kbuild: Cache a few more calls to the compiler")
Commit 3298b690b21c ("kbuild: Add a cache for generated variables")

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2: None

 Makefile               |   5 +--
 scripts/Kbuild.include | 101 +++++++------------------------------------------
 2 files changed, 16 insertions(+), 90 deletions(-)

diff --git a/Makefile b/Makefile
index ff62b8e..c55cf95 100644
--- a/Makefile
+++ b/Makefile
@@ -656,7 +656,7 @@ KBUILD_CFLAGS += $(call cc-ifversion, -lt, 0409, \
 KBUILD_CFLAGS	+= $(call cc-option,--param=allow-store-data-races=0)
 
 # check for 'asm goto'
-ifeq ($(call shell-cached,$(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC) $(KBUILD_CFLAGS)), y)
+ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC) $(KBUILD_CFLAGS)), y)
 	KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
 	KBUILD_AFLAGS += -DCC_HAVE_ASM_GOTO
 endif
@@ -811,7 +811,7 @@ KBUILD_CFLAGS	+= $(call cc-option,-fdata-sections,)
 endif
 
 # arch Makefile may override CC so keep this after arch Makefile is included
-NOSTDINC_FLAGS += -nostdinc -isystem $(call shell-cached,$(CC) -print-file-name=include)
+NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)
 CHECKFLAGS     += $(NOSTDINC_FLAGS)
 
 # warn about C99 declaration after statement
@@ -1615,7 +1615,6 @@ clean: $(clean-dirs)
 		-o -name '.*.d' -o -name '.*.tmp' -o -name '*.mod.c' \
 		-o -name '*.symtypes' -o -name 'modules.order' \
 		-o -name modules.builtin -o -name '.tmp_*.o.*' \
-		-o -name .cache.mk \
 		-o -name '*.c.[012]*.*' \
 		-o -name '*.ll' \
 		-o -name '*.gcno' \) -type f -print | xargs rm -f
diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index cce31ee..9f7eb10 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -8,8 +8,6 @@ squote  := '
 empty   :=
 space   := $(empty) $(empty)
 space_escape := _-_SPACE_-_
-right_paren := )
-left_paren := (
 
 ###
 # Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
@@ -82,71 +80,6 @@ cc-cross-prefix =  \
 			echo $(c);                                    \
 		fi)))
 
-# Tools for caching Makefile variables that are "expensive" to compute.
-#
-# Here we want to help deal with variables that take a long time to compute
-# by making it easy to store these variables in a cache.
-#
-# The canonical example here is testing for compiler flags.  On a simple system
-# each call to the compiler takes 10 ms, but on a system with a compiler that's
-# called through various wrappers it can take upwards of 100 ms.  If we have
-# 100 calls to the compiler this can take 1 second (on a simple system) or 10
-# seconds (on a complicated system).
-#
-# The "cache" will be in Makefile syntax and can be directly included.
-# Any time we try to reference a variable that's not in the cache we'll
-# calculate it and store it in the cache for next time.
-
-# Include values from last time
-make-cache := $(if $(KBUILD_EXTMOD),$(KBUILD_EXTMOD)/,$(if $(obj),$(obj)/)).cache.mk
-$(make-cache): ;
--include $(make-cache)
-
-cached-data := $(filter __cached_%, $(.VARIABLES))
-
-# If cache exceeds 1000 lines, shrink it down to 500.
-ifneq ($(word 1000,$(cached-data)),)
-$(shell tail -n 500 $(make-cache) > $(make-cache).tmp; \
-	mv $(make-cache).tmp $(make-cache))
-endif
-
-create-cache-dir := $(if $(KBUILD_SRC),$(if $(cache-data),,1))
-
-# Usage: $(call __sanitize-opt,Hello=Hola$(comma)Goodbye Adios)
-#
-# Convert all '$', ')', '(', '\', '=', ' ', ',', ':' to '_'
-__sanitize-opt = $(subst $$,_,$(subst $(right_paren),_,$(subst $(left_paren),_,$(subst \,_,$(subst =,_,$(subst $(space),_,$(subst $(comma),_,$(subst :,_,$(1)))))))))
-
-# Usage:   $(call shell-cached,shell_command)
-# Example: $(call shell-cached,md5sum /usr/bin/gcc)
-#
-# If we've already seen a call to this exact shell command (even in a
-# previous invocation of make!) we'll return the value.  If not, we'll
-# compute it and store the result for future runs.
-#
-# This is a bit of voodoo, but basic explanation is that if the variable
-# was undefined then we'll evaluate the shell command and store the result
-# into the variable.  We'll then store that value in the cache and finally
-# output the value.
-#
-# NOTE: The $$(2) here isn't actually a parameter to __run-and-store.  We
-# happen to know that the caller will have their shell command in $(2) so the
-# result of "call"ing this will produce a reference to that $(2).  The reason
-# for this strangeness is to avoid an extra level of eval (and escaping) of
-# $(2).
-define __run-and-store
-ifeq ($(origin $(1)),undefined)
-  $$(eval $(1) := $$(shell $$(2)))
-ifeq ($(create-cache-dir),1)
-  $$(shell mkdir -p $(dir $(make-cache)))
-  $$(eval create-cache-dir :=)
-endif
-  $$(shell echo '$(1) := $$($(1))' >> $(make-cache))
-endif
-endef
-__shell-cached = $(eval $(call __run-and-store,$(1)))$($(1))
-shell-cached = $(call __shell-cached,__cached_$(call __sanitize-opt,$(1)),$(1))
-
 # output directory for tests below
 TMPOUT := $(if $(KBUILD_EXTMOD),$(firstword $(KBUILD_EXTMOD))/)
 
@@ -154,36 +87,30 @@ TMPOUT := $(if $(KBUILD_EXTMOD),$(firstword $(KBUILD_EXTMOD))/)
 # Usage: option = $(call try-run, $(CC)...-o "$$TMP",option-ok,otherwise)
 # Exit code chooses option. "$$TMP" serves as a temporary file and is
 # automatically cleaned up.
-__try-run = set -e;			\
+try-run = $(shell set -e;		\
 	TMP="$(TMPOUT).$$$$.tmp";	\
 	TMPO="$(TMPOUT).$$$$.o";	\
 	if ($(1)) >/dev/null 2>&1;	\
 	then echo "$(2)";		\
 	else echo "$(3)";		\
 	fi;				\
-	rm -f "$$TMP" "$$TMPO"
-
-try-run = $(shell $(__try-run))
-
-# try-run-cached
-# This works like try-run, but the result is cached.
-try-run-cached = $(call shell-cached,$(__try-run))
+	rm -f "$$TMP" "$$TMPO")
 
 # as-option
 # Usage: cflags-y += $(call as-option,-Wa$(comma)-isa=foo,)
 
-as-option = $(call try-run-cached,\
+as-option = $(call try-run,\
 	$(CC) $(KBUILD_CFLAGS) $(1) -c -x assembler /dev/null -o "$$TMP",$(1),$(2))
 
 # as-instr
 # Usage: cflags-y += $(call as-instr,instr,option1,option2)
 
-as-instr = $(call try-run-cached,\
+as-instr = $(call try-run,\
 	printf "%b\n" "$(1)" | $(CC) $(KBUILD_AFLAGS) -c -x assembler -o "$$TMP" -,$(2),$(3))
 
 # __cc-option
 # Usage: MY_CFLAGS += $(call __cc-option,$(CC),$(MY_CFLAGS),-march=winchip-c6,-march=i586)
-__cc-option = $(call try-run-cached,\
+__cc-option = $(call try-run,\
 	$(1) -Werror $(2) $(3) -c -x c /dev/null -o "$$TMP",$(3),$(4))
 
 # Do not attempt to build with gcc plugins during cc-option tests.
@@ -203,23 +130,23 @@ hostcc-option = $(call __cc-option, $(HOSTCC),\
 
 # cc-option-yn
 # Usage: flag := $(call cc-option-yn,-march=winchip-c6)
-cc-option-yn = $(call try-run-cached,\
+cc-option-yn = $(call try-run,\
 	$(CC) -Werror $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) $(1) -c -x c /dev/null -o "$$TMP",y,n)
 
 # cc-disable-warning
 # Usage: cflags-y += $(call cc-disable-warning,unused-but-set-variable)
-cc-disable-warning = $(call try-run-cached,\
+cc-disable-warning = $(call try-run,\
 	$(CC) -Werror $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) -W$(strip $(1)) -c -x c /dev/null -o "$$TMP",-Wno-$(strip $(1)))
 
 # cc-name
 # Expands to either gcc or clang
-cc-name = $(call shell-cached,$(CC) -v 2>&1 | grep -q "clang version" && echo clang || echo gcc)
+cc-name = $(shell $(CC) -v 2>&1 | grep -q "clang version" && echo clang || echo gcc)
 
 # cc-version
-cc-version = $(call shell-cached,$(CONFIG_SHELL) $(srctree)/scripts/gcc-version.sh $(CC))
+cc-version = $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-version.sh $(CC))
 
 # cc-fullversion
-cc-fullversion = $(call shell-cached,$(CONFIG_SHELL) \
+cc-fullversion = $(shell $(CONFIG_SHELL) \
 	$(srctree)/scripts/gcc-version.sh -p $(CC))
 
 # cc-ifversion
@@ -232,21 +159,21 @@ cc-if-fullversion = $(shell [ $(cc-fullversion) $(1) $(2) ] && echo $(3) || echo
 
 # cc-ldoption
 # Usage: ldflags += $(call cc-ldoption, -Wl$(comma)--hash-style=both)
-cc-ldoption = $(call try-run-cached,\
+cc-ldoption = $(call try-run,\
 	$(CC) $(1) $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) -nostdlib -x c /dev/null -o "$$TMP",$(1),$(2))
 
 # ld-option
 # Usage: LDFLAGS += $(call ld-option, -X)
-ld-option = $(call try-run-cached, $(LD) $(LDFLAGS) $(1) -v,$(1),$(2))
+ld-option = $(call try-run, $(LD) $(LDFLAGS) $(1) -v,$(1),$(2))
 
 # ar-option
 # Usage: KBUILD_ARFLAGS := $(call ar-option,D)
 # Important: no spaces around options
-ar-option = $(call try-run-cached, $(AR) rc$(1) "$$TMP",$(1),$(2))
+ar-option = $(call try-run, $(AR) rc$(1) "$$TMP",$(1),$(2))
 
 # ld-version
 # Note this is mainly for HJ Lu's 3 number binutil versions
-ld-version = $(call shell-cached,$(LD) --version | $(srctree)/scripts/ld-version.sh)
+ld-version = $(shell $(LD) --version | $(srctree)/scripts/ld-version.sh)
 
 # ld-ifversion
 # Usage:  $(call ld-ifversion, -ge, 22252, y)
-- 
2.7.4

[PATCH v2 02/21] kbuild: remove CONFIG_CROSS_COMPILE support

[Masahiro Yamada]

Kbuild provides a couple of ways to specify CROSS_COMPILE:

[1] Command line
[2] Environment
[3] arch/*/Makefile (only some architectures)
[4] CONFIG_CROSS_COMPILE

[4] is problematic for the compiler capability tests in Kconfig.
CONFIG_CROSS_COMPILE allows users to change the compiler prefix from
'make menuconfig', etc.  It means, the compiler options would have
to be all re-calculated everytime CONFIG_CROSS_COMPILE is changed.

To avoid complexity and performance issues, I'd like to evaluate
the shell commands statically, i.e. only parsing Kconfig files.

I guess the majority is [1] or [2].  Currently, there are only
4 defconfig files that specify CONFIG_CROSS_COMPILE.
  arch/arm/configs/lpc18xx_defconfig
  arch/hexagon/configs/comet_defconfig
  arch/openrisc/configs/or1ksim_defconfig
  arch/openrisc/configs/simple_smp_defconfig

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2: None

 Makefile     | 3 ---
 init/Kconfig | 9 ---------
 2 files changed, 12 deletions(-)

diff --git a/Makefile b/Makefile
index c55cf95..5c395ed 100644
--- a/Makefile
+++ b/Makefile
@@ -316,12 +316,9 @@ SUBARCH := $(shell uname -m | sed -e s/i.86/x86/ -e s/x86_64/x86/ \
 # CROSS_COMPILE can be set on the command line
 # make CROSS_COMPILE=ia64-linux-
 # Alternatively CROSS_COMPILE can be set in the environment.
-# A third alternative is to store a setting in .config so that plain
-# "make" in the configured kernel build directory always uses that.
 # Default value for CROSS_COMPILE is not to prefix executables
 # Note: Some architectures assign CROSS_COMPILE in their arch/*/Makefile
 ARCH		?= $(SUBARCH)
-CROSS_COMPILE	?= $(CONFIG_CROSS_COMPILE:"%"=%)
 
 # Architecture as present in compile.h
 UTS_MACHINE 	:= $(ARCH)
diff --git a/init/Kconfig b/init/Kconfig
index e37f4b2..df18492 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -54,15 +54,6 @@ config INIT_ENV_ARG_LIMIT
 	  Maximum of each of the number of arguments and environment
 	  variables passed to init from the kernel command line.
 
-
-config CROSS_COMPILE
-	string "Cross-compiler tool prefix"
-	help
-	  Same as running 'make CROSS_COMPILE=prefix-' but stored for
-	  default make runs in this kernel build directory.  You don't
-	  need to set this unless you want the configured kernel build
-	  directory to select the cross-compiler automatically.
-
 config COMPILE_TEST
 	bool "Compile also drivers which will not load"
 	depends on !UML
-- 
2.7.4

[PATCH v2 03/21] kconfig: move and rename sym_expand_string_value()

[Masahiro Yamada]

This helper expands symbols contained in a string.  I am about to
change it to expand environments instead of symbols.  Also, I will
add function expansion later.

Rename it to expand_string_value(), and move it to util.c, which is
a more suitable place.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2: None

 scripts/kconfig/lkc.h       |  1 +
 scripts/kconfig/lkc_proto.h |  1 -
 scripts/kconfig/symbol.c    | 53 -------------------------------------------
 scripts/kconfig/util.c      | 55 ++++++++++++++++++++++++++++++++++++++++++++-
 scripts/kconfig/zconf.y     |  2 +-
 5 files changed, 56 insertions(+), 56 deletions(-)

diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
index f4394af..c8d9e55 100644
--- a/scripts/kconfig/lkc.h
+++ b/scripts/kconfig/lkc.h
@@ -111,6 +111,7 @@ void menu_finalize(struct menu *parent);
 void menu_set_type(int type);
 
 /* util.c */
+char *expand_string_value(const char *in);
 struct file *file_lookup(const char *name);
 int file_write_dep(const char *name);
 void *xmalloc(size_t size);
diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
index 9dc8abf..9884adc 100644
--- a/scripts/kconfig/lkc_proto.h
+++ b/scripts/kconfig/lkc_proto.h
@@ -31,7 +31,6 @@ extern struct symbol * symbol_hash[SYMBOL_HASHSIZE];
 
 struct symbol * sym_lookup(const char *name, int flags);
 struct symbol * sym_find(const char *name);
-char *sym_expand_string_value(const char *in);
 const char * sym_escape_string_value(const char *in);
 struct symbol ** sym_re_search(const char *pattern);
 const char * sym_type_name(enum symbol_type type);
diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
index f0b2e3b..03143b2 100644
--- a/scripts/kconfig/symbol.c
+++ b/scripts/kconfig/symbol.c
@@ -906,59 +906,6 @@ struct symbol *sym_find(const char *name)
 	return symbol;
 }
 
-/*
- * Expand symbol's names embedded in the string given in argument. Symbols'
- * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
- * the empty string.
- */
-char *sym_expand_string_value(const char *in)
-{
-	const char *src;
-	char *res;
-	size_t reslen;
-
-	/*
-	 * Note: 'in' might come from a token that's about to be
-	 * freed, so make sure to always allocate a new string
-	 */
-	reslen = strlen(in) + 1;
-	res = xmalloc(reslen);
-	res[0] = '\0';
-
-	while ((src = strchr(in, '$'))) {
-		char *p, name[SYMBOL_MAXLENGTH];
-		const char *symval = "";
-		struct symbol *sym;
-		size_t newlen;
-
-		strncat(res, in, src - in);
-		src++;
-
-		p = name;
-		while (isalnum(*src) || *src == '_')
-			*p++ = *src++;
-		*p = '\0';
-
-		sym = sym_find(name);
-		if (sym != NULL) {
-			sym_calc_value(sym);
-			symval = sym_get_string_value(sym);
-		}
-
-		newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
-		if (newlen > reslen) {
-			reslen = newlen;
-			res = xrealloc(res, reslen);
-		}
-
-		strcat(res, symval);
-		in = src;
-	}
-	strcat(res, in);
-
-	return res;
-}
-
 const char *sym_escape_string_value(const char *in)
 {
 	const char *p;
diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index c6f6e21..22201a4 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -10,11 +10,64 @@
 #include <string.h>
 #include "lkc.h"
 
+/*
+ * Expand symbol's names embedded in the string given in argument. Symbols'
+ * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
+ * the empty string.
+ */
+char *expand_string_value(const char *in)
+{
+	const char *src;
+	char *res;
+	size_t reslen;
+
+	/*
+	 * Note: 'in' might come from a token that's about to be
+	 * freed, so make sure to always allocate a new string
+	 */
+	reslen = strlen(in) + 1;
+	res = xmalloc(reslen);
+	res[0] = '\0';
+
+	while ((src = strchr(in, '$'))) {
+		char *p, name[SYMBOL_MAXLENGTH];
+		const char *symval = "";
+		struct symbol *sym;
+		size_t newlen;
+
+		strncat(res, in, src - in);
+		src++;
+
+		p = name;
+		while (isalnum(*src) || *src == '_')
+			*p++ = *src++;
+		*p = '\0';
+
+		sym = sym_find(name);
+		if (sym != NULL) {
+			sym_calc_value(sym);
+			symval = sym_get_string_value(sym);
+		}
+
+		newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
+		if (newlen > reslen) {
+			reslen = newlen;
+			res = xrealloc(res, reslen);
+		}
+
+		strcat(res, symval);
+		in = src;
+	}
+	strcat(res, in);
+
+	return res;
+}
+
 /* file already present in list? If not add it */
 struct file *file_lookup(const char *name)
 {
 	struct file *file;
-	char *file_name = sym_expand_string_value(name);
+	char *file_name = expand_string_value(name);
 
 	for (file = file_list; file; file = file->next) {
 		if (!strcmp(name, file->name)) {
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index ad6305b..262c464 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -547,7 +547,7 @@ void conf_parse(const char *name)
 
 	tmp = rootmenu.prompt->text;
 	rootmenu.prompt->text = _(rootmenu.prompt->text);
-	rootmenu.prompt->text = sym_expand_string_value(rootmenu.prompt->text);
+	rootmenu.prompt->text = expand_string_value(rootmenu.prompt->text);
 	free((char*)tmp);
 
 	menu_finalize(&rootmenu);
-- 
2.7.4

[PATCH v2 04/21] kconfig: reference environments directly and remove 'option env=' syntax

[Masahiro Yamada]

To get an environment value, Kconfig needs to define a symbol using
"option env=" syntax.  It is tedious to add a config entry for each
environment given that we need more environments such as 'CC', 'AS',
'srctree' etc. to evaluate the compiler capability in Kconfig.

Adding '$' to symbols is weird.  Kconfig can reference symbols directly
like this:

  config FOO
          string
          default BAR

So, I want to use the following syntax to get environment 'BAR' from
the system:

  config FOO
          string
          default $BAR

Looking at the code, the symbols prefixed with 'S' are expanded by:
 - conf_expand_value()
   This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
 - expand_string_value()
   This is used to expand strings in 'source' and 'mainmenu'

All of them are fixed values independent of user configuration.  So,
this kind of syntax should be moved to simply take the environment.

This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.

sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
should be be given from the environment.

ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
by 'default ARCH_DEFCONFIG'.

The environments are expanding in the lexer; when '$' is encountered,
it is expanded, and resulted strings are pushed back to the input
stream.  This makes the implementation simpler.

For example, the following code works.

[Example code]

  config TOOLCHAIN_LIST
          string
          default "My tools: CC=$CC, AS=$AS, CPP=$CPP"

[Result]

  $ make -s alldefconfig && tail -n 1 .config
  CONFIG_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

I tested all 'make *config' for arch architectures.
I confirmed this commit still produced the same result
(by my kconfig test tool).


Changes in v2:
  - Move the string expansion to the lexer phase.
  - Split environment helpers to env.c

 Documentation/kbuild/kconfig-language.txt |  8 ---
 Kconfig                                   |  4 --
 Makefile                                  |  3 +-
 arch/sh/Kconfig                           |  4 +-
 arch/sparc/Kconfig                        |  4 +-
 arch/tile/Kconfig                         |  2 +-
 arch/um/Kconfig.common                    |  4 --
 arch/x86/Kconfig                          |  4 +-
 arch/x86/um/Kconfig                       |  4 +-
 init/Kconfig                              | 10 +---
 scripts/kconfig/confdata.c                | 31 +---------
 scripts/kconfig/env.c                     | 95 +++++++++++++++++++++++++++++++
 scripts/kconfig/kconf_id.c                |  1 -
 scripts/kconfig/lkc.h                     |  8 +--
 scripts/kconfig/menu.c                    |  3 -
 scripts/kconfig/symbol.c                  | 56 ------------------
 scripts/kconfig/util.c                    | 75 ++++++++----------------
 scripts/kconfig/zconf.l                   | 20 ++++++-
 scripts/kconfig/zconf.y                   |  2 +-
 19 files changed, 158 insertions(+), 180 deletions(-)
 create mode 100644 scripts/kconfig/env.c

diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
index f5b9493..0e966e8 100644
--- a/Documentation/kbuild/kconfig-language.txt
+++ b/Documentation/kbuild/kconfig-language.txt
@@ -198,14 +198,6 @@ applicable everywhere (see syntax).
     enables the third modular state for all config symbols.
     At most one symbol may have the "modules" option set.
 
-  - "env"=<value>
-    This imports the environment variable into Kconfig. It behaves like
-    a default, except that the value comes from the environment, this
-    also means that the behaviour when mixing it with normal defaults is
-    undefined at this point. The symbol is currently not exported back
-    to the build environment (if this is desired, it can be done via
-    another symbol).
-
   - "allnoconfig_y"
     This declares the symbol as one that should have the value y when
     using "allnoconfig". Used for symbols that hide other symbols.
diff --git a/Kconfig b/Kconfig
index 8c4c1cb..e6ece5b 100644
--- a/Kconfig
+++ b/Kconfig
@@ -5,8 +5,4 @@
 #
 mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
 
-config SRCARCH
-	string
-	option env="SRCARCH"
-
 source "arch/$SRCARCH/Kconfig"
diff --git a/Makefile b/Makefile
index 5c395ed..4ae1486 100644
--- a/Makefile
+++ b/Makefile
@@ -284,7 +284,8 @@ include scripts/Kbuild.include
 # Read KERNELRELEASE from include/config/kernel.release (if it exists)
 KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
 KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
-export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
+UNAME_RELEASE := $(shell uname --release)
+export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
 
 # SUBARCH tells the usermode build what the underlying arch is.  That is set
 # first, and if a usermode build is happening, the "ARCH=um" on the command
diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
index 97fe293..14f3ef1 100644
--- a/arch/sh/Kconfig
+++ b/arch/sh/Kconfig
@@ -57,7 +57,7 @@ config SUPERH
 	  <http://www.linux-sh.org/>.
 
 config SUPERH32
-	def_bool ARCH = "sh"
+	def_bool "$ARCH" = "sh"
 	select HAVE_KPROBES
 	select HAVE_KRETPROBES
 	select HAVE_IOREMAP_PROT if MMU && !X2TLB
@@ -76,7 +76,7 @@ config SUPERH32
 	select HAVE_CC_STACKPROTECTOR
 
 config SUPERH64
-	def_bool ARCH = "sh64"
+	def_bool "$ARCH" = "sh64"
 	select HAVE_EXIT_THREAD
 	select KALLSYMS
 
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 8767e45..86b852e 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -1,6 +1,6 @@
 config 64BIT
-	bool "64-bit kernel" if ARCH = "sparc"
-	default ARCH = "sparc64"
+	bool "64-bit kernel" if "$ARCH" = "sparc"
+	default "$ARCH" = "sparc64"
 	help
 	  SPARC is a family of RISC microprocessors designed and marketed by
 	  Sun Microsystems, incorporated.  They are very widely found in Sun
diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
index ef9d403..acc2182 100644
--- a/arch/tile/Kconfig
+++ b/arch/tile/Kconfig
@@ -119,7 +119,7 @@ config HVC_TILE
 # Building with ARCH=tilegx (or ARCH=tile) implies using the
 # 64-bit TILE-Gx toolchain, so force CONFIG_TILEGX on.
 config TILEGX
-	def_bool ARCH != "tilepro"
+	def_bool "$ARCH" != "tilepro"
 	select ARCH_SUPPORTS_ATOMIC_RMW
 	select GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
 	select HAVE_ARCH_JUMP_LABEL
diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
index c68add8..07f84c8 100644
--- a/arch/um/Kconfig.common
+++ b/arch/um/Kconfig.common
@@ -54,10 +54,6 @@ config HZ
 	int
 	default 100
 
-config SUBARCH
-	string
-	option env="SUBARCH"
-
 config NR_CPUS
 	int
 	range 1 1
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 0fa71a7..986fb0a 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1,8 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
 # Select 32 or 64 bit
 config 64BIT
-	bool "64-bit kernel" if ARCH = "x86"
-	default ARCH != "i386"
+	bool "64-bit kernel" if "$ARCH" = "x86"
+	default "$ARCH" != "i386"
 	---help---
 	  Say yes to build a 64-bit kernel - formerly known as x86_64
 	  Say no to build a 32-bit kernel - formerly known as i386
diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
index 13ed827..d355413 100644
--- a/arch/x86/um/Kconfig
+++ b/arch/x86/um/Kconfig
@@ -16,8 +16,8 @@ config UML_X86
 	select GENERIC_FIND_FIRST_BIT
 
 config 64BIT
-	bool "64-bit kernel" if SUBARCH = "x86"
-	default SUBARCH != "i386"
+	bool "64-bit kernel" if $SUBARCH = "x86"
+	default $SUBARCH != "i386"
 
 config X86_32
 	def_bool !64BIT
diff --git a/init/Kconfig b/init/Kconfig
index df18492..b4814e6 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1,11 +1,3 @@
-config ARCH
-	string
-	option env="ARCH"
-
-config KERNELVERSION
-	string
-	option env="KERNELVERSION"
-
 config DEFCONFIG_LIST
 	string
 	depends on !UML
@@ -13,7 +5,7 @@ config DEFCONFIG_LIST
 	default "/lib/modules/$UNAME_RELEASE/.config"
 	default "/etc/kernel-config"
 	default "/boot/config-$UNAME_RELEASE"
-	default "$ARCH_DEFCONFIG"
+	default ARCH_DEFCONFIG
 	default "arch/$ARCH/defconfig"
 
 config CONSTRUCTORS
diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
index df26c7b..98c2014 100644
--- a/scripts/kconfig/confdata.c
+++ b/scripts/kconfig/confdata.c
@@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
 	return name ? name : "include/config/auto.conf";
 }
 
-static char *conf_expand_value(const char *in)
-{
-	struct symbol *sym;
-	const char *src;
-	static char res_value[SYMBOL_MAXLENGTH];
-	char *dst, name[SYMBOL_MAXLENGTH];
-
-	res_value[0] = 0;
-	dst = name;
-	while ((src = strchr(in, '$'))) {
-		strncat(res_value, in, src - in);
-		src++;
-		dst = name;
-		while (isalnum(*src) || *src == '_')
-			*dst++ = *src++;
-		*dst = 0;
-		sym = sym_lookup(name, 0);
-		sym_calc_value(sym);
-		strcat(res_value, sym_get_string_value(sym));
-		in = src;
-	}
-	strcat(res_value, in);
-
-	return res_value;
-}
-
 char *conf_get_default_confname(void)
 {
 	struct stat buf;
 	static char fullname[PATH_MAX+1];
 	char *env, *name;
 
-	name = conf_expand_value(conf_defname);
+	name = expand_string_value(conf_defname);
 	env = getenv(SRCTREE);
 	if (env) {
 		sprintf(fullname, "%s/%s", env, name);
@@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
 			if (expr_calc_value(prop->visible.expr) == no ||
 			    prop->expr->type != E_SYMBOL)
 				continue;
-			name = conf_expand_value(prop->expr->left.sym->name);
+			sym_calc_value(prop->expr->left.sym);
+			name = sym_get_string_value(prop->expr->left.sym);
 			in = zconf_fopen(name);
 			if (in) {
 				conf_message(_("using defaults found in %s"),
diff --git a/scripts/kconfig/env.c b/scripts/kconfig/env.c
new file mode 100644
index 0000000..9702f5c
--- /dev/null
+++ b/scripts/kconfig/env.c
@@ -0,0 +1,95 @@
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
+
+static LIST_HEAD(env_list);
+
+struct env {
+	char *name;
+	char *value;
+	struct list_head node;
+};
+
+static struct env *env_list_lookup(const char *name)
+{
+	struct env *e;
+
+	list_for_each_entry(e, &env_list, node) {
+		if (!strcmp(name, e->name))
+			return e;
+	}
+
+	return NULL;
+}
+
+static void env_list_add(const char *name, const char *value)
+{
+	struct env *e;
+
+	e = xmalloc(sizeof(*e));
+	e->name = xstrdup(name);
+	e->value = xstrdup(value);
+
+	list_add_tail(&e->node, &env_list);
+}
+
+static void env_list_del(struct env *e)
+{
+	list_del(&e->node);
+	free(e->name);
+	free(e->value);
+	free(e);
+}
+
+/* the returned pointer must be freed when done */
+static char *env_expand(const char *name)
+{
+	struct env *e;
+	const char *value;
+
+	e = env_list_lookup(name);
+	if (e)
+		return xstrdup(e->value);
+
+	value = getenv(name);
+	if (!value) {
+		fprintf(stderr, "environment variable \"%s\" undefined\n", name);
+		value = "";
+	}
+
+	/*
+	 * we need to remember all referenced environments.
+	 * They will be written out to include/config/auto.conf.cmd
+	 */
+	env_list_add(name, value);
+
+	return xstrdup(value);
+}
+
+/* works like env_expand, but 'name' does not need to be null-terminated */
+char *env_expand_n(const char *name, size_t n)
+{
+	char *tmp, *res;
+
+	tmp = xmalloc(n + 1);
+	memcpy(tmp, name, n);
+	*(tmp + n) = '\0';
+
+	res = env_expand(tmp);
+
+	free(tmp);
+
+	return res;
+}
+
+void env_write_dep(FILE *f, const char *autoconfig_name)
+{
+	struct env *env, *tmp;
+
+	list_for_each_entry_safe(env, tmp, &env_list, node) {
+		fprintf(f, "ifneq \"$(%s)\" \"%s\"\n", env->name, env->value);
+		fprintf(f, "%s: FORCE\n", autoconfig_name);
+		fprintf(f, "endif\n");
+		env_list_del(env);
+	}
+}
diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
index 3ea9c5f..b3e0ea0 100644
--- a/scripts/kconfig/kconf_id.c
+++ b/scripts/kconfig/kconf_id.c
@@ -32,7 +32,6 @@ static struct kconf_id kconf_id_array[] = {
 	{ "on",			T_ON,			TF_PARAM },
 	{ "modules",		T_OPT_MODULES,		TF_OPTION },
 	{ "defconfig_list",	T_OPT_DEFCONFIG_LIST,	TF_OPTION },
-	{ "env",		T_OPT_ENV,		TF_OPTION },
 	{ "allnoconfig_y",	T_OPT_ALLNOCONFIG_Y,	TF_OPTION },
 };
 
diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
index c8d9e55..03d007f 100644
--- a/scripts/kconfig/lkc.h
+++ b/scripts/kconfig/lkc.h
@@ -58,7 +58,6 @@ enum conf_def_mode {
 
 #define T_OPT_MODULES		1
 #define T_OPT_DEFCONFIG_LIST	2
-#define T_OPT_ENV		3
 #define T_OPT_ALLNOCONFIG_Y	4
 
 struct kconf_id {
@@ -95,6 +94,10 @@ static inline void xfwrite(const void *str, size_t len, size_t count, FILE *out)
 		fprintf(stderr, "Error in writing or end of file.\n");
 }
 
+/* env.c */
+char *env_expand_n(const char *name, size_t n);
+void env_write_dep(FILE *f, const char *auto_conf_name);
+
 /* menu.c */
 void _menu_init(void);
 void menu_warn(struct menu *menu, const char *fmt, ...);
@@ -135,9 +138,6 @@ void str_printf(struct gstr *gs, const char *fmt, ...);
 const char *str_get(struct gstr *gs);
 
 /* symbol.c */
-extern struct expr *sym_env_list;
-
-void sym_init(void);
 void sym_clear_all_valid(void);
 struct symbol *sym_choice_default(struct symbol *sym);
 const char *sym_get_string_default(struct symbol *sym);
diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
index 5c5c137..8148305 100644
--- a/scripts/kconfig/menu.c
+++ b/scripts/kconfig/menu.c
@@ -214,9 +214,6 @@ void menu_add_option(int token, char *arg)
 			zconf_error("trying to redefine defconfig symbol");
 		sym_defconfig_list->flags |= SYMBOL_AUTO;
 		break;
-	case T_OPT_ENV:
-		prop_add_env(arg);
-		break;
 	case T_OPT_ALLNOCONFIG_Y:
 		current_entry->sym->flags |= SYMBOL_ALLNOCONFIG_Y;
 		break;
diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
index 03143b2..7c9a88e 100644
--- a/scripts/kconfig/symbol.c
+++ b/scripts/kconfig/symbol.c
@@ -33,33 +33,6 @@ struct symbol *sym_defconfig_list;
 struct symbol *modules_sym;
 tristate modules_val;
 
-struct expr *sym_env_list;
-
-static void sym_add_default(struct symbol *sym, const char *def)
-{
-	struct property *prop = prop_alloc(P_DEFAULT, sym);
-
-	prop->expr = expr_alloc_symbol(sym_lookup(def, SYMBOL_CONST));
-}
-
-void sym_init(void)
-{
-	struct symbol *sym;
-	struct utsname uts;
-	static bool inited = false;
-
-	if (inited)
-		return;
-	inited = true;
-
-	uname(&uts);
-
-	sym = sym_lookup("UNAME_RELEASE", 0);
-	sym->type = S_STRING;
-	sym->flags |= SYMBOL_AUTO;
-	sym_add_default(sym, uts.release);
-}
-
 enum symbol_type sym_get_type(struct symbol *sym)
 {
 	enum symbol_type type = sym->type;
@@ -1348,32 +1321,3 @@ const char *prop_get_type_name(enum prop_type type)
 	}
 	return "unknown";
 }
-
-static void prop_add_env(const char *env)
-{
-	struct symbol *sym, *sym2;
-	struct property *prop;
-	char *p;
-
-	sym = current_entry->sym;
-	sym->flags |= SYMBOL_AUTO;
-	for_all_properties(sym, prop, P_ENV) {
-		sym2 = prop_get_symbol(prop);
-		if (strcmp(sym2->name, env))
-			menu_warn(current_entry, "redefining environment symbol from %s",
-				  sym2->name);
-		return;
-	}
-
-	prop = prop_alloc(P_ENV, sym);
-	prop->expr = expr_alloc_symbol(sym_lookup(env, SYMBOL_CONST));
-
-	sym_env_list = expr_alloc_one(E_LIST, sym_env_list);
-	sym_env_list->right.sym = sym;
-
-	p = getenv(env);
-	if (p)
-		sym_add_default(sym, p);
-	else
-		menu_warn(current_entry, "environment variable %s undefined", env);
-}
diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index 22201a4..136e497 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -8,16 +8,18 @@
 #include <stdarg.h>
 #include <stdlib.h>
 #include <string.h>
+
+#include "list.h"
 #include "lkc.h"
 
 /*
- * Expand symbol's names embedded in the string given in argument. Symbols'
- * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
+ * Expand environments embedded in the string given in argument. Environments
+ * to be expanded shall be prefixed by a '$'. Unknown environment expands to
  * the empty string.
  */
 char *expand_string_value(const char *in)
 {
-	const char *src;
+	const char *p, *q;
 	char *res;
 	size_t reslen;
 
@@ -25,39 +27,28 @@ char *expand_string_value(const char *in)
 	 * Note: 'in' might come from a token that's about to be
 	 * freed, so make sure to always allocate a new string
 	 */
-	reslen = strlen(in) + 1;
-	res = xmalloc(reslen);
-	res[0] = '\0';
-
-	while ((src = strchr(in, '$'))) {
-		char *p, name[SYMBOL_MAXLENGTH];
-		const char *symval = "";
-		struct symbol *sym;
-		size_t newlen;
-
-		strncat(res, in, src - in);
-		src++;
-
-		p = name;
-		while (isalnum(*src) || *src == '_')
-			*p++ = *src++;
-		*p = '\0';
-
-		sym = sym_find(name);
-		if (sym != NULL) {
-			sym_calc_value(sym);
-			symval = sym_get_string_value(sym);
-		}
+	res = xmalloc(1);
+	*res = '\0';
 
-		newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
-		if (newlen > reslen) {
-			reslen = newlen;
-			res = xrealloc(res, reslen);
-		}
+	while ((p = strchr(in, '$'))) {
+		char *new;
+
+		q = p + 1;
+		while (isalnum(*q) || *q == '_')
+			q++;
 
-		strcat(res, symval);
-		in = src;
+		new = env_expand_n(p + 1, q - p - 1);
+
+		reslen = strlen(res) + (p - in) + strlen(new) + 1;
+		res = xrealloc(res, reslen);
+		strncat(res, in, p - in);
+		strcat(res, new);
+		free(new);
+		in = q;
 	}
+
+	reslen = strlen(res) + strlen(in) + 1;
+	res = xrealloc(res, reslen);
 	strcat(res, in);
 
 	return res;
@@ -87,8 +78,6 @@ struct file *file_lookup(const char *name)
 /* write a dependency file as used by kbuild to track dependencies */
 int file_write_dep(const char *name)
 {
-	struct symbol *sym, *env_sym;
-	struct expr *e;
 	struct file *file;
 	FILE *out;
 
@@ -107,21 +96,7 @@ int file_write_dep(const char *name)
 	fprintf(out, "\n%s: \\\n"
 		     "\t$(deps_config)\n\n", conf_get_autoconfig_name());
 
-	expr_list_for_each_sym(sym_env_list, e, sym) {
-		struct property *prop;
-		const char *value;
-
-		prop = sym_get_env_prop(sym);
-		env_sym = prop_get_symbol(prop);
-		if (!env_sym)
-			continue;
-		value = getenv(env_sym->name);
-		if (!value)
-			value = "";
-		fprintf(out, "ifneq \"$(%s)\" \"%s\"\n", env_sym->name, value);
-		fprintf(out, "%s: FORCE\n", conf_get_autoconfig_name());
-		fprintf(out, "endif\n");
-	}
+	env_write_dep(out, conf_get_autoconfig_name());
 
 	fprintf(out, "\n$(deps_config): ;\n");
 	fclose(out);
diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
index 045093d..551ca47 100644
--- a/scripts/kconfig/zconf.l
+++ b/scripts/kconfig/zconf.l
@@ -35,6 +35,7 @@ struct buffer *current_buf;
 
 static int last_ts, first_ts;
 
+static void expand_string(const char *in);
 static void zconf_endhelp(void);
 static void zconf_endfile(void);
 
@@ -120,6 +121,7 @@ n	[A-Za-z0-9_-]
 }
 
 <PARAM>{
+	"$".*	expand_string(yytext);
 	"&&"	return T_AND;
 	"||"	return T_OR;
 	"("	return T_OPEN_PAREN;
@@ -157,12 +159,13 @@ n	[A-Za-z0-9_-]
 }
 
 <STRING>{
-	[^'"\\\n]+/\n	{
+	"$".*	expand_string(yytext);
+	[^$'"\\\n]+/\n	{
 		append_string(yytext, yyleng);
 		yylval.string = text;
 		return T_WORD_QUOTE;
 	}
-	[^'"\\\n]+	{
+	[^$'"\\\n]+	{
 		append_string(yytext, yyleng);
 	}
 	\\.?/\n	{
@@ -249,6 +252,19 @@ n	[A-Za-z0-9_-]
 }
 
 %%
+static void expand_string(const char *in)
+{
+	char *p, *q;
+
+	p = expand_string_value(in);
+
+	q = p + strlen(p);
+	while (q > p)
+		unput(*--q);
+
+	free(p);
+}
+
 void zconf_starthelp(void)
 {
 	new_string();
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index 262c464..4ff4ac9 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -534,7 +534,6 @@ void conf_parse(const char *name)
 
 	zconf_initscan(name);
 
-	sym_init();
 	_menu_init();
 
 	if (getenv("ZCONF_DEBUG"))
@@ -775,6 +774,7 @@ void zconfdump(FILE *out)
 }
 
 #include "zconf.lex.c"
+#include "env.c"
 #include "util.c"
 #include "confdata.c"
 #include "expr.c"
-- 
2.7.4

[PATCH v2 05/21] kconfig: remove string expansion in file_lookup()

[Masahiro Yamada]

There are two callers of file_lookup().

[1] zconf_initscan()
    This is used to open the first Kconfig.  However, it cannot
    contain environments in the file path because zconf_fopen() is
    called before file_lookup().  By swapping the call order,
    KBUILD_KCONFIG would be able to contain environments, but I do
    not see practical benefits to support it.

[2] zconf_nextfile()
    This is used to open the next file from 'source' statement like
        source "arch/$SRCARCH/Kconfig"
    but this has already been expanded in the lexer phase.

So, file_lookup() does not need to expand the given path.

By the way, file_lookup() was already buggy; it expanded a given path,
but it used the path before expansion for look-up:
        if (!strcmp(name, file->name)) {

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Simplify the patch.  Just remove text expansion.

 scripts/kconfig/util.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index 136e497..3d27c49 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -58,18 +58,16 @@ char *expand_string_value(const char *in)
 struct file *file_lookup(const char *name)
 {
 	struct file *file;
-	char *file_name = expand_string_value(name);
 
 	for (file = file_list; file; file = file->next) {
 		if (!strcmp(name, file->name)) {
-			free(file_name);
 			return file;
 		}
 	}
 
 	file = xmalloc(sizeof(*file));
 	memset(file, 0, sizeof(*file));
-	file->name = file_name;
+	file->name = xstrdup(name);
 	file->next = file_list;
 	file_list = file;
 	return file;
-- 
2.7.4

[PATCH v2 06/21] kconfig: remove string expansion for mainmenu after yyparse()

[Masahiro Yamada]

Now that environments are expanded in the lexer, conf_parse() does
not need to explicitly expand $ARCH and $KERNELVERSION in the mainmenu.

The hack introduced by commit 0724a7c32a54 ("kconfig: Don't leak
main menus during parsing") can go away.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Simplify the patch.  Just remove the text expansion.

 scripts/kconfig/zconf.y | 24 +++++-------------------
 1 file changed, 5 insertions(+), 19 deletions(-)

diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index 4ff4ac9..d8120c7 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -109,7 +109,7 @@ static struct menu *current_menu, *current_entry;
 %%
 input: nl start | start;
 
-start: mainmenu_stmt stmt_list | no_mainmenu_stmt stmt_list;
+start: mainmenu_stmt stmt_list | stmt_list;
 
 /* mainmenu entry */
 
@@ -118,19 +118,6 @@ mainmenu_stmt: T_MAINMENU prompt nl
 	menu_add_prompt(P_MENU, $2, NULL);
 };
 
-/* Default main menu, if there's no mainmenu entry */
-
-no_mainmenu_stmt: /* empty */
-{
-	/*
-	 * Hack: Keep the main menu title on the heap so we can safely free it
-	 * later regardless of whether it comes from the 'prompt' in
-	 * mainmenu_stmt or here
-	 */
-	menu_add_prompt(P_MENU, xstrdup("Linux Kernel Configuration"), NULL);
-};
-
-
 stmt_list:
 	  /* empty */
 	| stmt_list common_stmt
@@ -528,7 +515,6 @@ word_opt: /* empty */			{ $$ = NULL; }
 
 void conf_parse(const char *name)
 {
-	const char *tmp;
 	struct symbol *sym;
 	int i;
 
@@ -544,10 +530,10 @@ void conf_parse(const char *name)
 	if (!modules_sym)
 		modules_sym = sym_find( "n" );
 
-	tmp = rootmenu.prompt->text;
-	rootmenu.prompt->text = _(rootmenu.prompt->text);
-	rootmenu.prompt->text = expand_string_value(rootmenu.prompt->text);
-	free((char*)tmp);
+	if (!menu_has_prompt(&rootmenu)) {
+		current_entry = &rootmenu;
+		menu_add_prompt(P_MENU, "Linux Kernel Configuration", NULL);
+	}
 
 	menu_finalize(&rootmenu);
 	for_all_symbols(i, sym) {
-- 
2.7.4

[PATCH v2 07/21] kconfig: add function support and implement 'shell' function

[Masahiro Yamada]

This commit adds a new concept 'function' to do more text processing
in Kconfig.

A function call looks like this:

  $(function arg1, arg2, arg3, ...)

(Actually, this syntax was inspired by make.)

Real examples will look like this:

  $(shell echo hello world)
  $(cc-option -fstackprotector)

This commit adds the basic infrastructure to add, delete, evaluate
functions, and also the first built-in function $(shell ...).  This
accepts a single command to execute.  It returns the standard output
from it.

[Example code]

  config HELLO
          string
          default "$(shell echo hello world)"

  config Y
          def_bool $(shell echo y)

[Result]

  $ make -s alldefconfig && tail -n 2 .config
  CONFIG_HELLO="hello world"
  CONFIG_Y=y

Caveat:
Like environments, functions are expanded in the lexer.  You cannot
pass symbols to function arguments.  This is a limitation to simplify
the implementation.  I want to avoid the dynamic function evaluation,
which would introduce much more complexity.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Reminder for myself:
Update Documentation/kbuild/kconfig-language.txt


Changes in v2:
  - Use 'shell' for getting stdout from the comment.
    It was 'shell-stdout' in the previous version.
  - Symplify the implementation since the expansion has been moved to
    lexer.

 scripts/kconfig/function.c  | 170 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/kconfig/lkc_proto.h |   5 ++
 scripts/kconfig/util.c      |  46 +++++++++---
 scripts/kconfig/zconf.y     |   9 +++
 4 files changed, 222 insertions(+), 8 deletions(-)
 create mode 100644 scripts/kconfig/function.c

diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
new file mode 100644
index 0000000..913685f
--- /dev/null
+++ b/scripts/kconfig/function.c
@@ -0,0 +1,170 @@
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "list.h"
+
+#define FUNCTION_MAX_ARGS		10
+
+static LIST_HEAD(function_list);
+
+struct function {
+	char *name;
+	char *(*func)(struct function *f, int argc, char *argv[]);
+	struct list_head node;
+};
+
+static struct function *func_lookup(const char *name)
+{
+	struct function *f;
+
+	list_for_each_entry(f, &function_list, node) {
+		if (!strcmp(name, f->name))
+			return f;
+	}
+
+	return NULL;
+}
+
+static void func_add(const char *name,
+		     char *(*func)(struct function *f, int argc, char *argv[]))
+{
+	struct function *f;
+
+	f = func_lookup(name);
+	if (f) {
+		fprintf(stderr, "%s: function already exists. ignored.\n", name);
+		return;
+	}
+
+	f = xmalloc(sizeof(*f));
+	f->name = xstrdup(name);
+	f->func = func;
+
+	list_add_tail(&f->node, &function_list);
+}
+
+static void func_del(struct function *f)
+{
+	list_del(&f->node);
+	free(f->name);
+	free(f);
+}
+
+static char *func_call(int argc, char *argv[])
+{
+	struct function *f;
+
+	f = func_lookup(argv[0]);
+	if (!f) {
+		fprintf(stderr, "%s: function not found\n", argv[0]);
+		return NULL;
+	}
+
+	return f->func(f, argc, argv);
+}
+
+static char *func_eval(const char *func)
+{
+	char *expanded, *saveptr, *str, *token, *res;
+	const char *delim;
+	int argc = 0;
+	char *argv[FUNCTION_MAX_ARGS];
+
+	expanded = expand_string_value(func);
+
+	str = expanded;
+	delim = " ";
+
+	while ((token = strtok_r(str, delim, &saveptr))) {
+		argv[argc++] = token;
+		str = NULL;
+		delim = ",";
+	}
+
+	res = func_call(argc, argv);
+
+	free(expanded);
+
+	return res ?: xstrdup("");
+}
+
+char *func_eval_n(const char *func, size_t n)
+{
+	char *tmp, *res;
+
+	tmp = xmalloc(n + 1);
+	memcpy(tmp, func, n);
+	*(tmp + n) = '\0';
+
+	res = func_eval(tmp);
+
+	free(tmp);
+
+	return res;
+}
+
+/* built-in functions */
+static char *do_shell(struct function *f, int argc, char *argv[])
+{
+	static const char *pre = "(";
+	static const char *post = ") 2>/dev/null";
+	FILE *p;
+	char buf[256];
+	char *cmd;
+	int ret;
+
+	if (argc != 2)
+		return NULL;
+
+	/*
+	 * Surround the command with ( ) in case it is piped commands.
+	 * Also, redirect stderr to /dev/null.
+	 */
+	cmd = xmalloc(strlen(pre) + strlen(argv[1]) + strlen(post) + 1);
+	strcpy(cmd, pre);
+	strcat(cmd, argv[1]);
+	strcat(cmd, post);
+
+	p = popen(cmd, "r");
+	if (!p) {
+		perror(cmd);
+		goto free;
+	}
+	if (fgets(buf, sizeof(buf), p)) {
+		size_t len = strlen(buf);
+
+		if (buf[len - 1] == '\n')
+			buf[len - 1] = '\0';
+	} else {
+		buf[0] = '\0';
+	}
+
+	ret = pclose(p);
+	if (ret == -1)
+		perror(cmd);
+
+free:
+	free(cmd);
+
+	return xstrdup(buf);
+}
+
+void func_init(void)
+{
+	/* register built-in functions */
+	func_add("shell", do_shell);
+}
+
+void func_exit(void)
+{
+	struct function *f, *tmp;
+
+	/* unregister all functions */
+	list_for_each_entry_safe(f, tmp, &function_list, node)
+		func_del(f);
+}
diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
index 9884adc..09a4f53 100644
--- a/scripts/kconfig/lkc_proto.h
+++ b/scripts/kconfig/lkc_proto.h
@@ -48,5 +48,10 @@ const char * sym_get_string_value(struct symbol *sym);
 
 const char * prop_get_type_name(enum prop_type type);
 
+/* function.c */
+char *func_eval_n(const char *func, size_t n);
+void func_init(void);
+void func_exit(void);
+
 /* expr.c */
 void expr_print(struct expr *e, void (*fn)(void *, struct symbol *, const char *), void *data, int prevtoken);
diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index 3d27c49..218b051 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -13,9 +13,10 @@
 #include "lkc.h"
 
 /*
- * Expand environments embedded in the string given in argument. Environments
- * to be expanded shall be prefixed by a '$'. Unknown environment expands to
- * the empty string.
+ * Expand environments and functions embedded in the string given in argument.
+ * Environments to be expanded shall be prefixed by a '$'. Functions to be
+ * evaluated shall be surrounded by $(). Unknown environment/function expands
+ * to the empty string.
  */
 char *expand_string_value(const char *in)
 {
@@ -33,11 +34,40 @@ char *expand_string_value(const char *in)
 	while ((p = strchr(in, '$'))) {
 		char *new;
 
-		q = p + 1;
-		while (isalnum(*q) || *q == '_')
-			q++;
-
-		new = env_expand_n(p + 1, q - p - 1);
+		/*
+		 * If the next character is '(', it is a function.
+		 * Otherwise, environment.
+		 */
+		if (*(p + 1) == '(') {
+			int nest = 0;
+
+			q = p + 2;
+			while (1) {
+				if (*q == '\0') {
+					fprintf(stderr,
+						"unterminated function: %s\n",
+						p);
+					new = xstrdup("");
+					break;
+				} else if (*q == '(') {
+					nest++;
+				} else if (*q == ')') {
+					if (nest-- == 0) {
+						new = func_eval_n(p + 2,
+								  q - p - 2);
+						q++;
+						break;
+					}
+				}
+				q++;
+			}
+		} else {
+			q = p + 1;
+			while (isalnum(*q) || *q == '_')
+				q++;
+
+			new = env_expand_n(p + 1, q - p - 1);
+		}
 
 		reslen = strlen(res) + (p - in) + strlen(new) + 1;
 		res = xrealloc(res, reslen);
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index d8120c7..feaea18 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -520,11 +520,19 @@ void conf_parse(const char *name)
 
 	zconf_initscan(name);
 
+	func_init();
 	_menu_init();
 
 	if (getenv("ZCONF_DEBUG"))
 		yydebug = 1;
 	yyparse();
+
+	/*
+	 * Currently, functions are evaluated only when Kconfig files are
+	 * parsed. We can free functions here.
+	 */
+	func_exit();
+
 	if (yynerrs)
 		exit(1);
 	if (!modules_sym)
@@ -765,4 +773,5 @@ void zconfdump(FILE *out)
 #include "confdata.c"
 #include "expr.c"
 #include "symbol.c"
+#include "function.c"
 #include "menu.c"
-- 
2.7.4

[PATCH v2 08/21] kconfig: replace $UNAME_RELEASE with function call

[Masahiro Yamada]

Now that 'shell' function is supported, this can be self-contained in
Kconfig.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2: None

 Makefile     | 3 +--
 init/Kconfig | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index 4ae1486..5c395ed 100644
--- a/Makefile
+++ b/Makefile
@@ -284,8 +284,7 @@ include scripts/Kbuild.include
 # Read KERNELRELEASE from include/config/kernel.release (if it exists)
 KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
 KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
-UNAME_RELEASE := $(shell uname --release)
-export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
+export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
 
 # SUBARCH tells the usermode build what the underlying arch is.  That is set
 # first, and if a usermode build is happening, the "ARCH=um" on the command
diff --git a/init/Kconfig b/init/Kconfig
index b4814e6..0d6fe20 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2,9 +2,9 @@ config DEFCONFIG_LIST
 	string
 	depends on !UML
 	option defconfig_list
-	default "/lib/modules/$UNAME_RELEASE/.config"
+	default "/lib/modules/$(shell uname --release)/.config"
 	default "/etc/kernel-config"
-	default "/boot/config-$UNAME_RELEASE"
+	default "/boot/config-$(shell uname --release)"
 	default ARCH_DEFCONFIG
 	default "arch/$ARCH/defconfig"
 
-- 
2.7.4

[PATCH v2 09/21] kconfig: add 'macro' keyword to support user-defined function

[Masahiro Yamada]

Now, we got a basic ability to test compiler capability in Kconfig.

config CC_HAS_STACKPROTECTOR
        def_bool $(shell (($CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null) && echo y) || echo n)

This works, but it is ugly to repeat this long boilerplate.

We want to describe like this:

config CC_HAS_STACKPROTECTOR
        bool
        default $(cc-option -fstack-protector)

It is straight-forward to add a new function, but I do not like to
hard-code specialized functions like this.  Hence, here is another
feature to add functions from Kconfig files.

A user-defined function is defined with a special keyword 'macro'.
It can be referenced in the same way as built-in functions.  This
feature was also inspired by Makefile where user-defined functions
are referenced by $(call func-name, args...), but I omitted the 'call'
to makes it shorter.

The macro definition can contain $(1), $(2), ... which will be replaced
with arguments from the caller.  The macro works just as a textual
shorthand, which is also expanded in the lexer phase.

[Example Code]

  macro success $(shell ($(1) && echo y) || echo n)

  config TRUE
          bool "true"
          default $(success true)

  config FALSE
          bool "false"
          default $(success false)

[Result]

  $ make -s alldefconfig
  $ tail -n 2 .config
  CONFIG_TRUE=y
  # CONFIG_FALSE is not set

[Example Code]

  macro success $(shell ($(1) && echo y) || echo n)

  macro cc-option $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)

  config CC_HAS_STACKPROTECTOR
          def_bool $(cc-option -fstack-protector)

[Result]
  $ make -s alldefconfig
  $ tail -n 1 .config
  CONFIG_CC_HAS_STACKPROTECTOR=y

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Reminder for myself:
Update Documentation/kbuild/kconfig-language.txt

Changes in v2:
  - Use 'macro' directly instead of inside the string type symbol.

 scripts/kconfig/function.c  | 59 +++++++++++++++++++++++++++++++++++++++++++--
 scripts/kconfig/lkc_proto.h |  1 +
 scripts/kconfig/zconf.l     | 31 ++++++++++++++++++++++++
 3 files changed, 89 insertions(+), 2 deletions(-)

diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
index 913685f..389bb44 100644
--- a/scripts/kconfig/function.c
+++ b/scripts/kconfig/function.c
@@ -15,6 +15,7 @@ static LIST_HEAD(function_list);
 struct function {
 	char *name;
 	char *(*func)(struct function *f, int argc, char *argv[]);
+	char *macro;
 	struct list_head node;
 };
 
@@ -31,7 +32,8 @@ static struct function *func_lookup(const char *name)
 }
 
 static void func_add(const char *name,
-		     char *(*func)(struct function *f, int argc, char *argv[]))
+		     char *(*func)(struct function *f, int argc, char *argv[]),
+		     const char *macro)
 {
 	struct function *f;
 
@@ -44,6 +46,7 @@ static void func_add(const char *name,
 	f = xmalloc(sizeof(*f));
 	f->name = xstrdup(name);
 	f->func = func;
+	f->macro = macro ? xstrdup(macro) : NULL;
 
 	list_add_tail(&f->node, &function_list);
 }
@@ -51,6 +54,7 @@ static void func_add(const char *name,
 static void func_del(struct function *f)
 {
 	list_del(&f->node);
+	free(f->macro);
 	free(f->name);
 	free(f);
 }
@@ -108,6 +112,57 @@ char *func_eval_n(const char *func, size_t n)
 	return res;
 }
 
+/* run user-defined function */
+static char *do_macro(struct function *f, int argc, char *argv[])
+{
+	char *new;
+	char *src, *p, *res;
+	size_t newlen;
+	int n;
+
+	new = xmalloc(1);
+	*new = 0;
+
+	/*
+	 * This is a format string. $(1), $(2), ... must be replaced with
+	 * function arguments.
+	 */
+	src = f->macro;
+	p = src;
+
+	while ((p = strstr(p, "$("))) {
+		if (isdigit(p[2]) && p[3] == ')') {
+			n = p[2] - '0';
+			if (n < argc) {
+				newlen = strlen(new) + (p - src) +
+							strlen(argv[n]) + 1;
+				new = xrealloc(new, newlen);
+				strncat(new, src, p - src);
+				strcat(new, argv[n]);
+				src = p + 4;
+			}
+			p += 2;
+		}
+		p += 2;
+	}
+
+	newlen = strlen(new) + strlen(src) + 1;
+	new = xrealloc(new, newlen);
+	strcat(new, src);
+
+	res = expand_string_value(new);
+
+	free(new);
+
+	return res;
+}
+
+/* add user-defined function (macro) */
+void func_add_macro(const char *name, const char *macro)
+{
+	func_add(name, do_macro, macro);
+}
+
 /* built-in functions */
 static char *do_shell(struct function *f, int argc, char *argv[])
 {
@@ -157,7 +212,7 @@ static char *do_shell(struct function *f, int argc, char *argv[])
 void func_init(void)
 {
 	/* register built-in functions */
-	func_add("shell", do_shell);
+	func_add("shell", do_shell, NULL);
 }
 
 void func_exit(void)
diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
index 09a4f53..48699c0 100644
--- a/scripts/kconfig/lkc_proto.h
+++ b/scripts/kconfig/lkc_proto.h
@@ -50,6 +50,7 @@ const char * prop_get_type_name(enum prop_type type);
 
 /* function.c */
 char *func_eval_n(const char *func, size_t n);
+void func_add_macro(const char *name, const char *macro);
 void func_init(void);
 void func_exit(void);
 
diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
index 551ca47..6a18c68 100644
--- a/scripts/kconfig/zconf.l
+++ b/scripts/kconfig/zconf.l
@@ -74,6 +74,36 @@ static void warn_ignored_character(char chr)
 	        "%s:%d:warning: ignoring unsupported character '%c'\n",
 	        zconf_curname(), zconf_lineno(), chr);
 }
+
+static void handle_macro(const char *text)
+{
+	char *p, *q;
+
+	while (isspace(*text))
+		text++;
+
+	p = xstrdup(text);
+
+	q = p;
+	while (isalnum(*q) || *q == '_' || *q == '-')
+		q++;
+
+	if (q == p || !*q) {
+		yyerror("invalid\n");
+		goto free;
+	}
+
+	*q = '\0';
+
+	q++;
+	while (isspace(*q))
+		q++;
+
+	func_add_macro(p, q);
+free:
+	free(p);
+}
+
 %}
 
 n	[A-Za-z0-9_-]
@@ -82,6 +112,7 @@ n	[A-Za-z0-9_-]
 	int str = 0;
 	int ts, i;
 
+"macro"[ \t].*	handle_macro(yytext + 6);
 [ \t]*#.*\n	|
 [ \t]*\n	{
 	return T_EOL;
-- 
2.7.4

[PATCH v2 10/21] kconfig: add 'success' and 'cc-option' macros

[Masahiro Yamada]

'cc-option' will be the most frequently used macro.  It evaluates to 'y'
if the given argument is supported by the compiler, or 'n' otherwise.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Implement 'success' as a macro

 init/Kconfig | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/init/Kconfig b/init/Kconfig
index 0d6fe20..c456030 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -8,6 +8,12 @@ config DEFCONFIG_LIST
 	default ARCH_DEFCONFIG
 	default "arch/$ARCH/defconfig"
 
+# expanded to y if the command exits with 0, n otherwise
+macro success $(shell ($(1) && echo y) || echo n)
+
+# expanded to y if the given compiler flag is supported, n otherwise
+macro cc-option $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
+
 config CONSTRUCTORS
 	bool
 	depends on !UML
-- 
2.7.4

[PATCH v2 11/21] stack-protector: test compiler capability in Kconfig and drop AUTO mode

[Masahiro Yamada]

Move the test for -fstack-protector(-strong) option to Kconfig.

If the compiler does not support the option, the corresponding menu
is automatically hidden.  If _STRONG is not supported, it will fall
back to _REGULAR.  If _REGULAR is not supported, it will be disabled.
This means, _AUTO is implicitly handled by the dependency solver of
Kconfig, hence removed.

I also turned the 'choice' into only two boolean symbols.  The use of
'choice' is not a good idea here, because all of all{yes,mod,no}config
would choose the first visible value, while we want allnoconfig to
disable as many features as possible.

X86 has additional shell scripts in case the compiler supports the
option, but generates broken code.  I added CC_HAS_SANE_STACKPROTECTOR
to test this.  I had to add -m32 to gcc-x86_32-has-stack-protector.sh
to make it work correctly.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Describe $(cc-option ...) directly in depends on context

 Makefile                                  | 93 ++-----------------------------
 arch/Kconfig                              | 29 +++-------
 arch/x86/Kconfig                          |  8 ++-
 scripts/gcc-x86_32-has-stack-protector.sh |  7 +--
 scripts/gcc-x86_64-has-stack-protector.sh |  5 --
 5 files changed, 22 insertions(+), 120 deletions(-)

diff --git a/Makefile b/Makefile
index 5c395ed..5cadffa 100644
--- a/Makefile
+++ b/Makefile
@@ -675,55 +675,11 @@ ifneq ($(CONFIG_FRAME_WARN),0)
 KBUILD_CFLAGS += $(call cc-option,-Wframe-larger-than=${CONFIG_FRAME_WARN})
 endif
 
-# This selects the stack protector compiler flag. Testing it is delayed
-# until after .config has been reprocessed, in the prepare-compiler-check
-# target.
-ifdef CONFIG_CC_STACKPROTECTOR_AUTO
-  stackp-flag := $(call cc-option,-fstack-protector-strong,$(call cc-option,-fstack-protector))
-  stackp-name := AUTO
-else
-ifdef CONFIG_CC_STACKPROTECTOR_REGULAR
-  stackp-flag := -fstack-protector
-  stackp-name := REGULAR
-else
-ifdef CONFIG_CC_STACKPROTECTOR_STRONG
-  stackp-flag := -fstack-protector-strong
-  stackp-name := STRONG
-else
-  # If either there is no stack protector for this architecture or
-  # CONFIG_CC_STACKPROTECTOR_NONE is selected, we're done, and $(stackp-name)
-  # is empty, skipping all remaining stack protector tests.
-  #
-  # Force off for distro compilers that enable stack protector by default.
-  KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
-endif
-endif
-endif
-# Find arch-specific stack protector compiler sanity-checking script.
-ifdef stackp-name
-ifneq ($(stackp-flag),)
-  stackp-path := $(srctree)/scripts/gcc-$(SRCARCH)_$(BITS)-has-stack-protector.sh
-  stackp-check := $(wildcard $(stackp-path))
-  # If the wildcard test matches a test script, run it to check functionality.
-  ifdef stackp-check
-    ifneq ($(shell $(CONFIG_SHELL) $(stackp-check) $(CC) $(KBUILD_CPPFLAGS) $(biarch)),y)
-      stackp-broken := y
-    endif
-  endif
-  ifndef stackp-broken
-    # If the stack protector is functional, enable code that depends on it.
-    KBUILD_CPPFLAGS += -DCONFIG_CC_STACKPROTECTOR
-    # Either we've already detected the flag (for AUTO) or we'll fail the
-    # build in the prepare-compiler-check rule (for specific flag).
-    KBUILD_CFLAGS += $(stackp-flag)
-  else
-    # We have to make sure stack protector is unconditionally disabled if
-    # the compiler is broken (in case we're going to continue the build in
-    # AUTO mode).
-    KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
-  endif
-endif
-endif
+stackp-flags-y					:= -fno-stack-protector
+stackp-flags-$(CONFIG_CC_STACKPROTECTOR)	:= -fstack-protector
+stackp-flags-$(CONFIG_CC_STACKPROTECTOR_STRONG)	:= -fstack-protector-strong
+
+KBUILD_CFLAGS += $(stackp-flags-y)
 
 ifeq ($(cc-name),clang)
 KBUILD_CPPFLAGS += $(call cc-option,-Qunused-arguments,)
@@ -1092,7 +1048,7 @@ endif
 # prepare2 creates a makefile if using a separate output directory.
 # From this point forward, .config has been reprocessed, so any rules
 # that need to depend on updated CONFIG_* values can be checked here.
-prepare2: prepare3 prepare-compiler-check outputmakefile asm-generic
+prepare2: prepare3 outputmakefile asm-generic
 
 prepare1: prepare2 $(version_h) $(autoksyms_h) include/generated/utsrelease.h \
                    include/config/auto.conf
@@ -1118,43 +1074,6 @@ uapi-asm-generic:
 PHONY += prepare-objtool
 prepare-objtool: $(objtool_target)
 
-# Check for CONFIG flags that require compiler support. Abort the build
-# after .config has been processed, but before the kernel build starts.
-#
-# For security-sensitive CONFIG options, we don't want to fallback and/or
-# silently change which compiler flags will be used, since that leads to
-# producing kernels with different security feature characteristics
-# depending on the compiler used. (For example, "But I selected
-# CC_STACKPROTECTOR_STRONG! Why did it build with _REGULAR?!")
-PHONY += prepare-compiler-check
-prepare-compiler-check: FORCE
-# Make sure compiler supports requested stack protector flag.
-ifdef stackp-name
-  # Warn about CONFIG_CC_STACKPROTECTOR_AUTO having found no option.
-  ifeq ($(stackp-flag),)
-	@echo CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-		  Compiler does not support any known stack-protector >&2
-  else
-  # Fail if specifically requested stack protector is missing.
-  ifeq ($(call cc-option, $(stackp-flag)),)
-	@echo Cannot use CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-		  $(stackp-flag) not supported by compiler >&2 && exit 1
-  endif
-  endif
-endif
-# Make sure compiler does not have buggy stack-protector support. If a
-# specific stack-protector was requested, fail the build, otherwise warn.
-ifdef stackp-broken
-  ifeq ($(stackp-name),AUTO)
-	@echo CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-                  $(stackp-flag) available but compiler is broken: disabling >&2
-  else
-	@echo Cannot use CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-                  $(stackp-flag) available but compiler is broken >&2 && exit 1
-  endif
-endif
-	@:
-
 # Generate some files
 # ---------------------------------------------------------------------------
 
diff --git a/arch/Kconfig b/arch/Kconfig
index 8e0d665..b42378d 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -535,13 +535,13 @@ config HAVE_CC_STACKPROTECTOR
 	bool
 	help
 	  An arch should select this symbol if:
-	  - its compiler supports the -fstack-protector option
 	  - it has implemented a stack canary (e.g. __stack_chk_guard)
 
-choice
-	prompt "Stack Protector buffer overflow detection"
+config CC_STACKPROTECTOR
+	bool "Stack Protector buffer overflow detection"
 	depends on HAVE_CC_STACKPROTECTOR
-	default CC_STACKPROTECTOR_AUTO
+	depends on $(cc-option -fstack-protector)
+	default y
 	help
 	  This option turns on the "stack-protector" GCC feature. This
 	  feature puts, at the beginning of functions, a canary value on
@@ -551,14 +551,6 @@ choice
 	  overwrite the canary, which gets detected and the attack is then
 	  neutralized via a kernel panic.
 
-config CC_STACKPROTECTOR_NONE
-	bool "None"
-	help
-	  Disable "stack-protector" GCC feature.
-
-config CC_STACKPROTECTOR_REGULAR
-	bool "Regular"
-	help
 	  Functions will have the stack-protector canary logic added if they
 	  have an 8-byte or larger character array on the stack.
 
@@ -570,7 +562,10 @@ config CC_STACKPROTECTOR_REGULAR
 	  by about 0.3%.
 
 config CC_STACKPROTECTOR_STRONG
-	bool "Strong"
+	bool "Strong Stack Protector"
+	depends on CC_STACKPROTECTOR
+	depends on $(cc-option -fstack-protector-strong)
+	default y
 	help
 	  Functions will have the stack-protector canary logic added in any
 	  of the following conditions:
@@ -588,14 +583,6 @@ config CC_STACKPROTECTOR_STRONG
 	  about 20% of all kernel functions, which increases the kernel code
 	  size by about 2%.
 
-config CC_STACKPROTECTOR_AUTO
-	bool "Automatic"
-	help
-	  If the compiler supports it, the best available stack-protector
-	  option will be chosen.
-
-endchoice
-
 config LD_DEAD_CODE_DATA_ELIMINATION
 	bool
 	help
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 986fb0a..946bd9b 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -124,7 +124,7 @@ config X86
 	select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64
 	select HAVE_ARCH_VMAP_STACK		if X86_64
 	select HAVE_ARCH_WITHIN_STACK_FRAMES
-	select HAVE_CC_STACKPROTECTOR
+	select HAVE_CC_STACKPROTECTOR		if CC_HAS_SANE_STACKPROTECTOR
 	select HAVE_CMPXCHG_DOUBLE
 	select HAVE_CMPXCHG_LOCAL
 	select HAVE_CONTEXT_TRACKING		if X86_64
@@ -340,6 +340,12 @@ config PGTABLE_LEVELS
 	default 2
 
 source "init/Kconfig"
+
+config CC_HAS_SANE_STACKPROTECTOR
+	bool
+	default $(success $srctree/scripts/gcc-x86_64-has-stack-protector.sh $CC) if 64BIT
+	default $(success $srctree/scripts/gcc-x86_32-has-stack-protector.sh $CC)
+
 source "kernel/Kconfig.freezer"
 
 menu "Processor type and features"
diff --git a/scripts/gcc-x86_32-has-stack-protector.sh b/scripts/gcc-x86_32-has-stack-protector.sh
index 6b2aeef..f5c1194 100755
--- a/scripts/gcc-x86_32-has-stack-protector.sh
+++ b/scripts/gcc-x86_32-has-stack-protector.sh
@@ -1,9 +1,4 @@
 #!/bin/sh
 # SPDX-License-Identifier: GPL-2.0
 
-echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -O0 -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
-if [ "$?" -eq "0" ] ; then
-	echo y
-else
-	echo n
-fi
+echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -m32 -O0 -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
diff --git a/scripts/gcc-x86_64-has-stack-protector.sh b/scripts/gcc-x86_64-has-stack-protector.sh
index 4a48bdc..3755af0 100755
--- a/scripts/gcc-x86_64-has-stack-protector.sh
+++ b/scripts/gcc-x86_64-has-stack-protector.sh
@@ -2,8 +2,3 @@
 # SPDX-License-Identifier: GPL-2.0
 
 echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -O0 -mcmodel=kernel -fno-PIE -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
-if [ "$?" -eq "0" ] ; then
-	echo y
-else
-	echo n
-fi
-- 
2.7.4

[PATCH v2 12/21] kconfig: show compiler version text in the top comment

[Masahiro Yamada]

The kernel configuration phase is now tightly coupled with the compiler
in use.  It will be nice to show the compiler information in Kconfig.

The compiler information will be displayed like this:

  $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- config
  scripts/kconfig/conf  --oldaskconfig Kconfig
  *
  * Linux/arm64 4.16.0-rc1 Kernel Configuration
  *
  *
  * Compiler: aarch64-linux-gnu-gcc (Linaro GCC 7.2-2017.11) 7.2.1 20171011
  *
  *
  * General setup
  *
  Compile also drivers which will not load (COMPILE_TEST) [N/y/?]

If you use GUI methods such as menuconfig, it will be displayed in the
top menu.

This is simply implemented by using 'comment'.  So, it will be saved
into the .config file as well.

This commit has a very important meaning.  If the compiler is upgraded,
Kconfig must be re-run since different compilers have different sets
of supported options.

All referenced environments are written to include/config/auto.conf.cmd
so that any environment change triggers syncconfig, and prompt the user
to input new values if needed.

With this commit, something like follows will be added to
include/config/auto.conf.cmd

  ifneq "$(CC_VERSION_TEXT)" "aarch64-linux-gnu-gcc (Linaro GCC 7.2-2017.11) 7.2.1 20171011"
  include/config/auto.conf: FORCE
  endif

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2: None

 Kconfig  | 2 ++
 Makefile | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/Kconfig b/Kconfig
index e6ece5b..99ed4b8 100644
--- a/Kconfig
+++ b/Kconfig
@@ -5,4 +5,6 @@
 #
 mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
 
+comment "Compiler: $CC_VERSION_TEXT"
+
 source "arch/$SRCARCH/Kconfig"
diff --git a/Makefile b/Makefile
index 5cadffa..3dc9eb8 100644
--- a/Makefile
+++ b/Makefile
@@ -450,6 +450,8 @@ export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_LDFLAGS_MODULE
 export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL
 export KBUILD_ARFLAGS
 
+export CC_VERSION_TEXT := $(shell $(CC) --version | head -n 1)
+
 # When compiling out-of-tree modules, put MODVERDIR in the module
 # tree rather than in the kernel tree. The kernel tree might
 # even be read-only.
-- 
2.7.4

[PATCH v2 13/21] kconfig: add CC_IS_GCC and GCC_VERSION

[Masahiro Yamada]

This will be useful to specify the required compiler version,
like this:

config FOO
        bool "Use Foo"
        depends on GCC_VERSION >= 408000
        help
          This feature requires GCC 4.8 or newer.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2: None

 init/Kconfig | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/init/Kconfig b/init/Kconfig
index c456030..923cab5 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -14,6 +14,14 @@ macro success $(shell ($(1) && echo y) || echo n)
 # expanded to y if the given compiler flag is supported, n otherwise
 macro cc-option $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
 
+config CC_IS_GCC
+	def_bool $(success $CC --version | grep -q gcc)
+
+config GCC_VERSION
+	int
+	default $(shell $srctree/scripts/gcc-version.sh -p $CC | sed 's/^0*//') if CC_IS_GCC
+	default 0
+
 config CONSTRUCTORS
 	bool
 	depends on !UML
-- 
2.7.4

[PATCH v2 14/21] kconfig: add CC_IS_CLANG and CLANG_VERSION

[Masahiro Yamada]

This will be useful to describe the clang version dependency.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2: None

 init/Kconfig             |  7 +++++++
 scripts/clang-version.sh | 24 +++++++++++-------------
 2 files changed, 18 insertions(+), 13 deletions(-)

diff --git a/init/Kconfig b/init/Kconfig
index 923cab5..5998034 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -22,6 +22,13 @@ config GCC_VERSION
 	default $(shell $srctree/scripts/gcc-version.sh -p $CC | sed 's/^0*//') if CC_IS_GCC
 	default 0
 
+config CC_IS_CLANG
+	def_bool $(success $CC --version | grep -q clang)
+
+config CLANG_VERSION
+	int
+	default $(shell $srctree/scripts/clang-version.sh $CC)
+
 config CONSTRUCTORS
 	bool
 	depends on !UML
diff --git a/scripts/clang-version.sh b/scripts/clang-version.sh
index 9780efa..d8002d2 100755
--- a/scripts/clang-version.sh
+++ b/scripts/clang-version.sh
@@ -10,24 +10,22 @@
 # clang-5.0.1 etc.
 #
 
-if [ "$1" = "-p" ] ; then
-	with_patchlevel=1;
-	shift;
-fi
-
 compiler="$*"
 
 if [ ${#compiler} -eq 0 ]; then
-	echo "Error: No compiler specified."
-	printf "Usage:\n\t$0 <clang-command>\n"
+	echo "Error: No compiler specified." >&2
+	printf "Usage:\n\t$0 <clang-command>\n" >&2
+	echo 0
+	exit 1
+fi
+
+if !( $compiler --version | grep -q clang) ; then
+	echo "This is not clang." >&2
+	echo 0
 	exit 1
 fi
 
 MAJOR=$(echo __clang_major__ | $compiler -E -x c - | tail -n 1)
 MINOR=$(echo __clang_minor__ | $compiler -E -x c - | tail -n 1)
-if [ "x$with_patchlevel" != "x" ] ; then
-	PATCHLEVEL=$(echo __clang_patchlevel__ | $compiler -E -x c - | tail -n 1)
-	printf "%02d%02d%02d\\n" $MAJOR $MINOR $PATCHLEVEL
-else
-	printf "%02d%02d\\n" $MAJOR $MINOR
-fi
+PATCHLEVEL=$(echo __clang_patchlevel__ | $compiler -E -x c - | tail -n 1)
+printf "%d%02d%02d\\n" $MAJOR $MINOR $PATCHLEVEL
-- 
2.7.4

[PATCH v2 15/21] gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT

[Masahiro Yamada]

CONFIG_GCOV_FORMAT_AUTODETECT compiles either gcc_3_4.c or gcc_4_7.c
according to your GCC version.

We can achieve the equivalent behavior by setting reasonable dependency
with the knowledge of the compiler version.

If GCC older than 4.7 is used, GCOV_FORMAT_3_4 is the default, but users
are still allowed to select GCOV_FORMAT_4_7 in case the newer format is
back-ported.

On the other hand, If GCC 4.7 or newer is used, there is no reason to
use GCOV_FORMAT_3_4, so it should be hidden.

If you down-grade your compiler GCC 4.7 or older, oldconfig/syncconfig
will display a prompt for the choice because GCOV_FORMAT_3_4 becomes
visible as a new symbol.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Move the 'depends on' to the choice value.

 kernel/gcov/Kconfig  | 17 +++++------------
 kernel/gcov/Makefile |  2 --
 2 files changed, 5 insertions(+), 14 deletions(-)

diff --git a/kernel/gcov/Kconfig b/kernel/gcov/Kconfig
index 1276aab..1e3823f 100644
--- a/kernel/gcov/Kconfig
+++ b/kernel/gcov/Kconfig
@@ -53,23 +53,16 @@ config GCOV_PROFILE_ALL
 choice
 	prompt "Specify GCOV format"
 	depends on GCOV_KERNEL
-	default GCOV_FORMAT_AUTODETECT
 	---help---
-	The gcov format is usually determined by the GCC version, but there are
+	The gcov format is usually determined by the GCC version, and the
+	default is chosen according to your GCC version. However, there are
 	exceptions where format changes are integrated in lower-version GCCs.
-	In such a case use this option to adjust the format used in the kernel
-	accordingly.
-
-	If unsure, choose "Autodetect".
-
-config GCOV_FORMAT_AUTODETECT
-	bool "Autodetect"
-	---help---
-	Select this option to use the format that corresponds to your GCC
-	version.
+	In such a case, change this option to adjust the format used in the
+	kernel accordingly.
 
 config GCOV_FORMAT_3_4
 	bool "GCC 3.4 format"
+	depends on CC_IS_GCC && GCC_VERSION < 40700
 	---help---
 	Select this option to use the format defined by GCC 3.4.
 
diff --git a/kernel/gcov/Makefile b/kernel/gcov/Makefile
index c6c50e5..ff06d64 100644
--- a/kernel/gcov/Makefile
+++ b/kernel/gcov/Makefile
@@ -4,5 +4,3 @@ ccflags-y := -DSRCTREE='"$(srctree)"' -DOBJTREE='"$(objtree)"'
 obj-y := base.o fs.o
 obj-$(CONFIG_GCOV_FORMAT_3_4) += gcc_3_4.o
 obj-$(CONFIG_GCOV_FORMAT_4_7) += gcc_4_7.o
-obj-$(CONFIG_GCOV_FORMAT_AUTODETECT) += $(call cc-ifversion, -lt, 0407, \
-							gcc_3_4.o, gcc_4_7.o)
-- 
2.7.4

[PATCH v2 16/21] kcov: imply GCC_PLUGINS and GCC_PLUGIN_SANCOV instead of select'ing them

[Masahiro Yamada]

As Documentation/kbuild/kconfig-language.txt notes, 'select' should be
used with care - it forces a lower limit of another symbol, ignoring
the dependency.  In this case, KCOV can select GCC_PLUGINS even if
'depends on HAVE_GCC_PLUGINS' is unmet.

'imply' is modest enough to observe the dependency, and this makes
sense.  If you enable KCOV, you will probably want to enable
GCC_PLUGIN_SANCOV, but it should not break the dependency.

I also remove unneeded code, I just happened to notice.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
  - Drop depends on GCC_VERSION

---

Changes in v2: None

 lib/Kconfig.debug | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 5be22e4..8243c9d 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -743,8 +743,8 @@ config KCOV
 	bool "Code coverage for fuzzing"
 	depends on ARCH_HAS_KCOV
 	select DEBUG_FS
-	select GCC_PLUGINS if !COMPILE_TEST
-	select GCC_PLUGIN_SANCOV if !COMPILE_TEST
+	imply GCC_PLUGINS
+	imply GCC_PLUGIN_SANCOV
 	help
 	  KCOV exposes kernel code coverage information in a form suitable
 	  for coverage-guided fuzzing (randomized testing).
@@ -758,7 +758,6 @@ config KCOV
 config KCOV_ENABLE_COMPARISONS
 	bool "Enable comparison operands collection by KCOV"
 	depends on KCOV
-	default n
 	help
 	  KCOV also exposes operands of every comparison in the instrumented
 	  code along with operand sizes and PCs of the comparison instructions.
@@ -768,7 +767,7 @@ config KCOV_ENABLE_COMPARISONS
 config KCOV_INSTRUMENT_ALL
 	bool "Instrument all code by default"
 	depends on KCOV
-	default y if KCOV
+	default y
 	help
 	  If you are doing generic system call fuzzing (like e.g. syzkaller),
 	  then you will want to instrument the whole kernel and you should
-- 
2.7.4

[PATCH v2 17/21] gcc-plugins: always build plugins with C++

[Masahiro Yamada]

If the target compiler is GCC 4.8 or newer, plugins are compiled with
HOSTCXX.  Otherwise, gcc-plugin.sh will select HOSTCC or HOSTCXX.

To simplify things, drop the HOSTCC support for plugins.  If you use
GCC plugins, the requirement for the target compiler is GCC 4.8 (or
older compiler with necessary features backported).  This is a feature
for advanced users, so this requirement would not be a big deal.
Currently, building plugins requires GCC 4.5, so this is not a big jump.

Remove the code that was needed to build plugins with HOSTCC as a
preparation for Kconfig migration.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2: None

 scripts/Makefile.gcc-plugins | 11 +++--------
 scripts/gcc-plugin.sh        | 38 +++-----------------------------------
 scripts/gcc-plugins/Makefile | 15 ++++-----------
 3 files changed, 10 insertions(+), 54 deletions(-)

diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index b2a95af..b0f9108 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -1,7 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 ifdef CONFIG_GCC_PLUGINS
-  __PLUGINCC := $(call cc-ifversion, -ge, 0408, $(HOSTCXX), $(HOSTCC))
-  PLUGINCC := $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-plugin.sh "$(__PLUGINCC)" "$(HOSTCXX)" "$(CC)")
+  PLUGINCC := $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-plugin.sh $(HOSTCXX) $(CC))
 
   SANCOV_PLUGIN := -fplugin=$(objtree)/scripts/gcc-plugins/sancov_plugin.so
 
@@ -65,12 +64,8 @@ ifdef CONFIG_GCC_PLUGINS
 	  @echo "Cannot use CONFIG_GCC_PLUGINS: plugin support on gcc <= 5.1 is buggy on powerpc, please upgrade to gcc 5.2 or newer" >&2 && exit 1
         endif
       endif
-      ifeq ($(call cc-ifversion, -ge, 0405, y), y)
-	$(Q)$(srctree)/scripts/gcc-plugin.sh --show-error "$(__PLUGINCC)" "$(HOSTCXX)" "$(CC)" || true
-	@echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
-      else
-	@echo "Cannot use CONFIG_GCC_PLUGINS: your gcc version does not support plugins, you should upgrade it to at least gcc 4.5" >&2 && exit 1
-      endif
+      $(Q)$(srctree)/scripts/gcc-plugin.sh --show-error $(HOSTCXX) $(CC) || true
+      @echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
     endif
   endif
 endif
diff --git a/scripts/gcc-plugin.sh b/scripts/gcc-plugin.sh
index d3caefe..0edbdae 100755
--- a/scripts/gcc-plugin.sh
+++ b/scripts/gcc-plugin.sh
@@ -8,42 +8,10 @@ if [ "$1" = "--show-error" ] ; then
 	shift || true
 fi
 
-gccplugins_dir=$($3 -print-file-name=plugin)
-plugincc=$($1 -E -x c++ - -o /dev/null -I"${srctree}"/gcc-plugins -I"${gccplugins_dir}"/include 2>&1 <<EOF
-#include "gcc-common.h"
-#if BUILDING_GCC_VERSION >= 4008 || defined(ENABLE_BUILD_WITH_CXX)
-#warning $2 CXX
-#else
-#warning $1 CC
-#endif
-EOF
-)
-
-if [ $? -ne 0 ]
-then
-	if [ -n "$SHOW_ERROR" ] ; then
-		echo "${plugincc}" >&2
-	fi
-	exit 1
-fi
-
-case "$plugincc" in
-	*"$1 CC"*)
-		echo "$1"
-		exit 0
-		;;
-
-	*"$2 CXX"*)
-		# the c++ compiler needs another test, see below
-		;;
-
-	*)
-		exit 1
-		;;
-esac
+gccplugins_dir=$($2 -print-file-name=plugin)
 
 # we need a c++ compiler that supports the designated initializer GNU extension
-plugincc=$($2 -c -x c++ -std=gnu++98 - -fsyntax-only -I"${srctree}"/gcc-plugins -I"${gccplugins_dir}"/include 2>&1 <<EOF
+plugincc=$($1 -c -x c++ -std=gnu++98 - -fsyntax-only -I"${srctree}"/gcc-plugins -I"${gccplugins_dir}"/include 2>&1 <<EOF
 #include "gcc-common.h"
 class test {
 public:
@@ -56,7 +24,7 @@ EOF
 
 if [ $? -eq 0 ]
 then
-	echo "$2"
+	echo "$1"
 	exit 0
 fi
 
diff --git a/scripts/gcc-plugins/Makefile b/scripts/gcc-plugins/Makefile
index e2ff425..47d5f69 100644
--- a/scripts/gcc-plugins/Makefile
+++ b/scripts/gcc-plugins/Makefile
@@ -1,17 +1,10 @@
 # SPDX-License-Identifier: GPL-2.0
 GCC_PLUGINS_DIR := $(shell $(CC) -print-file-name=plugin)
 
-ifeq ($(PLUGINCC),$(HOSTCC))
-  HOSTLIBS := hostlibs
-  HOST_EXTRACFLAGS += -I$(GCC_PLUGINS_DIR)/include -I$(src) -std=gnu99 -ggdb
-  export HOST_EXTRACFLAGS
-else
-  HOSTLIBS := hostcxxlibs
-  HOST_EXTRACXXFLAGS += -I$(GCC_PLUGINS_DIR)/include -I$(src) -std=gnu++98 -fno-rtti
-  HOST_EXTRACXXFLAGS += -fno-exceptions -fasynchronous-unwind-tables -ggdb
-  HOST_EXTRACXXFLAGS += -Wno-narrowing -Wno-unused-variable
-  export HOST_EXTRACXXFLAGS
-endif
+HOSTLIBS := hostcxxlibs
+HOST_EXTRACXXFLAGS += -I$(GCC_PLUGINS_DIR)/include -I$(src) -std=gnu++98 -fno-rtti
+HOST_EXTRACXXFLAGS += -fno-exceptions -fasynchronous-unwind-tables -ggdb
+HOST_EXTRACXXFLAGS += -Wno-narrowing -Wno-unused-variable
 
 ifneq ($(CFLAGS_KCOV), $(SANCOV_PLUGIN))
   GCC_PLUGIN := $(filter-out $(SANCOV_PLUGIN), $(GCC_PLUGIN))
-- 
2.7.4

[PATCH v2 18/21] gcc-plugins: move GCC version check for PowerPC to Kconfig

[Masahiro Yamada]

For PowerPC, GCC 5.2 is the requirement for GCC plugins.  Move the
version check to Kconfig so that the GCC plugin menus will be hidden
if an older compiler is in use.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
---

Changes in v2: None

 arch/powerpc/Kconfig         | 2 +-
 scripts/Makefile.gcc-plugins | 8 --------
 2 files changed, 1 insertion(+), 9 deletions(-)

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 73ce5dd..b8474c2 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -195,7 +195,7 @@ config PPC
 	select HAVE_FTRACE_MCOUNT_RECORD
 	select HAVE_FUNCTION_GRAPH_TRACER
 	select HAVE_FUNCTION_TRACER
-	select HAVE_GCC_PLUGINS
+	select HAVE_GCC_PLUGINS			if GCC_VERSION >= 50200
 	select HAVE_GENERIC_GUP
 	select HAVE_HW_BREAKPOINT		if PERF_EVENTS && (PPC_BOOK3S || PPC_8xx)
 	select HAVE_IDE
diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index b0f9108..25da4c0 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -56,14 +56,6 @@ gcc-plugins-check: FORCE
 ifdef CONFIG_GCC_PLUGINS
   ifeq ($(PLUGINCC),)
     ifneq ($(GCC_PLUGINS_CFLAGS),)
-      # Various gccs between 4.5 and 5.1 have bugs on powerpc due to missing
-      # header files. gcc <= 4.6 doesn't work at all, gccs from 4.8 to 5.1 have
-      # issues with 64-bit targets.
-      ifeq ($(ARCH),powerpc)
-        ifeq ($(call cc-ifversion, -le, 0501, y), y)
-	  @echo "Cannot use CONFIG_GCC_PLUGINS: plugin support on gcc <= 5.1 is buggy on powerpc, please upgrade to gcc 5.2 or newer" >&2 && exit 1
-        endif
-      endif
       $(Q)$(srctree)/scripts/gcc-plugin.sh --show-error $(HOSTCXX) $(CC) || true
       @echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
     endif
-- 
2.7.4

[PATCH v2 19/21] gcc-plugins: test GCC plugin support in Kconfig

[Masahiro Yamada]

Run scripts/gcc-plugin.sh from Kconfig.  Users can enable GCC_PLUGINS
only when it is supported.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2: None

 arch/Kconfig                 |  4 +++
 scripts/Makefile.gcc-plugins | 82 ++++++++++++++++----------------------------
 scripts/gcc-plugin.sh        |  1 -
 3 files changed, 33 insertions(+), 54 deletions(-)

diff --git a/arch/Kconfig b/arch/Kconfig
index b42378d..88cc925 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -407,9 +407,13 @@ config HAVE_GCC_PLUGINS
 	  An arch should select this symbol if it supports building with
 	  GCC plugins.
 
+config CC_HAS_GCC_PLUGINS
+	bool
+
 menuconfig GCC_PLUGINS
 	bool "GCC plugins"
 	depends on HAVE_GCC_PLUGINS
+	depends on $(success $srctree/scripts/gcc-plugin.sh $HOSTCXX $CC)
 	depends on !COMPILE_TEST
 	help
 	  GCC plugins are loadable modules that provide extra features to the
diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index 25da4c0..19d0d5b 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -1,71 +1,47 @@
 # SPDX-License-Identifier: GPL-2.0
-ifdef CONFIG_GCC_PLUGINS
-  PLUGINCC := $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-plugin.sh $(HOSTCXX) $(CC))
-
-  SANCOV_PLUGIN := -fplugin=$(objtree)/scripts/gcc-plugins/sancov_plugin.so
+SANCOV_PLUGIN := -fplugin=$(objtree)/scripts/gcc-plugins/sancov_plugin.so
 
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_CYC_COMPLEXITY)	+= cyc_complexity_plugin.so
+gcc-plugin-$(CONFIG_GCC_PLUGIN_CYC_COMPLEXITY)	+= cyc_complexity_plugin.so
 
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= latent_entropy_plugin.so
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= -DLATENT_ENTROPY_PLUGIN
-  ifdef CONFIG_GCC_PLUGIN_LATENT_ENTROPY
-    DISABLE_LATENT_ENTROPY_PLUGIN			+= -fplugin-arg-latent_entropy_plugin-disable
-  endif
+gcc-plugin-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= latent_entropy_plugin.so
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= -DLATENT_ENTROPY_PLUGIN
+ifdef CONFIG_GCC_PLUGIN_LATENT_ENTROPY
+  DISABLE_LATENT_ENTROPY_PLUGIN			+= -fplugin-arg-latent_entropy_plugin-disable
+endif
 
-  ifdef CONFIG_GCC_PLUGIN_SANCOV
-    ifeq ($(CFLAGS_KCOV),)
-      # It is needed because of the gcc-plugin.sh and gcc version checks.
-      gcc-plugin-$(CONFIG_GCC_PLUGIN_SANCOV)           += sancov_plugin.so
+ifdef CONFIG_GCC_PLUGIN_SANCOV
+  ifeq ($(CFLAGS_KCOV),)
+    # It is needed because of the gcc-plugin.sh and gcc version checks.
+    gcc-plugin-$(CONFIG_GCC_PLUGIN_SANCOV)           += sancov_plugin.so
 
-      ifneq ($(PLUGINCC),)
-        CFLAGS_KCOV := $(SANCOV_PLUGIN)
-      else
-        $(warning warning: cannot use CONFIG_KCOV: -fsanitize-coverage=trace-pc is not supported by compiler)
-      endif
-    endif
+    CFLAGS_KCOV := $(SANCOV_PLUGIN)
   endif
+endif
 
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= structleak_plugin.so
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_VERBOSE)	+= -fplugin-arg-structleak_plugin-verbose
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)	+= -fplugin-arg-structleak_plugin-byref-all
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= -DSTRUCTLEAK_PLUGIN
-
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= randomize_layout_plugin.so
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= -DRANDSTRUCT_PLUGIN
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT_PERFORMANCE)	+= -fplugin-arg-randomize_layout_plugin-performance-mode
+gcc-plugin-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= structleak_plugin.so
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_VERBOSE)	+= -fplugin-arg-structleak_plugin-verbose
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)	+= -fplugin-arg-structleak_plugin-byref-all
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= -DSTRUCTLEAK_PLUGIN
 
-  GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y))
+gcc-plugin-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= randomize_layout_plugin.so
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= -DRANDSTRUCT_PLUGIN
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT_PERFORMANCE)	+= -fplugin-arg-randomize_layout_plugin-performance-mode
 
-  export PLUGINCC GCC_PLUGINS_CFLAGS GCC_PLUGIN GCC_PLUGIN_SUBDIR
-  export SANCOV_PLUGIN DISABLE_LATENT_ENTROPY_PLUGIN
+GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y))
 
-  ifneq ($(PLUGINCC),)
-    # SANCOV_PLUGIN can be only in CFLAGS_KCOV because avoid duplication.
-    GCC_PLUGINS_CFLAGS := $(filter-out $(SANCOV_PLUGIN), $(GCC_PLUGINS_CFLAGS))
-  endif
+export GCC_PLUGINS_CFLAGS GCC_PLUGIN GCC_PLUGIN_SUBDIR
+export SANCOV_PLUGIN DISABLE_LATENT_ENTROPY_PLUGIN
 
-  KBUILD_CFLAGS += $(GCC_PLUGINS_CFLAGS)
-  GCC_PLUGIN := $(gcc-plugin-y)
-  GCC_PLUGIN_SUBDIR := $(gcc-plugin-subdir-y)
-endif
+# SANCOV_PLUGIN can be only in CFLAGS_KCOV because avoid duplication.
+GCC_PLUGINS_CFLAGS := $(filter-out $(SANCOV_PLUGIN), $(GCC_PLUGINS_CFLAGS))
 
-# If plugins aren't supported, abort the build before hard-to-read compiler
-# errors start getting spewed by the main build.
-PHONY += gcc-plugins-check
-gcc-plugins-check: FORCE
-ifdef CONFIG_GCC_PLUGINS
-  ifeq ($(PLUGINCC),)
-    ifneq ($(GCC_PLUGINS_CFLAGS),)
-      $(Q)$(srctree)/scripts/gcc-plugin.sh --show-error $(HOSTCXX) $(CC) || true
-      @echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
-    endif
-  endif
-endif
-	@:
+KBUILD_CFLAGS += $(GCC_PLUGINS_CFLAGS)
+GCC_PLUGIN := $(gcc-plugin-y)
+GCC_PLUGIN_SUBDIR := $(gcc-plugin-subdir-y)
 
 # Actually do the build, if requested.
 PHONY += gcc-plugins
-gcc-plugins: scripts_basic gcc-plugins-check
+gcc-plugins: scripts_basic
 ifdef CONFIG_GCC_PLUGINS
 	$(Q)$(MAKE) $(build)=scripts/gcc-plugins
 endif
diff --git a/scripts/gcc-plugin.sh b/scripts/gcc-plugin.sh
index 0edbdae..b18c69c 100755
--- a/scripts/gcc-plugin.sh
+++ b/scripts/gcc-plugin.sh
@@ -24,7 +24,6 @@ EOF
 
 if [ $? -eq 0 ]
 then
-	echo "$1"
 	exit 0
 fi
 
-- 
2.7.4

[PATCH v2 20/21] gcc-plugins: enable GCC_PLUGINS for COMPILE_TEST

[Masahiro Yamada]

The plugin availability is checked in Kconfig, so all{yes,mod}config
will not be bothered.  Remove 'depends on !COMPILE_TEST'.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Remove more 'depends on'

 arch/Kconfig | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/arch/Kconfig b/arch/Kconfig
index 88cc925..b07094c 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -414,7 +414,6 @@ menuconfig GCC_PLUGINS
 	bool "GCC plugins"
 	depends on HAVE_GCC_PLUGINS
 	depends on $(success $srctree/scripts/gcc-plugin.sh $HOSTCXX $CC)
-	depends on !COMPILE_TEST
 	help
 	  GCC plugins are loadable modules that provide extra features to the
 	  compiler. They are useful for runtime instrumentation and static analysis.
@@ -424,7 +423,6 @@ menuconfig GCC_PLUGINS
 config GCC_PLUGIN_CYC_COMPLEXITY
 	bool "Compute the cyclomatic complexity of a function" if EXPERT
 	depends on GCC_PLUGINS
-	depends on !COMPILE_TEST
 	help
 	  The complexity M of a function's control flow graph is defined as:
 	   M = E - N + 2P
@@ -487,7 +485,6 @@ config GCC_PLUGIN_STRUCTLEAK_BYREF_ALL
 config GCC_PLUGIN_STRUCTLEAK_VERBOSE
 	bool "Report forcefully initialized variables"
 	depends on GCC_PLUGIN_STRUCTLEAK
-	depends on !COMPILE_TEST
 	help
 	  This option will cause a warning to be printed each time the
 	  structleak plugin finds a variable it thinks needs to be
@@ -527,7 +524,6 @@ config GCC_PLUGIN_RANDSTRUCT
 config GCC_PLUGIN_RANDSTRUCT_PERFORMANCE
 	bool "Use cacheline-aware structure randomization"
 	depends on GCC_PLUGIN_RANDSTRUCT
-	depends on !COMPILE_TEST
 	help
 	  If you say Y here, the RANDSTRUCT randomization will make a
 	  best effort at restricting randomization to cacheline-sized
-- 
2.7.4

[PATCH v2 21/21] arm64: move GCC version check for ARCH_SUPPORTS_INT128 to Kconfig

[Masahiro Yamada]

This becomes much neater in Kconfig.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Newly added

 arch/arm64/Kconfig  | 1 +
 arch/arm64/Makefile | 2 --
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 7381eeb..0c97f40 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -43,6 +43,7 @@ config ARM64
 	select ARCH_USE_QUEUED_RWLOCKS
 	select ARCH_SUPPORTS_MEMORY_FAILURE
 	select ARCH_SUPPORTS_ATOMIC_RMW
+	select ARCH_SUPPORTS_INT128 if GCC_VERSION >= 50000
 	select ARCH_SUPPORTS_NUMA_BALANCING
 	select ARCH_WANT_COMPAT_IPC_PARSE_VERSION
 	select ARCH_WANT_FRAME_POINTERS
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index b481b4a..7318165 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -57,8 +57,6 @@ KBUILD_AFLAGS	+= $(lseinstr) $(brokengasinst)
 KBUILD_CFLAGS	+= $(call cc-option,-mabi=lp64)
 KBUILD_AFLAGS	+= $(call cc-option,-mabi=lp64)
 
-KBUILD_CFLAGS	+= $(call cc-ifversion, -ge, 0500, -DCONFIG_ARCH_SUPPORTS_INT128)
-
 ifeq ($(CONFIG_CPU_BIG_ENDIAN), y)
 KBUILD_CPPFLAGS	+= -mbig-endian
 CHECKFLAGS	+= -D__AARCH64EB__
-- 
2.7.4

Re: [PATCH v2 15/21] gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT

[Peter Oberparleiter]

On 27.03.2018 07:29, Masahiro Yamada wrote:
> CONFIG_GCOV_FORMAT_AUTODETECT compiles either gcc_3_4.c or gcc_4_7.c
> according to your GCC version.
> 
> We can achieve the equivalent behavior by setting reasonable dependency
> with the knowledge of the compiler version.
> 
> If GCC older than 4.7 is used, GCOV_FORMAT_3_4 is the default, but users
> are still allowed to select GCOV_FORMAT_4_7 in case the newer format is
> back-ported.
> 
> On the other hand, If GCC 4.7 or newer is used, there is no reason to
> use GCOV_FORMAT_3_4, so it should be hidden.
> 
> If you down-grade your compiler GCC 4.7 or older, oldconfig/syncconfig
> will display a prompt for the choice because GCOV_FORMAT_3_4 becomes
> visible as a new symbol.

Looks sane to me.

Acked-by: Peter Oberparleiter <oberpar@linux.vnet.ibm.com>

-- 
Peter Oberparleiter
Linux on Z Development - IBM Germany

Re: [PATCH v2 00/21] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

2018-03-27 14:29 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:
>
> Here is v2 to start to move compiler capability tests to Kconfig.
>
> V1:
> https://lkml.org/lkml/2018/2/16/610
>
> I brushed up the implementation even more.
>
> Major changes for v2:
> [1] Environments and functions are expanded in the lexer (zconf.l)
>     The parser (zconf.y) receives expanded tokens.  This simplifies
>     the implementation.
>
> [2] Implement only one built-in function 'shell'.  This returns
>     stdout from the command.  We need something to return 'y' or 'n'.
>     This is now implemented as a macro 'success'.
>
> [3] Macros (user-defined function) are defined by 'macro' keyword
>     instead of string type symbol.
>
>     ex1) macro success $(shell ($(1) && echo y) || echo n)
>     ex2) macro cc-option $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
>
> My plan for v3
> --------------
>
> The MW is approaching, but I'd like to make it more lovely if
> I have some time.
>
> While I was writing these patches, I noticed a kind of similarity
> between Makefile and Kconfig.
>
> Makefile can define variables and functions to do text processing
> during the parse phase.  After parsing, it builds up dependency
> graph, and moves on to the execution phase.  Then, it generates
> objects as needed by checking time-stamps.
>
> [Makefile]
>
>   OBJ := foo.o
>   SRC := foo.c
>   CC := gcc
>
>   $(OBJ): $(SRC)
>             $(CC) -c -o $(OBJ) $(SRC)
>
> [Makefile after text processing]
>
>   foo.o: foo.c
>           gcc -c -o foo.o foo.c
>
> Now, we are adding tools for text processing to Kconfig.  Functions
> are statically expanded by the lexer.  After all source files are
> parsed, Kconfig moves on to the evaluation phase.  Then, users can
> tweak symbol values from menus.
>
> So, we should be able to describe Kconfig like follows:
>
> [Kconfig]
>
>    MY_SYMBOL := CC_STACKPROTECTOR
>    MY_PROMPT := "Stackprotector"
>    success = $(shell ($(1) && echo y) || echo n)
>    cc-option = $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
>
>    config $(MY_SYMBOL)
>            bool $(MY_PROMPT)
>            depends on $(cc-option -fstack-protector)
>
> [Kconfig after text processing]
>
>    config CC_STACKPROTECTOR
>            bool "Stackptector"
>            depends on y
>
> So, I think it is better to use '=' assignment for defining functions.
> I will remove the 'macro' keyword.
> Also, support variables.  A variable is a function with no argument.
> So, it will be easy to implement it as a sub-set of function feature.
> It is better to support two flavors of assignments, '=' and ':=' as well.
>
> Masahiro Yamada (21):
>   kbuild: remove kbuild cache
>   kbuild: remove CONFIG_CROSS_COMPILE support
>   kconfig: move and rename sym_expand_string_value()
>   kconfig: reference environments directly and remove 'option env='
>     syntax
>   kconfig: remove string expansion in file_lookup()
>   kconfig: remove string expansion for mainmenu after yyparse()
>   kconfig: add function support and implement 'shell' function
>   kconfig: replace $UNAME_RELEASE with function call
>   kconfig: add 'macro' keyword to support user-defined function
>   kconfig: add 'success' and 'cc-option' macros
>   stack-protector: test compiler capability in Kconfig and drop AUTO
>     mode
>   kconfig: show compiler version text in the top comment
>   kconfig: add CC_IS_GCC and GCC_VERSION
>   kconfig: add CC_IS_CLANG and CLANG_VERSION
>   gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT
>   kcov: imply GCC_PLUGINS and GCC_PLUGIN_SANCOV instead of select'ing
>     them
>   gcc-plugins: always build plugins with C++
>   gcc-plugins: move GCC version check for PowerPC to Kconfig
>   gcc-plugins: test GCC plugin support in Kconfig
>   gcc-plugins: enable GCC_PLUGINS for COMPILE_TEST
>   arm64: move GCC version check for ARCH_SUPPORTS_INT128 to Kconfig
>
>  Documentation/kbuild/kconfig-language.txt |   8 --
>  Kconfig                                   |   4 +-
>  Makefile                                  | 103 ++------------
>  arch/Kconfig                              |  37 ++---
>  arch/arm64/Kconfig                        |   1 +
>  arch/arm64/Makefile                       |   2 -
>  arch/powerpc/Kconfig                      |   2 +-
>  arch/sh/Kconfig                           |   4 +-
>  arch/sparc/Kconfig                        |   4 +-
>  arch/tile/Kconfig                         |   2 +-
>  arch/um/Kconfig.common                    |   4 -
>  arch/x86/Kconfig                          |  12 +-
>  arch/x86/um/Kconfig                       |   4 +-
>  init/Kconfig                              |  44 +++---
>  kernel/gcov/Kconfig                       |  17 +--
>  kernel/gcov/Makefile                      |   2 -
>  lib/Kconfig.debug                         |   7 +-
>  scripts/Kbuild.include                    | 101 ++------------
>  scripts/Makefile.gcc-plugins              |  95 ++++---------
>  scripts/clang-version.sh                  |  24 ++--
>  scripts/gcc-plugin.sh                     |  37 +----
>  scripts/gcc-plugins/Makefile              |  15 +-
>  scripts/gcc-x86_32-has-stack-protector.sh |   7 +-
>  scripts/gcc-x86_64-has-stack-protector.sh |   5 -
>  scripts/kconfig/confdata.c                |  31 +---
>  scripts/kconfig/env.c                     |  95 +++++++++++++
>  scripts/kconfig/function.c                | 225 ++++++++++++++++++++++++++++++
>  scripts/kconfig/kconf_id.c                |   1 -
>  scripts/kconfig/lkc.h                     |   9 +-
>  scripts/kconfig/lkc_proto.h               |   7 +-
>  scripts/kconfig/menu.c                    |   3 -
>  scripts/kconfig/symbol.c                  | 109 ---------------
>  scripts/kconfig/util.c                    |  96 ++++++++++---
>  scripts/kconfig/zconf.l                   |  51 ++++++-
>  scripts/kconfig/zconf.y                   |  35 ++---
>  35 files changed, 608 insertions(+), 595 deletions(-)
>  create mode 100644 scripts/kconfig/env.c
>  create mode 100644 scripts/kconfig/function.c
>
> --
> 2.7.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



For convenience, the whole of this series is available at
the following branch.

git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
 kconfig-shell-v2


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 21/21] arm64: move GCC version check for ARCH_SUPPORTS_INT128 to Kconfig

[Will Deacon]

On Tue, Mar 27, 2018 at 02:29:35PM +0900, Masahiro Yamada wrote:
> This becomes much neater in Kconfig.
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
> 
> Changes in v2:
>   - Newly added
> 
>  arch/arm64/Kconfig  | 1 +
>  arch/arm64/Makefile | 2 --
>  2 files changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 7381eeb..0c97f40 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -43,6 +43,7 @@ config ARM64
>  	select ARCH_USE_QUEUED_RWLOCKS
>  	select ARCH_SUPPORTS_MEMORY_FAILURE
>  	select ARCH_SUPPORTS_ATOMIC_RMW
> +	select ARCH_SUPPORTS_INT128 if GCC_VERSION >= 50000
>  	select ARCH_SUPPORTS_NUMA_BALANCING
>  	select ARCH_WANT_COMPAT_IPC_PARSE_VERSION
>  	select ARCH_WANT_FRAME_POINTERS
> diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
> index b481b4a..7318165 100644
> --- a/arch/arm64/Makefile
> +++ b/arch/arm64/Makefile
> @@ -57,8 +57,6 @@ KBUILD_AFLAGS	+= $(lseinstr) $(brokengasinst)
>  KBUILD_CFLAGS	+= $(call cc-option,-mabi=lp64)
>  KBUILD_AFLAGS	+= $(call cc-option,-mabi=lp64)
>  
> -KBUILD_CFLAGS	+= $(call cc-ifversion, -ge, 0500, -DCONFIG_ARCH_SUPPORTS_INT128)
> -

If this does what it claims to do, then I agree that it's much cleaner!

Acked-by: Will Deacon <will.deacon@arm.com>

Will

Re: [PATCH v2 12/21] kconfig: show compiler version text in the top comment

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> The kernel configuration phase is now tightly coupled with the compiler
> in use.  It will be nice to show the compiler information in Kconfig.
>
> The compiler information will be displayed like this:
>
>   $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- config
>   scripts/kconfig/conf  --oldaskconfig Kconfig
>   *
>   * Linux/arm64 4.16.0-rc1 Kernel Configuration
>   *
>   *
>   * Compiler: aarch64-linux-gnu-gcc (Linaro GCC 7.2-2017.11) 7.2.1 20171011
>   *
>   *
>   * General setup
>   *
>   Compile also drivers which will not load (COMPILE_TEST) [N/y/?]
>
> If you use GUI methods such as menuconfig, it will be displayed in the
> top menu.
>
> This is simply implemented by using 'comment'.  So, it will be saved
> into the .config file as well.
>
> This commit has a very important meaning.  If the compiler is upgraded,
> Kconfig must be re-run since different compilers have different sets
> of supported options.
>
> All referenced environments are written to include/config/auto.conf.cmd
> so that any environment change triggers syncconfig, and prompt the user
> to input new values if needed.
>
> With this commit, something like follows will be added to
> include/config/auto.conf.cmd
>
>   ifneq "$(CC_VERSION_TEXT)" "aarch64-linux-gnu-gcc (Linaro GCC 7.2-2017.11) 7.2.1 20171011"
>   include/config/auto.conf: FORCE
>   endif
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

I like seeing this in the config. Good idea!

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

> ---
>
> Changes in v2: None
>
>  Kconfig  | 2 ++
>  Makefile | 2 ++
>  2 files changed, 4 insertions(+)
>
> diff --git a/Kconfig b/Kconfig
> index e6ece5b..99ed4b8 100644
> --- a/Kconfig
> +++ b/Kconfig
> @@ -5,4 +5,6 @@
>  #
>  mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
>
> +comment "Compiler: $CC_VERSION_TEXT"
> +
>  source "arch/$SRCARCH/Kconfig"
> diff --git a/Makefile b/Makefile
> index 5cadffa..3dc9eb8 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -450,6 +450,8 @@ export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_LDFLAGS_MODULE
>  export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL
>  export KBUILD_ARFLAGS
>
> +export CC_VERSION_TEXT := $(shell $(CC) --version | head -n 1)
> +
>  # When compiling out-of-tree modules, put MODVERDIR in the module
>  # tree rather than in the kernel tree. The kernel tree might
>  # even be read-only.
> --
> 2.7.4
>



-- 
Kees Cook
Pixel Security

Re: [PATCH v2 01/21] kbuild: remove kbuild cache

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> The kbuild cache was introduced to remember the result of shell
> commands, some of which are expensive to compute, such as
> $(call cc-option,...).
>
> However, this turned out not so clever as I had first expected.
> Actually, it is problematic.  For example, "$(CC) -print-file-name"
> is cached.  If the compiler is updated, the stale search path causes
> build error, which is difficult to figure out.  Another problem
> scenario is cache files could be touched while install targets are
> running under the root permission.  We can patch them if desired,
> but the build infrastructure is getting uglier and uglier.
>
> Now, we are going to move compiler flag tests to the configuration
> phase.  If this is completed, the result of compiler tests will be
> naturally cached in the .config file.  We will not have performance
> issues of incremental building since this testing only happens at
> Kconfig time.
>
> To start this work with a cleaner code base, remove the kbuild
> cache first.
>
> Revert the following commits:
> Commit 9a234a2e3843 ("kbuild: create directory for make cache only when necessary")
> Commit e17c400ae194 ("kbuild: shrink .cache.mk when it exceeds 1000 lines")
> Commit 4e56207130ed ("kbuild: Cache a few more calls to the compiler")
> Commit 3298b690b21c ("kbuild: Add a cache for generated variables")
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 02/21] kbuild: remove CONFIG_CROSS_COMPILE support

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Kbuild provides a couple of ways to specify CROSS_COMPILE:
>
> [1] Command line
> [2] Environment
> [3] arch/*/Makefile (only some architectures)
> [4] CONFIG_CROSS_COMPILE
>
> [4] is problematic for the compiler capability tests in Kconfig.
> CONFIG_CROSS_COMPILE allows users to change the compiler prefix from
> 'make menuconfig', etc.  It means, the compiler options would have
> to be all re-calculated everytime CONFIG_CROSS_COMPILE is changed.
>
> To avoid complexity and performance issues, I'd like to evaluate
> the shell commands statically, i.e. only parsing Kconfig files.
>
> I guess the majority is [1] or [2].  Currently, there are only
> 4 defconfig files that specify CONFIG_CROSS_COMPILE.
>   arch/arm/configs/lpc18xx_defconfig
>   arch/hexagon/configs/comet_defconfig
>   arch/openrisc/configs/or1ksim_defconfig
>   arch/openrisc/configs/simple_smp_defconfig
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 03/21] kconfig: move and rename sym_expand_string_value()

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> This helper expands symbols contained in a string.  I am about to
> change it to expand environments instead of symbols.  Also, I will
> add function expansion later.
>
> Rename it to expand_string_value(), and move it to util.c, which is
> a more suitable place.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 04/21] kconfig: reference environments directly and remove 'option env=' syntax

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> To get an environment value, Kconfig needs to define a symbol using
> "option env=" syntax.  It is tedious to add a config entry for each
> environment given that we need more environments such as 'CC', 'AS',
> 'srctree' etc. to evaluate the compiler capability in Kconfig.
>
> Adding '$' to symbols is weird.  Kconfig can reference symbols directly
> like this:
>
>   config FOO
>           string
>           default BAR
>
> So, I want to use the following syntax to get environment 'BAR' from
> the system:
>
>   config FOO
>           string
>           default $BAR
>
> Looking at the code, the symbols prefixed with 'S' are expanded by:
>  - conf_expand_value()
>    This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
>  - expand_string_value()
>    This is used to expand strings in 'source' and 'mainmenu'
>
> All of them are fixed values independent of user configuration.  So,
> this kind of syntax should be moved to simply take the environment.
>
> This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
> 'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.
>
> sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
> should be be given from the environment.
>
> ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
> by 'default ARCH_DEFCONFIG'.
>
> The environments are expanding in the lexer; when '$' is encountered,
> it is expanded, and resulted strings are pushed back to the input
> stream.  This makes the implementation simpler.
>
> For example, the following code works.
>
> [Example code]
>
>   config TOOLCHAIN_LIST
>           string
>           default "My tools: CC=$CC, AS=$AS, CPP=$CPP"
>
> [Result]
>
>   $ make -s alldefconfig && tail -n 1 .config
>   CONFIG_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

For easier review this patch could maybe get split into a couple (i.e.
refactoring of things like env_write_dep()) but it's probably fine
as-is. :)

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 05/21] kconfig: remove string expansion in file_lookup()

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> There are two callers of file_lookup().
>
> [1] zconf_initscan()
>     This is used to open the first Kconfig.  However, it cannot
>     contain environments in the file path because zconf_fopen() is
>     called before file_lookup().  By swapping the call order,
>     KBUILD_KCONFIG would be able to contain environments, but I do
>     not see practical benefits to support it.
>
> [2] zconf_nextfile()
>     This is used to open the next file from 'source' statement like
>         source "arch/$SRCARCH/Kconfig"
>     but this has already been expanded in the lexer phase.
>
> So, file_lookup() does not need to expand the given path.
>
> By the way, file_lookup() was already buggy; it expanded a given path,
> but it used the path before expansion for look-up:
>         if (!strcmp(name, file->name)) {
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 06/21] kconfig: remove string expansion for mainmenu after yyparse()

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Now that environments are expanded in the lexer, conf_parse() does
> not need to explicitly expand $ARCH and $KERNELVERSION in the mainmenu.
>
> The hack introduced by commit 0724a7c32a54 ("kconfig: Don't leak
> main menus during parsing") can go away.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 07/21] kconfig: add function support and implement 'shell' function

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> This commit adds a new concept 'function' to do more text processing
> in Kconfig.
>
> A function call looks like this:
>
>   $(function arg1, arg2, arg3, ...)
>
> (Actually, this syntax was inspired by make.)
>
> Real examples will look like this:
>
>   $(shell echo hello world)
>   $(cc-option -fstackprotector)
>
> This commit adds the basic infrastructure to add, delete, evaluate
> functions, and also the first built-in function $(shell ...).  This
> accepts a single command to execute.  It returns the standard output
> from it.
>
> [Example code]
>
>   config HELLO
>           string
>           default "$(shell echo hello world)"
>
>   config Y
>           def_bool $(shell echo y)
>
> [Result]
>
>   $ make -s alldefconfig && tail -n 2 .config
>   CONFIG_HELLO="hello world"
>   CONFIG_Y=y
>
> Caveat:
> Like environments, functions are expanded in the lexer.  You cannot
> pass symbols to function arguments.  This is a limitation to simplify
> the implementation.  I want to avoid the dynamic function evaluation,
> which would introduce much more complexity.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Reminder for myself:
> Update Documentation/kbuild/kconfig-language.txt

Yeah, this needs to be included here, especially given the "cannot
pass symbols" aspect which might surprise people.

> [...]
> +/* built-in functions */
> +static char *do_shell(struct function *f, int argc, char *argv[])
> +{
> +       static const char *pre = "(";
> +       static const char *post = ") 2>/dev/null";

Right now the search and help screens in menuconfig just show the line
a config is defined and nothing more. I think it would be extremely
handy to include shell output here in some way. Especially when trying
to answer questions like "why aren't GCC plugins available?" it's got
quite a bit harder to debug.

Could we capture the output (especially stderr) for these kinds of hints?

Beyond that, looks good!

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 08/21] kconfig: replace $UNAME_RELEASE with function call

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Now that 'shell' function is supported, this can be self-contained in
> Kconfig.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 09/21] kconfig: add 'macro' keyword to support user-defined function

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Now, we got a basic ability to test compiler capability in Kconfig.
>
> config CC_HAS_STACKPROTECTOR
>         def_bool $(shell (($CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null) && echo y) || echo n)
>
> This works, but it is ugly to repeat this long boilerplate.
>
> We want to describe like this:
>
> config CC_HAS_STACKPROTECTOR
>         bool
>         default $(cc-option -fstack-protector)
>
> It is straight-forward to add a new function, but I do not like to
> hard-code specialized functions like this.  Hence, here is another
> feature to add functions from Kconfig files.
>
> A user-defined function is defined with a special keyword 'macro'.
> It can be referenced in the same way as built-in functions.  This
> feature was also inspired by Makefile where user-defined functions
> are referenced by $(call func-name, args...), but I omitted the 'call'
> to makes it shorter.
>
> The macro definition can contain $(1), $(2), ... which will be replaced
> with arguments from the caller.  The macro works just as a textual
> shorthand, which is also expanded in the lexer phase.
>
> [Example Code]
>
>   macro success $(shell ($(1) && echo y) || echo n)
>
>   config TRUE
>           bool "true"
>           default $(success true)
>
>   config FALSE
>           bool "false"
>           default $(success false)
>
> [Result]
>
>   $ make -s alldefconfig
>   $ tail -n 2 .config
>   CONFIG_TRUE=y
>   # CONFIG_FALSE is not set
>
> [Example Code]
>
>   macro success $(shell ($(1) && echo y) || echo n)
>
>   macro cc-option $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
>
>   config CC_HAS_STACKPROTECTOR
>           def_bool $(cc-option -fstack-protector)
>
> [Result]
>   $ make -s alldefconfig
>   $ tail -n 1 .config
>   CONFIG_CC_HAS_STACKPROTECTOR=y
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Cool, I like this "macro" idea for keeping this clean.

> ---
>
> Reminder for myself:
> Update Documentation/kbuild/kconfig-language.txt

Yes please. :)

Otherwise, looks good!

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 10/21] kconfig: add 'success' and 'cc-option' macros

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 'cc-option' will be the most frequently used macro.  It evaluates to 'y'
> if the given argument is supported by the compiler, or 'n' otherwise.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

> ---
>
> Changes in v2:
>   - Implement 'success' as a macro
>
>  init/Kconfig | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/init/Kconfig b/init/Kconfig
> index 0d6fe20..c456030 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -8,6 +8,12 @@ config DEFCONFIG_LIST
>         default ARCH_DEFCONFIG
>         default "arch/$ARCH/defconfig"
>
> +# expanded to y if the command exits with 0, n otherwise
> +macro success $(shell ($(1) && echo y) || echo n)
> +
> +# expanded to y if the given compiler flag is supported, n otherwise
> +macro cc-option $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)

This should be fine since most cc-option uses are quite simple. The
specialized per-arch and other-flag-dependency checks can just
continue to live in Makefile if we actually need them.

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 11/21] stack-protector: test compiler capability in Kconfig and drop AUTO mode

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Move the test for -fstack-protector(-strong) option to Kconfig.
>
> If the compiler does not support the option, the corresponding menu
> is automatically hidden.  If _STRONG is not supported, it will fall
> back to _REGULAR.  If _REGULAR is not supported, it will be disabled.
> This means, _AUTO is implicitly handled by the dependency solver of
> Kconfig, hence removed.
>
> I also turned the 'choice' into only two boolean symbols.  The use of
> 'choice' is not a good idea here, because all of all{yes,mod,no}config
> would choose the first visible value, while we want allnoconfig to
> disable as many features as possible.
>
> X86 has additional shell scripts in case the compiler supports the
> option, but generates broken code.  I added CC_HAS_SANE_STACKPROTECTOR
> to test this.  I had to add -m32 to gcc-x86_32-has-stack-protector.sh
> to make it work correctly.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

This looks really good. Notes below...

> ---
>
> Changes in v2:
>   - Describe $(cc-option ...) directly in depends on context
>
>  Makefile                                  | 93 ++-----------------------------
>  arch/Kconfig                              | 29 +++-------
>  arch/x86/Kconfig                          |  8 ++-
>  scripts/gcc-x86_32-has-stack-protector.sh |  7 +--
>  scripts/gcc-x86_64-has-stack-protector.sh |  5 --
>  5 files changed, 22 insertions(+), 120 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 5c395ed..5cadffa 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -675,55 +675,11 @@ ifneq ($(CONFIG_FRAME_WARN),0)
>  KBUILD_CFLAGS += $(call cc-option,-Wframe-larger-than=${CONFIG_FRAME_WARN})
>  endif
>
> -# This selects the stack protector compiler flag. Testing it is delayed
> -# until after .config has been reprocessed, in the prepare-compiler-check
> -# target.
> -ifdef CONFIG_CC_STACKPROTECTOR_AUTO
> -  stackp-flag := $(call cc-option,-fstack-protector-strong,$(call cc-option,-fstack-protector))
> -  stackp-name := AUTO
> -else
> -ifdef CONFIG_CC_STACKPROTECTOR_REGULAR
> -  stackp-flag := -fstack-protector
> -  stackp-name := REGULAR
> -else
> -ifdef CONFIG_CC_STACKPROTECTOR_STRONG
> -  stackp-flag := -fstack-protector-strong
> -  stackp-name := STRONG
> -else
> -  # If either there is no stack protector for this architecture or
> -  # CONFIG_CC_STACKPROTECTOR_NONE is selected, we're done, and $(stackp-name)
> -  # is empty, skipping all remaining stack protector tests.
> -  #
> -  # Force off for distro compilers that enable stack protector by default.
> -  KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
> -endif
> -endif
> -endif
> -# Find arch-specific stack protector compiler sanity-checking script.
> -ifdef stackp-name
> -ifneq ($(stackp-flag),)
> -  stackp-path := $(srctree)/scripts/gcc-$(SRCARCH)_$(BITS)-has-stack-protector.sh
> -  stackp-check := $(wildcard $(stackp-path))
> -  # If the wildcard test matches a test script, run it to check functionality.
> -  ifdef stackp-check
> -    ifneq ($(shell $(CONFIG_SHELL) $(stackp-check) $(CC) $(KBUILD_CPPFLAGS) $(biarch)),y)
> -      stackp-broken := y
> -    endif
> -  endif
> -  ifndef stackp-broken
> -    # If the stack protector is functional, enable code that depends on it.
> -    KBUILD_CPPFLAGS += -DCONFIG_CC_STACKPROTECTOR
> -    # Either we've already detected the flag (for AUTO) or we'll fail the
> -    # build in the prepare-compiler-check rule (for specific flag).
> -    KBUILD_CFLAGS += $(stackp-flag)
> -  else
> -    # We have to make sure stack protector is unconditionally disabled if
> -    # the compiler is broken (in case we're going to continue the build in
> -    # AUTO mode).

Let's keep this comment (slightly rewritten) since the reason for
setting this flag isn't obvious.

> -    KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
> -  endif
> -endif
> -endif
> +stackp-flags-y                                 := -fno-stack-protector

This is a (minor?) regression in my testing. Making this unconditional
may break for a compiler built without stack-protector. It should be
rare, but it's technically possible. Perhaps:

stackp-flags-y := ($call cc-option, -fno-stack-protector)

> +stackp-flags-$(CONFIG_CC_STACKPROTECTOR)       := -fstack-protector
> +stackp-flags-$(CONFIG_CC_STACKPROTECTOR_STRONG)        := -fstack-protector-strong
> +
> +KBUILD_CFLAGS += $(stackp-flags-y)
> [...]
> diff --git a/arch/Kconfig b/arch/Kconfig
> index 8e0d665..b42378d 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -535,13 +535,13 @@ config HAVE_CC_STACKPROTECTOR
>         bool
>         help
>           An arch should select this symbol if:
> -         - its compiler supports the -fstack-protector option

Please leave this note: it's still valid. An arch must still have
compiler support for this to be sensible.

>           - it has implemented a stack canary (e.g. __stack_chk_guard)
> [...]

Otherwise, this tests well for me. Nicely done!

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 13/21] kconfig: add CC_IS_GCC and GCC_VERSION

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> This will be useful to specify the required compiler version,
> like this:
>
> config FOO
>         bool "Use Foo"
>         depends on GCC_VERSION >= 408000
>         help
>           This feature requires GCC 4.8 or newer.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 14/21] kconfig: add CC_IS_CLANG and CLANG_VERSION

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> This will be useful to describe the clang version dependency.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

One aspect of clang version checking that may be "weird" (and likely
won't be meaningful for very long as we're about to raise the minimum
gcc version), is that clang claims to be gcc version 4.2 if you look
at #defines. This is already handled in compiler.h, but may be weird
in Kconfig.

Regardless:

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 15/21] gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> CONFIG_GCOV_FORMAT_AUTODETECT compiles either gcc_3_4.c or gcc_4_7.c
> according to your GCC version.
>
> We can achieve the equivalent behavior by setting reasonable dependency
> with the knowledge of the compiler version.
>
> If GCC older than 4.7 is used, GCOV_FORMAT_3_4 is the default, but users
> are still allowed to select GCOV_FORMAT_4_7 in case the newer format is
> back-ported.
>
> On the other hand, If GCC 4.7 or newer is used, there is no reason to
> use GCOV_FORMAT_3_4, so it should be hidden.
>
> If you down-grade your compiler GCC 4.7 or older, oldconfig/syncconfig
> will display a prompt for the choice because GCOV_FORMAT_3_4 becomes
> visible as a new symbol.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 16/21] kcov: imply GCC_PLUGINS and GCC_PLUGIN_SANCOV instead of select'ing them

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> As Documentation/kbuild/kconfig-language.txt notes, 'select' should be
> used with care - it forces a lower limit of another symbol, ignoring
> the dependency.  In this case, KCOV can select GCC_PLUGINS even if
> 'depends on HAVE_GCC_PLUGINS' is unmet.
>
> 'imply' is modest enough to observe the dependency, and this makes
> sense.  If you enable KCOV, you will probably want to enable
> GCC_PLUGIN_SANCOV, but it should not break the dependency.
>
> I also remove unneeded code, I just happened to notice.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 17/21] gcc-plugins: always build plugins with C++

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> If the target compiler is GCC 4.8 or newer, plugins are compiled with
> HOSTCXX.  Otherwise, gcc-plugin.sh will select HOSTCC or HOSTCXX.
>
> To simplify things, drop the HOSTCC support for plugins.  If you use
> GCC plugins, the requirement for the target compiler is GCC 4.8 (or
> older compiler with necessary features backported).  This is a feature
> for advanced users, so this requirement would not be a big deal.
> Currently, building plugins requires GCC 4.5, so this is not a big jump.

Emese already NAKed this, and I'd agree. 4.5 to 4.8 is quite a jump
(see all the threads on minimum gcc versions). Until we're at 4.8 as a
minimum, let's leave this as-is. "To simply things" doesn't seem like
a good enough reason to me.

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 18/21] gcc-plugins: move GCC version check for PowerPC to Kconfig

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> For PowerPC, GCC 5.2 is the requirement for GCC plugins.  Move the
> version check to Kconfig so that the GCC plugin menus will be hidden
> if an older compiler is in use.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
> ---
>
> Changes in v2: None
>
>  arch/powerpc/Kconfig         | 2 +-
>  scripts/Makefile.gcc-plugins | 8 --------
>  2 files changed, 1 insertion(+), 9 deletions(-)
>
> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> index 73ce5dd..b8474c2 100644
> --- a/arch/powerpc/Kconfig
> +++ b/arch/powerpc/Kconfig
> @@ -195,7 +195,7 @@ config PPC
>         select HAVE_FTRACE_MCOUNT_RECORD
>         select HAVE_FUNCTION_GRAPH_TRACER
>         select HAVE_FUNCTION_TRACER
> -       select HAVE_GCC_PLUGINS
> +       select HAVE_GCC_PLUGINS                 if GCC_VERSION >= 50200
>         select HAVE_GENERIC_GUP
>         select HAVE_HW_BREAKPOINT               if PERF_EVENTS && (PPC_BOOK3S || PPC_8xx)
>         select HAVE_IDE
> diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
> index b0f9108..25da4c0 100644
> --- a/scripts/Makefile.gcc-plugins
> +++ b/scripts/Makefile.gcc-plugins
> @@ -56,14 +56,6 @@ gcc-plugins-check: FORCE
>  ifdef CONFIG_GCC_PLUGINS
>    ifeq ($(PLUGINCC),)
>      ifneq ($(GCC_PLUGINS_CFLAGS),)
> -      # Various gccs between 4.5 and 5.1 have bugs on powerpc due to missing
> -      # header files. gcc <= 4.6 doesn't work at all, gccs from 4.8 to 5.1 have
> -      # issues with 64-bit targets.
> -      ifeq ($(ARCH),powerpc)
> -        ifeq ($(call cc-ifversion, -le, 0501, y), y)
> -         @echo "Cannot use CONFIG_GCC_PLUGINS: plugin support on gcc <= 5.1 is buggy on powerpc, please upgrade to gcc 5.2 or newer" >&2 && exit 1

I think we should retain the "buggy" note as a comment in Kconfig
since that knowledge isn't obvious just from the GCC_VERSION test.

> -        endif
> -      endif
>        $(Q)$(srctree)/scripts/gcc-plugin.sh --show-error $(HOSTCXX) $(CC) || true
>        @echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
>      endif
> --
> 2.7.4
>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 19/21] gcc-plugins: test GCC plugin support in Kconfig

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Run scripts/gcc-plugin.sh from Kconfig.  Users can enable GCC_PLUGINS
> only when it is supported.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Changes in v2: None
>
>  arch/Kconfig                 |  4 +++
>  scripts/Makefile.gcc-plugins | 82 ++++++++++++++++----------------------------
>  scripts/gcc-plugin.sh        |  1 -
>  3 files changed, 33 insertions(+), 54 deletions(-)
>
> diff --git a/arch/Kconfig b/arch/Kconfig
> index b42378d..88cc925 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -407,9 +407,13 @@ config HAVE_GCC_PLUGINS
>           An arch should select this symbol if it supports building with
>           GCC plugins.
>
> +config CC_HAS_GCC_PLUGINS
> +       bool
> +

This doesn't seem used anywhere?

>  menuconfig GCC_PLUGINS
>         bool "GCC plugins"
>         depends on HAVE_GCC_PLUGINS
> +       depends on $(success $srctree/scripts/gcc-plugin.sh $HOSTCXX $CC)
>         depends on !COMPILE_TEST
>         help
>           GCC plugins are loadable modules that provide extra features to the
> diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
> index 25da4c0..19d0d5b 100644
> --- a/scripts/Makefile.gcc-plugins
> +++ b/scripts/Makefile.gcc-plugins
> [...]
> -# If plugins aren't supported, abort the build before hard-to-read compiler
> -# errors start getting spewed by the main build.
> -PHONY += gcc-plugins-check
> -gcc-plugins-check: FORCE
> -ifdef CONFIG_GCC_PLUGINS
> -  ifeq ($(PLUGINCC),)
> -    ifneq ($(GCC_PLUGINS_CFLAGS),)
> -      $(Q)$(srctree)/scripts/gcc-plugin.sh --show-error $(HOSTCXX) $(CC) || true
> -      @echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1

As mentioned in the other email, we lose the error reporting. Now the
lack of plugins is just a silent =n in menuconfig. Keeping
--show-error in the Kconfig call and retaining stderr would be nice.

I need to do some further testing with SANCOV, but otherwise this all
looks correct, and my testing shows it behaving correctly.

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 20/21] gcc-plugins: enable GCC_PLUGINS for COMPILE_TEST

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> The plugin availability is checked in Kconfig, so all{yes,mod}config
> will not be bothered.  Remove 'depends on !COMPILE_TEST'.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Changes in v2:
>   - Remove more 'depends on'
>
>  arch/Kconfig | 4 ----
>  1 file changed, 4 deletions(-)
>
> diff --git a/arch/Kconfig b/arch/Kconfig
> index 88cc925..b07094c 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -414,7 +414,6 @@ menuconfig GCC_PLUGINS
>         bool "GCC plugins"
>         depends on HAVE_GCC_PLUGINS
>         depends on $(success $srctree/scripts/gcc-plugin.sh $HOSTCXX $CC)
> -       depends on !COMPILE_TEST
>         help
>           GCC plugins are loadable modules that provide extra features to the
>           compiler. They are useful for runtime instrumentation and static analysis.

This one is nice, yes! We get coverage for anyone with plugin support.

> @@ -424,7 +423,6 @@ menuconfig GCC_PLUGINS
>  config GCC_PLUGIN_CYC_COMPLEXITY
>         bool "Compute the cyclomatic complexity of a function" if EXPERT
>         depends on GCC_PLUGINS
> -       depends on !COMPILE_TEST
>         help
>           The complexity M of a function's control flow graph is defined as:
>            M = E - N + 2P
> @@ -487,7 +485,6 @@ config GCC_PLUGIN_STRUCTLEAK_BYREF_ALL
>  config GCC_PLUGIN_STRUCTLEAK_VERBOSE
>         bool "Report forcefully initialized variables"
>         depends on GCC_PLUGIN_STRUCTLEAK
> -       depends on !COMPILE_TEST
>         help
>           This option will cause a warning to be printed each time the
>           structleak plugin finds a variable it thinks needs to be
> @@ -527,7 +524,6 @@ config GCC_PLUGIN_RANDSTRUCT
>  config GCC_PLUGIN_RANDSTRUCT_PERFORMANCE
>         bool "Use cacheline-aware structure randomization"
>         depends on GCC_PLUGIN_RANDSTRUCT
> -       depends on !COMPILE_TEST
>         help
>           If you say Y here, the RANDSTRUCT randomization will make a
>           best effort at restricting randomization to cacheline-sized

All three of these, though, please leave as they were. They're either
needlessly noisy for COMPILE_TEST (GCC_PLUGIN_CYC_COMPLEXITY and
*_VERBOSE), or reduce the COMPILE_TEST coverage
(GCC_PLUGIN_RANDSTRUCT_PERFORMANCE).

(And perhaps a comment is needed to explain these to avoid confusion
in the future?)

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 14/21] kconfig: add CC_IS_CLANG and CLANG_VERSION

[Masahiro Yamada]

2018-03-28 20:22 GMT+09:00 Kees Cook <keescook@chromium.org>:
> On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> This will be useful to describe the clang version dependency.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>
> One aspect of clang version checking that may be "weird" (and likely
> won't be meaningful for very long as we're about to raise the minimum
> gcc version), is that clang claims to be gcc version 4.2 if you look
> at #defines. This is already handled in compiler.h, but may be weird
> in Kconfig.
>


CONFIG_CLANG_VERSION gives me real clang version.


$ clang --version
clang version 7.0.0 (http://llvm.org/git/clang.git
f1d1f4c4591e81cfa80034df094b26d7eb4fa92b)
(http://llvm.org/git/llvm.git
d157ba2f80790d23d717c4ff3f5bee30d98d6a51)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/masahiro/toolchains/clang-latest/bin

$ make CC=clang defconfig
*** Default configuration is based on 'x86_64_defconfig'
#
# configuration written to .config
#
$ grep  CLANG_VERSION .config
CONFIG_CLANG_VERSION=70000



I am just porting
https://patchwork.kernel.org/patch/10085765/
to Kconfig.


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 16/21] kcov: imply GCC_PLUGINS and GCC_PLUGIN_SANCOV instead of select'ing them

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> As Documentation/kbuild/kconfig-language.txt notes, 'select' should be
> used with care - it forces a lower limit of another symbol, ignoring
> the dependency.  In this case, KCOV can select GCC_PLUGINS even if
> 'depends on HAVE_GCC_PLUGINS' is unmet.
>
> 'imply' is modest enough to observe the dependency, and this makes
> sense.  If you enable KCOV, you will probably want to enable
> GCC_PLUGIN_SANCOV, but it should not break the dependency.
>
> I also remove unneeded code, I just happened to notice.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>   - Drop depends on GCC_VERSION
>
> ---
>
> Changes in v2: None
>
>  lib/Kconfig.debug | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
> index 5be22e4..8243c9d 100644
> --- a/lib/Kconfig.debug
> +++ b/lib/Kconfig.debug
> @@ -743,8 +743,8 @@ config KCOV
>         bool "Code coverage for fuzzing"
>         depends on ARCH_HAS_KCOV
>         select DEBUG_FS
> -       select GCC_PLUGINS if !COMPILE_TEST
> -       select GCC_PLUGIN_SANCOV if !COMPILE_TEST
> +       imply GCC_PLUGINS
> +       imply GCC_PLUGIN_SANCOV
>         help
>           KCOV exposes kernel code coverage information in a form suitable
>           for coverage-guided fuzzing (randomized testing).

Actually, I think this can be improved, as KCOV should depend on either:

-fsanitize-coverage=trace-pc support (in gcc 6+, see scripts/Makefile.kcov)

or

GCC_PLUGIN_SANCOV (which provides the trace-pc support via the plugin
for gcc before 6)

> @@ -758,7 +758,6 @@ config KCOV
>  config KCOV_ENABLE_COMPARISONS
>         bool "Enable comparison operands collection by KCOV"
>         depends on KCOV
> -       default n
>         help
>           KCOV also exposes operands of every comparison in the instrumented
>           code along with operand sizes and PCs of the comparison instructions.

Similarly, this depends on -fsanitize-coverage=trace-cmp

> @@ -768,7 +767,7 @@ config KCOV_ENABLE_COMPARISONS
>  config KCOV_INSTRUMENT_ALL
>         bool "Instrument all code by default"
>         depends on KCOV
> -       default y if KCOV
> +       default y
>         help
>           If you are doing generic system call fuzzing (like e.g. syzkaller),
>           then you will want to instrument the whole kernel and you should
> --
> 2.7.4
>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 21/21] arm64: move GCC version check for ARCH_SUPPORTS_INT128 to Kconfig

[Kees Cook]

On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> This becomes much neater in Kconfig.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Is there a reason this doesn't have a cc-option test, or was this a
silent addition to the compiler? (And as such, should there be a
comment for this in the Kconfig?)

Regardless:

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 04/21] kconfig: reference environments directly and remove 'option env=' syntax

[Ulf Magnusson]

I've been kinda busy lately, so that's why I disappeared.

I'll try to go over this patchset in more detail over the weekend.

On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> To get an environment value, Kconfig needs to define a symbol using
> "option env=" syntax.  It is tedious to add a config entry for each
> environment given that we need more environments such as 'CC', 'AS',
> 'srctree' etc. to evaluate the compiler capability in Kconfig.
>
> Adding '$' to symbols is weird.  Kconfig can reference symbols directly
> like this:
>
>   config FOO
>           string
>           default BAR
>
> So, I want to use the following syntax to get environment 'BAR' from
> the system:
>
>   config FOO
>           string
>           default $BAR
>
> Looking at the code, the symbols prefixed with 'S' are expanded by:
>  - conf_expand_value()
>    This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
>  - expand_string_value()
>    This is used to expand strings in 'source' and 'mainmenu'
>
> All of them are fixed values independent of user configuration.  So,
> this kind of syntax should be moved to simply take the environment.
>
> This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
> 'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.
>
> sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
> should be be given from the environment.
>
> ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
> by 'default ARCH_DEFCONFIG'.
>
> The environments are expanding in the lexer; when '$' is encountered,
> it is expanded, and resulted strings are pushed back to the input
> stream.  This makes the implementation simpler.
>
> For example, the following code works.
>
> [Example code]
>
>   config TOOLCHAIN_LIST
>           string
>           default "My tools: CC=$CC, AS=$AS, CPP=$CPP"
>
> [Result]
>
>   $ make -s alldefconfig && tail -n 1 .config
>   CONFIG_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> I tested all 'make *config' for arch architectures.
> I confirmed this commit still produced the same result
> (by my kconfig test tool).
>
>
> Changes in v2:
>   - Move the string expansion to the lexer phase.
>   - Split environment helpers to env.c
>
>  Documentation/kbuild/kconfig-language.txt |  8 ---
>  Kconfig                                   |  4 --
>  Makefile                                  |  3 +-
>  arch/sh/Kconfig                           |  4 +-
>  arch/sparc/Kconfig                        |  4 +-
>  arch/tile/Kconfig                         |  2 +-
>  arch/um/Kconfig.common                    |  4 --
>  arch/x86/Kconfig                          |  4 +-
>  arch/x86/um/Kconfig                       |  4 +-
>  init/Kconfig                              | 10 +---
>  scripts/kconfig/confdata.c                | 31 +---------
>  scripts/kconfig/env.c                     | 95 +++++++++++++++++++++++++++++++
>  scripts/kconfig/kconf_id.c                |  1 -
>  scripts/kconfig/lkc.h                     |  8 +--
>  scripts/kconfig/menu.c                    |  3 -
>  scripts/kconfig/symbol.c                  | 56 ------------------
>  scripts/kconfig/util.c                    | 75 ++++++++----------------
>  scripts/kconfig/zconf.l                   | 20 ++++++-
>  scripts/kconfig/zconf.y                   |  2 +-
>  19 files changed, 158 insertions(+), 180 deletions(-)
>  create mode 100644 scripts/kconfig/env.c
>
> diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
> index f5b9493..0e966e8 100644
> --- a/Documentation/kbuild/kconfig-language.txt
> +++ b/Documentation/kbuild/kconfig-language.txt
> @@ -198,14 +198,6 @@ applicable everywhere (see syntax).
>      enables the third modular state for all config symbols.
>      At most one symbol may have the "modules" option set.
>
> -  - "env"=<value>
> -    This imports the environment variable into Kconfig. It behaves like
> -    a default, except that the value comes from the environment, this
> -    also means that the behaviour when mixing it with normal defaults is
> -    undefined at this point. The symbol is currently not exported back
> -    to the build environment (if this is desired, it can be done via
> -    another symbol).
> -
>    - "allnoconfig_y"
>      This declares the symbol as one that should have the value y when
>      using "allnoconfig". Used for symbols that hide other symbols.
> diff --git a/Kconfig b/Kconfig
> index 8c4c1cb..e6ece5b 100644
> --- a/Kconfig
> +++ b/Kconfig
> @@ -5,8 +5,4 @@
>  #
>  mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
>
> -config SRCARCH
> -       string
> -       option env="SRCARCH"
> -
>  source "arch/$SRCARCH/Kconfig"
> diff --git a/Makefile b/Makefile
> index 5c395ed..4ae1486 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -284,7 +284,8 @@ include scripts/Kbuild.include
>  # Read KERNELRELEASE from include/config/kernel.release (if it exists)
>  KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
>  KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
> -export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
> +UNAME_RELEASE := $(shell uname --release)
> +export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
>
>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>  # first, and if a usermode build is happening, the "ARCH=um" on the command
> diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
> index 97fe293..14f3ef1 100644
> --- a/arch/sh/Kconfig
> +++ b/arch/sh/Kconfig
> @@ -57,7 +57,7 @@ config SUPERH
>           <http://www.linux-sh.org/>.
>
>  config SUPERH32
> -       def_bool ARCH = "sh"
> +       def_bool "$ARCH" = "sh"
>         select HAVE_KPROBES
>         select HAVE_KRETPROBES
>         select HAVE_IOREMAP_PROT if MMU && !X2TLB
> @@ -76,7 +76,7 @@ config SUPERH32
>         select HAVE_CC_STACKPROTECTOR
>
>  config SUPERH64
> -       def_bool ARCH = "sh64"
> +       def_bool "$ARCH" = "sh64"
>         select HAVE_EXIT_THREAD
>         select KALLSYMS
>
> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
> index 8767e45..86b852e 100644
> --- a/arch/sparc/Kconfig
> +++ b/arch/sparc/Kconfig
> @@ -1,6 +1,6 @@
>  config 64BIT
> -       bool "64-bit kernel" if ARCH = "sparc"
> -       default ARCH = "sparc64"
> +       bool "64-bit kernel" if "$ARCH" = "sparc"
> +       default "$ARCH" = "sparc64"
>         help
>           SPARC is a family of RISC microprocessors designed and marketed by
>           Sun Microsystems, incorporated.  They are very widely found in Sun
> diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
> index ef9d403..acc2182 100644
> --- a/arch/tile/Kconfig
> +++ b/arch/tile/Kconfig
> @@ -119,7 +119,7 @@ config HVC_TILE
>  # Building with ARCH=tilegx (or ARCH=tile) implies using the
>  # 64-bit TILE-Gx toolchain, so force CONFIG_TILEGX on.
>  config TILEGX
> -       def_bool ARCH != "tilepro"
> +       def_bool "$ARCH" != "tilepro"
>         select ARCH_SUPPORTS_ATOMIC_RMW
>         select GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
>         select HAVE_ARCH_JUMP_LABEL
> diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
> index c68add8..07f84c8 100644
> --- a/arch/um/Kconfig.common
> +++ b/arch/um/Kconfig.common
> @@ -54,10 +54,6 @@ config HZ
>         int
>         default 100
>
> -config SUBARCH
> -       string
> -       option env="SUBARCH"
> -
>  config NR_CPUS
>         int
>         range 1 1
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 0fa71a7..986fb0a 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1,8 +1,8 @@
>  # SPDX-License-Identifier: GPL-2.0
>  # Select 32 or 64 bit
>  config 64BIT
> -       bool "64-bit kernel" if ARCH = "x86"
> -       default ARCH != "i386"
> +       bool "64-bit kernel" if "$ARCH" = "x86"
> +       default "$ARCH" != "i386"
>         ---help---
>           Say yes to build a 64-bit kernel - formerly known as x86_64
>           Say no to build a 32-bit kernel - formerly known as i386
> diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
> index 13ed827..d355413 100644
> --- a/arch/x86/um/Kconfig
> +++ b/arch/x86/um/Kconfig
> @@ -16,8 +16,8 @@ config UML_X86
>         select GENERIC_FIND_FIRST_BIT
>
>  config 64BIT
> -       bool "64-bit kernel" if SUBARCH = "x86"
> -       default SUBARCH != "i386"
> +       bool "64-bit kernel" if $SUBARCH = "x86"
> +       default $SUBARCH != "i386"
>
>  config X86_32
>         def_bool !64BIT
> diff --git a/init/Kconfig b/init/Kconfig
> index df18492..b4814e6 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1,11 +1,3 @@
> -config ARCH
> -       string
> -       option env="ARCH"
> -
> -config KERNELVERSION
> -       string
> -       option env="KERNELVERSION"
> -
>  config DEFCONFIG_LIST
>         string
>         depends on !UML
> @@ -13,7 +5,7 @@ config DEFCONFIG_LIST
>         default "/lib/modules/$UNAME_RELEASE/.config"
>         default "/etc/kernel-config"
>         default "/boot/config-$UNAME_RELEASE"
> -       default "$ARCH_DEFCONFIG"
> +       default ARCH_DEFCONFIG
>         default "arch/$ARCH/defconfig"
>
>  config CONSTRUCTORS
> diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
> index df26c7b..98c2014 100644
> --- a/scripts/kconfig/confdata.c
> +++ b/scripts/kconfig/confdata.c
> @@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
>         return name ? name : "include/config/auto.conf";
>  }
>
> -static char *conf_expand_value(const char *in)
> -{
> -       struct symbol *sym;
> -       const char *src;
> -       static char res_value[SYMBOL_MAXLENGTH];
> -       char *dst, name[SYMBOL_MAXLENGTH];
> -
> -       res_value[0] = 0;
> -       dst = name;
> -       while ((src = strchr(in, '$'))) {
> -               strncat(res_value, in, src - in);
> -               src++;
> -               dst = name;
> -               while (isalnum(*src) || *src == '_')
> -                       *dst++ = *src++;
> -               *dst = 0;
> -               sym = sym_lookup(name, 0);
> -               sym_calc_value(sym);
> -               strcat(res_value, sym_get_string_value(sym));
> -               in = src;
> -       }
> -       strcat(res_value, in);
> -
> -       return res_value;
> -}
> -
>  char *conf_get_default_confname(void)
>  {
>         struct stat buf;
>         static char fullname[PATH_MAX+1];
>         char *env, *name;
>
> -       name = conf_expand_value(conf_defname);
> +       name = expand_string_value(conf_defname);
>         env = getenv(SRCTREE);
>         if (env) {
>                 sprintf(fullname, "%s/%s", env, name);
> @@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
>                         if (expr_calc_value(prop->visible.expr) == no ||
>                             prop->expr->type != E_SYMBOL)
>                                 continue;
> -                       name = conf_expand_value(prop->expr->left.sym->name);
> +                       sym_calc_value(prop->expr->left.sym);
> +                       name = sym_get_string_value(prop->expr->left.sym);
>                         in = zconf_fopen(name);
>                         if (in) {
>                                 conf_message(_("using defaults found in %s"),
> diff --git a/scripts/kconfig/env.c b/scripts/kconfig/env.c
> new file mode 100644
> index 0000000..9702f5c
> --- /dev/null
> +++ b/scripts/kconfig/env.c
> @@ -0,0 +1,95 @@
> +// SPDX-License-Identifier: GPL-2.0
> +//
> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
> +
> +static LIST_HEAD(env_list);
> +
> +struct env {
> +       char *name;
> +       char *value;
> +       struct list_head node;
> +};
> +
> +static struct env *env_list_lookup(const char *name)
> +{
> +       struct env *e;
> +
> +       list_for_each_entry(e, &env_list, node) {
> +               if (!strcmp(name, e->name))
> +                       return e;
> +       }
> +
> +       return NULL;
> +}
> +
> +static void env_list_add(const char *name, const char *value)
> +{
> +       struct env *e;
> +
> +       e = xmalloc(sizeof(*e));
> +       e->name = xstrdup(name);
> +       e->value = xstrdup(value);
> +
> +       list_add_tail(&e->node, &env_list);
> +}
> +
> +static void env_list_del(struct env *e)
> +{
> +       list_del(&e->node);
> +       free(e->name);
> +       free(e->value);
> +       free(e);
> +}
> +
> +/* the returned pointer must be freed when done */
> +static char *env_expand(const char *name)
> +{
> +       struct env *e;
> +       const char *value;
> +
> +       e = env_list_lookup(name);
> +       if (e)
> +               return xstrdup(e->value);
> +
> +       value = getenv(name);
> +       if (!value) {
> +               fprintf(stderr, "environment variable \"%s\" undefined\n", name);
> +               value = "";
> +       }
> +
> +       /*
> +        * we need to remember all referenced environments.
> +        * They will be written out to include/config/auto.conf.cmd
> +        */
> +       env_list_add(name, value);
> +
> +       return xstrdup(value);
> +}
> +
> +/* works like env_expand, but 'name' does not need to be null-terminated */
> +char *env_expand_n(const char *name, size_t n)
> +{
> +       char *tmp, *res;
> +
> +       tmp = xmalloc(n + 1);
> +       memcpy(tmp, name, n);
> +       *(tmp + n) = '\0';
> +
> +       res = env_expand(tmp);
> +
> +       free(tmp);
> +
> +       return res;
> +}
> +
> +void env_write_dep(FILE *f, const char *autoconfig_name)
> +{
> +       struct env *env, *tmp;
> +
> +       list_for_each_entry_safe(env, tmp, &env_list, node) {
> +               fprintf(f, "ifneq \"$(%s)\" \"%s\"\n", env->name, env->value);
> +               fprintf(f, "%s: FORCE\n", autoconfig_name);
> +               fprintf(f, "endif\n");
> +               env_list_del(env);
> +       }
> +}
> diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
> index 3ea9c5f..b3e0ea0 100644
> --- a/scripts/kconfig/kconf_id.c
> +++ b/scripts/kconfig/kconf_id.c
> @@ -32,7 +32,6 @@ static struct kconf_id kconf_id_array[] = {
>         { "on",                 T_ON,                   TF_PARAM },
>         { "modules",            T_OPT_MODULES,          TF_OPTION },
>         { "defconfig_list",     T_OPT_DEFCONFIG_LIST,   TF_OPTION },
> -       { "env",                T_OPT_ENV,              TF_OPTION },
>         { "allnoconfig_y",      T_OPT_ALLNOCONFIG_Y,    TF_OPTION },
>  };
>
> diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
> index c8d9e55..03d007f 100644
> --- a/scripts/kconfig/lkc.h
> +++ b/scripts/kconfig/lkc.h
> @@ -58,7 +58,6 @@ enum conf_def_mode {
>
>  #define T_OPT_MODULES          1
>  #define T_OPT_DEFCONFIG_LIST   2
> -#define T_OPT_ENV              3
>  #define T_OPT_ALLNOCONFIG_Y    4
>
>  struct kconf_id {
> @@ -95,6 +94,10 @@ static inline void xfwrite(const void *str, size_t len, size_t count, FILE *out)
>                 fprintf(stderr, "Error in writing or end of file.\n");
>  }
>
> +/* env.c */
> +char *env_expand_n(const char *name, size_t n);
> +void env_write_dep(FILE *f, const char *auto_conf_name);
> +
>  /* menu.c */
>  void _menu_init(void);
>  void menu_warn(struct menu *menu, const char *fmt, ...);
> @@ -135,9 +138,6 @@ void str_printf(struct gstr *gs, const char *fmt, ...);
>  const char *str_get(struct gstr *gs);
>
>  /* symbol.c */
> -extern struct expr *sym_env_list;
> -
> -void sym_init(void);
>  void sym_clear_all_valid(void);
>  struct symbol *sym_choice_default(struct symbol *sym);
>  const char *sym_get_string_default(struct symbol *sym);
> diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
> index 5c5c137..8148305 100644
> --- a/scripts/kconfig/menu.c
> +++ b/scripts/kconfig/menu.c
> @@ -214,9 +214,6 @@ void menu_add_option(int token, char *arg)
>                         zconf_error("trying to redefine defconfig symbol");
>                 sym_defconfig_list->flags |= SYMBOL_AUTO;
>                 break;
> -       case T_OPT_ENV:
> -               prop_add_env(arg);
> -               break;
>         case T_OPT_ALLNOCONFIG_Y:
>                 current_entry->sym->flags |= SYMBOL_ALLNOCONFIG_Y;
>                 break;
> diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
> index 03143b2..7c9a88e 100644
> --- a/scripts/kconfig/symbol.c
> +++ b/scripts/kconfig/symbol.c
> @@ -33,33 +33,6 @@ struct symbol *sym_defconfig_list;
>  struct symbol *modules_sym;
>  tristate modules_val;
>
> -struct expr *sym_env_list;
> -
> -static void sym_add_default(struct symbol *sym, const char *def)
> -{
> -       struct property *prop = prop_alloc(P_DEFAULT, sym);
> -
> -       prop->expr = expr_alloc_symbol(sym_lookup(def, SYMBOL_CONST));
> -}
> -
> -void sym_init(void)
> -{
> -       struct symbol *sym;
> -       struct utsname uts;
> -       static bool inited = false;
> -
> -       if (inited)
> -               return;
> -       inited = true;
> -
> -       uname(&uts);
> -
> -       sym = sym_lookup("UNAME_RELEASE", 0);
> -       sym->type = S_STRING;
> -       sym->flags |= SYMBOL_AUTO;
> -       sym_add_default(sym, uts.release);
> -}
> -
>  enum symbol_type sym_get_type(struct symbol *sym)
>  {
>         enum symbol_type type = sym->type;
> @@ -1348,32 +1321,3 @@ const char *prop_get_type_name(enum prop_type type)
>         }
>         return "unknown";
>  }
> -
> -static void prop_add_env(const char *env)
> -{
> -       struct symbol *sym, *sym2;
> -       struct property *prop;
> -       char *p;
> -
> -       sym = current_entry->sym;
> -       sym->flags |= SYMBOL_AUTO;
> -       for_all_properties(sym, prop, P_ENV) {
> -               sym2 = prop_get_symbol(prop);
> -               if (strcmp(sym2->name, env))
> -                       menu_warn(current_entry, "redefining environment symbol from %s",
> -                                 sym2->name);
> -               return;
> -       }
> -
> -       prop = prop_alloc(P_ENV, sym);
> -       prop->expr = expr_alloc_symbol(sym_lookup(env, SYMBOL_CONST));
> -
> -       sym_env_list = expr_alloc_one(E_LIST, sym_env_list);
> -       sym_env_list->right.sym = sym;
> -
> -       p = getenv(env);
> -       if (p)
> -               sym_add_default(sym, p);
> -       else
> -               menu_warn(current_entry, "environment variable %s undefined", env);
> -}
> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
> index 22201a4..136e497 100644
> --- a/scripts/kconfig/util.c
> +++ b/scripts/kconfig/util.c
> @@ -8,16 +8,18 @@
>  #include <stdarg.h>
>  #include <stdlib.h>
>  #include <string.h>
> +
> +#include "list.h"
>  #include "lkc.h"
>
>  /*
> - * Expand symbol's names embedded in the string given in argument. Symbols'
> - * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
> + * Expand environments embedded in the string given in argument. Environments
> + * to be expanded shall be prefixed by a '$'. Unknown environment expands to
>   * the empty string.
>   */
>  char *expand_string_value(const char *in)
>  {
> -       const char *src;
> +       const char *p, *q;
>         char *res;
>         size_t reslen;
>
> @@ -25,39 +27,28 @@ char *expand_string_value(const char *in)
>          * Note: 'in' might come from a token that's about to be
>          * freed, so make sure to always allocate a new string
>          */
> -       reslen = strlen(in) + 1;
> -       res = xmalloc(reslen);
> -       res[0] = '\0';
> -
> -       while ((src = strchr(in, '$'))) {
> -               char *p, name[SYMBOL_MAXLENGTH];
> -               const char *symval = "";
> -               struct symbol *sym;
> -               size_t newlen;
> -
> -               strncat(res, in, src - in);
> -               src++;
> -
> -               p = name;
> -               while (isalnum(*src) || *src == '_')
> -                       *p++ = *src++;
> -               *p = '\0';
> -
> -               sym = sym_find(name);
> -               if (sym != NULL) {
> -                       sym_calc_value(sym);
> -                       symval = sym_get_string_value(sym);
> -               }
> +       res = xmalloc(1);
> +       *res = '\0';
>
> -               newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
> -               if (newlen > reslen) {
> -                       reslen = newlen;
> -                       res = xrealloc(res, reslen);
> -               }
> +       while ((p = strchr(in, '$'))) {
> +               char *new;
> +
> +               q = p + 1;
> +               while (isalnum(*q) || *q == '_')
> +                       q++;
>
> -               strcat(res, symval);
> -               in = src;
> +               new = env_expand_n(p + 1, q - p - 1);
> +
> +               reslen = strlen(res) + (p - in) + strlen(new) + 1;
> +               res = xrealloc(res, reslen);
> +               strncat(res, in, p - in);
> +               strcat(res, new);
> +               free(new);
> +               in = q;
>         }
> +
> +       reslen = strlen(res) + strlen(in) + 1;
> +       res = xrealloc(res, reslen);
>         strcat(res, in);
>
>         return res;
> @@ -87,8 +78,6 @@ struct file *file_lookup(const char *name)
>  /* write a dependency file as used by kbuild to track dependencies */
>  int file_write_dep(const char *name)
>  {
> -       struct symbol *sym, *env_sym;
> -       struct expr *e;
>         struct file *file;
>         FILE *out;
>
> @@ -107,21 +96,7 @@ int file_write_dep(const char *name)
>         fprintf(out, "\n%s: \\\n"
>                      "\t$(deps_config)\n\n", conf_get_autoconfig_name());
>
> -       expr_list_for_each_sym(sym_env_list, e, sym) {
> -               struct property *prop;
> -               const char *value;
> -
> -               prop = sym_get_env_prop(sym);
> -               env_sym = prop_get_symbol(prop);
> -               if (!env_sym)
> -                       continue;
> -               value = getenv(env_sym->name);
> -               if (!value)
> -                       value = "";
> -               fprintf(out, "ifneq \"$(%s)\" \"%s\"\n", env_sym->name, value);
> -               fprintf(out, "%s: FORCE\n", conf_get_autoconfig_name());
> -               fprintf(out, "endif\n");
> -       }
> +       env_write_dep(out, conf_get_autoconfig_name());
>
>         fprintf(out, "\n$(deps_config): ;\n");
>         fclose(out);
> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
> index 045093d..551ca47 100644
> --- a/scripts/kconfig/zconf.l
> +++ b/scripts/kconfig/zconf.l
> @@ -35,6 +35,7 @@ struct buffer *current_buf;
>
>  static int last_ts, first_ts;
>
> +static void expand_string(const char *in);
>  static void zconf_endhelp(void);
>  static void zconf_endfile(void);
>
> @@ -120,6 +121,7 @@ n   [A-Za-z0-9_-]
>  }
>
>  <PARAM>{
> +       "$".*   expand_string(yytext);
>         "&&"    return T_AND;
>         "||"    return T_OR;
>         "("     return T_OPEN_PAREN;
> @@ -157,12 +159,13 @@ n [A-Za-z0-9_-]
>  }
>
>  <STRING>{
> -       [^'"\\\n]+/\n   {
> +       "$".*   expand_string(yytext);
> +       [^$'"\\\n]+/\n  {
>                 append_string(yytext, yyleng);
>                 yylval.string = text;
>                 return T_WORD_QUOTE;
>         }
> -       [^'"\\\n]+      {
> +       [^$'"\\\n]+     {
>                 append_string(yytext, yyleng);
>         }
>         \\.?/\n {
> @@ -249,6 +252,19 @@ n  [A-Za-z0-9_-]
>  }
>
>  %%
> +static void expand_string(const char *in)
> +{
> +       char *p, *q;
> +
> +       p = expand_string_value(in);
> +
> +       q = p + strlen(p);
> +       while (q > p)
> +               unput(*--q);
> +
> +       free(p);
> +}
> +

I like the simplicity of this approach, but I suspect it might be too simple.

For example, the following breaks with a syntax error if $ENV has any
double quotes in its value:

    config FOO
        string "foo"
        default "$ENV"

The following will only work as expected if $ENV expands to a valid
Kconfig symbol name. If it doesn't, random stuff will happen (most
likely a syntax error).

    config FOO
        string "foo"
        default $ENV

The reason it works if $ENV expands to a valid symbol name is that
undefined symbols get their name as their (string) value. If the
symbol happens to be defined, it will be referenced, which seems
confusing too.

In general, that reinterpretation of expanded values feels a bit icky
to me, and as something that might add complexity to Kconfig for
little value. If $ENV outside of quotes absolutely must be supported,
I think it should be a shorthand for "$ENV" (which means "constant
value" in Kconfig speak).

>  void zconf_starthelp(void)
>  {
>         new_string();
> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
> index 262c464..4ff4ac9 100644
> --- a/scripts/kconfig/zconf.y
> +++ b/scripts/kconfig/zconf.y
> @@ -534,7 +534,6 @@ void conf_parse(const char *name)
>
>         zconf_initscan(name);
>
> -       sym_init();
>         _menu_init();
>
>         if (getenv("ZCONF_DEBUG"))
> @@ -775,6 +774,7 @@ void zconfdump(FILE *out)
>  }
>
>  #include "zconf.lex.c"
> +#include "env.c"
>  #include "util.c"
>  #include "confdata.c"
>  #include "expr.c"
> --
> 2.7.4
>

Cheers,
Ulf

Re: [PATCH v2 07/21] kconfig: add function support and implement 'shell' function

[Ulf Magnusson]

On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> This commit adds a new concept 'function' to do more text processing
> in Kconfig.
>
> A function call looks like this:
>
>   $(function arg1, arg2, arg3, ...)
>
> (Actually, this syntax was inspired by make.)
>
> Real examples will look like this:
>
>   $(shell echo hello world)
>   $(cc-option -fstackprotector)
>
> This commit adds the basic infrastructure to add, delete, evaluate
> functions, and also the first built-in function $(shell ...).  This
> accepts a single command to execute.  It returns the standard output
> from it.
>
> [Example code]
>
>   config HELLO
>           string
>           default "$(shell echo hello world)"
>
>   config Y
>           def_bool $(shell echo y)
>
> [Result]
>
>   $ make -s alldefconfig && tail -n 2 .config
>   CONFIG_HELLO="hello world"
>   CONFIG_Y=y
>
> Caveat:
> Like environments, functions are expanded in the lexer.  You cannot
> pass symbols to function arguments.  This is a limitation to simplify
> the implementation.  I want to avoid the dynamic function evaluation,
> which would introduce much more complexity.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Reminder for myself:
> Update Documentation/kbuild/kconfig-language.txt
>
>
> Changes in v2:
>   - Use 'shell' for getting stdout from the comment.
>     It was 'shell-stdout' in the previous version.
>   - Symplify the implementation since the expansion has been moved to
>     lexer.
>
>  scripts/kconfig/function.c  | 170 ++++++++++++++++++++++++++++++++++++++++++++
>  scripts/kconfig/lkc_proto.h |   5 ++
>  scripts/kconfig/util.c      |  46 +++++++++---
>  scripts/kconfig/zconf.y     |   9 +++
>  4 files changed, 222 insertions(+), 8 deletions(-)
>  create mode 100644 scripts/kconfig/function.c
>

The gotcha from 04/21 ("kconfig: reference environments directly and
remove 'option env=' syntax") applies here too. For example, the
following will work:

    config A
        bool "A"
        default $(shell echo "B && C")

Some people might argue that that's a feature (I sense a mess down the
road if people start depending on it), but just in case you hadn't
thought of it.

Similarly, the following will only work as expected if 'cmd' outputs
the name of an undefined Kconfig symbol:

    config A
        string
        default $(shell cmd)

You could argue that people should add quotes there though (though
that's broken at the moment if the output from 'cmd' includes a
quote).

Quotes in Kconfig speak just mean "constant value". Problem is it's
undocumented, and no one would intuitively expect it.

Cheers,
Ulf

Re: [PATCH v2 04/21] kconfig: reference environments directly and remove 'option env=' syntax

[Ulf Magnusson]

On Thu, Mar 29, 2018 at 4:19 AM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
> I've been kinda busy lately, so that's why I disappeared.
>
> I'll try to go over this patchset in more detail over the weekend.
>
> On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> To get an environment value, Kconfig needs to define a symbol using
>> "option env=" syntax.  It is tedious to add a config entry for each
>> environment given that we need more environments such as 'CC', 'AS',
>> 'srctree' etc. to evaluate the compiler capability in Kconfig.
>>
>> Adding '$' to symbols is weird.  Kconfig can reference symbols directly
>> like this:
>>
>>   config FOO
>>           string
>>           default BAR
>>
>> So, I want to use the following syntax to get environment 'BAR' from
>> the system:
>>
>>   config FOO
>>           string
>>           default $BAR
>>
>> Looking at the code, the symbols prefixed with 'S' are expanded by:
>>  - conf_expand_value()
>>    This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
>>  - expand_string_value()
>>    This is used to expand strings in 'source' and 'mainmenu'
>>
>> All of them are fixed values independent of user configuration.  So,
>> this kind of syntax should be moved to simply take the environment.
>>
>> This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
>> 'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.
>>
>> sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
>> should be be given from the environment.
>>
>> ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
>> by 'default ARCH_DEFCONFIG'.
>>
>> The environments are expanding in the lexer; when '$' is encountered,
>> it is expanded, and resulted strings are pushed back to the input
>> stream.  This makes the implementation simpler.
>>
>> For example, the following code works.
>>
>> [Example code]
>>
>>   config TOOLCHAIN_LIST
>>           string
>>           default "My tools: CC=$CC, AS=$AS, CPP=$CPP"
>>
>> [Result]
>>
>>   $ make -s alldefconfig && tail -n 1 .config
>>   CONFIG_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> I tested all 'make *config' for arch architectures.
>> I confirmed this commit still produced the same result
>> (by my kconfig test tool).
>>
>>
>> Changes in v2:
>>   - Move the string expansion to the lexer phase.
>>   - Split environment helpers to env.c
>>
>>  Documentation/kbuild/kconfig-language.txt |  8 ---
>>  Kconfig                                   |  4 --
>>  Makefile                                  |  3 +-
>>  arch/sh/Kconfig                           |  4 +-
>>  arch/sparc/Kconfig                        |  4 +-
>>  arch/tile/Kconfig                         |  2 +-
>>  arch/um/Kconfig.common                    |  4 --
>>  arch/x86/Kconfig                          |  4 +-
>>  arch/x86/um/Kconfig                       |  4 +-
>>  init/Kconfig                              | 10 +---
>>  scripts/kconfig/confdata.c                | 31 +---------
>>  scripts/kconfig/env.c                     | 95 +++++++++++++++++++++++++++++++
>>  scripts/kconfig/kconf_id.c                |  1 -
>>  scripts/kconfig/lkc.h                     |  8 +--
>>  scripts/kconfig/menu.c                    |  3 -
>>  scripts/kconfig/symbol.c                  | 56 ------------------
>>  scripts/kconfig/util.c                    | 75 ++++++++----------------
>>  scripts/kconfig/zconf.l                   | 20 ++++++-
>>  scripts/kconfig/zconf.y                   |  2 +-
>>  19 files changed, 158 insertions(+), 180 deletions(-)
>>  create mode 100644 scripts/kconfig/env.c
>>
>> diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
>> index f5b9493..0e966e8 100644
>> --- a/Documentation/kbuild/kconfig-language.txt
>> +++ b/Documentation/kbuild/kconfig-language.txt
>> @@ -198,14 +198,6 @@ applicable everywhere (see syntax).
>>      enables the third modular state for all config symbols.
>>      At most one symbol may have the "modules" option set.
>>
>> -  - "env"=<value>
>> -    This imports the environment variable into Kconfig. It behaves like
>> -    a default, except that the value comes from the environment, this
>> -    also means that the behaviour when mixing it with normal defaults is
>> -    undefined at this point. The symbol is currently not exported back
>> -    to the build environment (if this is desired, it can be done via
>> -    another symbol).
>> -
>>    - "allnoconfig_y"
>>      This declares the symbol as one that should have the value y when
>>      using "allnoconfig". Used for symbols that hide other symbols.
>> diff --git a/Kconfig b/Kconfig
>> index 8c4c1cb..e6ece5b 100644
>> --- a/Kconfig
>> +++ b/Kconfig
>> @@ -5,8 +5,4 @@
>>  #
>>  mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
>>
>> -config SRCARCH
>> -       string
>> -       option env="SRCARCH"
>> -
>>  source "arch/$SRCARCH/Kconfig"
>> diff --git a/Makefile b/Makefile
>> index 5c395ed..4ae1486 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -284,7 +284,8 @@ include scripts/Kbuild.include
>>  # Read KERNELRELEASE from include/config/kernel.release (if it exists)
>>  KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
>>  KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
>> -export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
>> +UNAME_RELEASE := $(shell uname --release)
>> +export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
>>
>>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>>  # first, and if a usermode build is happening, the "ARCH=um" on the command
>> diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
>> index 97fe293..14f3ef1 100644
>> --- a/arch/sh/Kconfig
>> +++ b/arch/sh/Kconfig
>> @@ -57,7 +57,7 @@ config SUPERH
>>           <http://www.linux-sh.org/>.
>>
>>  config SUPERH32
>> -       def_bool ARCH = "sh"
>> +       def_bool "$ARCH" = "sh"
>>         select HAVE_KPROBES
>>         select HAVE_KRETPROBES
>>         select HAVE_IOREMAP_PROT if MMU && !X2TLB
>> @@ -76,7 +76,7 @@ config SUPERH32
>>         select HAVE_CC_STACKPROTECTOR
>>
>>  config SUPERH64
>> -       def_bool ARCH = "sh64"
>> +       def_bool "$ARCH" = "sh64"
>>         select HAVE_EXIT_THREAD
>>         select KALLSYMS
>>
>> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
>> index 8767e45..86b852e 100644
>> --- a/arch/sparc/Kconfig
>> +++ b/arch/sparc/Kconfig
>> @@ -1,6 +1,6 @@
>>  config 64BIT
>> -       bool "64-bit kernel" if ARCH = "sparc"
>> -       default ARCH = "sparc64"
>> +       bool "64-bit kernel" if "$ARCH" = "sparc"
>> +       default "$ARCH" = "sparc64"
>>         help
>>           SPARC is a family of RISC microprocessors designed and marketed by
>>           Sun Microsystems, incorporated.  They are very widely found in Sun
>> diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
>> index ef9d403..acc2182 100644
>> --- a/arch/tile/Kconfig
>> +++ b/arch/tile/Kconfig
>> @@ -119,7 +119,7 @@ config HVC_TILE
>>  # Building with ARCH=tilegx (or ARCH=tile) implies using the
>>  # 64-bit TILE-Gx toolchain, so force CONFIG_TILEGX on.
>>  config TILEGX
>> -       def_bool ARCH != "tilepro"
>> +       def_bool "$ARCH" != "tilepro"
>>         select ARCH_SUPPORTS_ATOMIC_RMW
>>         select GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
>>         select HAVE_ARCH_JUMP_LABEL
>> diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
>> index c68add8..07f84c8 100644
>> --- a/arch/um/Kconfig.common
>> +++ b/arch/um/Kconfig.common
>> @@ -54,10 +54,6 @@ config HZ
>>         int
>>         default 100
>>
>> -config SUBARCH
>> -       string
>> -       option env="SUBARCH"
>> -
>>  config NR_CPUS
>>         int
>>         range 1 1
>> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>> index 0fa71a7..986fb0a 100644
>> --- a/arch/x86/Kconfig
>> +++ b/arch/x86/Kconfig
>> @@ -1,8 +1,8 @@
>>  # SPDX-License-Identifier: GPL-2.0
>>  # Select 32 or 64 bit
>>  config 64BIT
>> -       bool "64-bit kernel" if ARCH = "x86"
>> -       default ARCH != "i386"
>> +       bool "64-bit kernel" if "$ARCH" = "x86"
>> +       default "$ARCH" != "i386"
>>         ---help---
>>           Say yes to build a 64-bit kernel - formerly known as x86_64
>>           Say no to build a 32-bit kernel - formerly known as i386
>> diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
>> index 13ed827..d355413 100644
>> --- a/arch/x86/um/Kconfig
>> +++ b/arch/x86/um/Kconfig
>> @@ -16,8 +16,8 @@ config UML_X86
>>         select GENERIC_FIND_FIRST_BIT
>>
>>  config 64BIT
>> -       bool "64-bit kernel" if SUBARCH = "x86"
>> -       default SUBARCH != "i386"
>> +       bool "64-bit kernel" if $SUBARCH = "x86"
>> +       default $SUBARCH != "i386"
>>
>>  config X86_32
>>         def_bool !64BIT
>> diff --git a/init/Kconfig b/init/Kconfig
>> index df18492..b4814e6 100644
>> --- a/init/Kconfig
>> +++ b/init/Kconfig
>> @@ -1,11 +1,3 @@
>> -config ARCH
>> -       string
>> -       option env="ARCH"
>> -
>> -config KERNELVERSION
>> -       string
>> -       option env="KERNELVERSION"
>> -
>>  config DEFCONFIG_LIST
>>         string
>>         depends on !UML
>> @@ -13,7 +5,7 @@ config DEFCONFIG_LIST
>>         default "/lib/modules/$UNAME_RELEASE/.config"
>>         default "/etc/kernel-config"
>>         default "/boot/config-$UNAME_RELEASE"
>> -       default "$ARCH_DEFCONFIG"
>> +       default ARCH_DEFCONFIG
>>         default "arch/$ARCH/defconfig"
>>
>>  config CONSTRUCTORS
>> diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
>> index df26c7b..98c2014 100644
>> --- a/scripts/kconfig/confdata.c
>> +++ b/scripts/kconfig/confdata.c
>> @@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
>>         return name ? name : "include/config/auto.conf";
>>  }
>>
>> -static char *conf_expand_value(const char *in)
>> -{
>> -       struct symbol *sym;
>> -       const char *src;
>> -       static char res_value[SYMBOL_MAXLENGTH];
>> -       char *dst, name[SYMBOL_MAXLENGTH];
>> -
>> -       res_value[0] = 0;
>> -       dst = name;
>> -       while ((src = strchr(in, '$'))) {
>> -               strncat(res_value, in, src - in);
>> -               src++;
>> -               dst = name;
>> -               while (isalnum(*src) || *src == '_')
>> -                       *dst++ = *src++;
>> -               *dst = 0;
>> -               sym = sym_lookup(name, 0);
>> -               sym_calc_value(sym);
>> -               strcat(res_value, sym_get_string_value(sym));
>> -               in = src;
>> -       }
>> -       strcat(res_value, in);
>> -
>> -       return res_value;
>> -}
>> -
>>  char *conf_get_default_confname(void)
>>  {
>>         struct stat buf;
>>         static char fullname[PATH_MAX+1];
>>         char *env, *name;
>>
>> -       name = conf_expand_value(conf_defname);
>> +       name = expand_string_value(conf_defname);
>>         env = getenv(SRCTREE);
>>         if (env) {
>>                 sprintf(fullname, "%s/%s", env, name);
>> @@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
>>                         if (expr_calc_value(prop->visible.expr) == no ||
>>                             prop->expr->type != E_SYMBOL)
>>                                 continue;
>> -                       name = conf_expand_value(prop->expr->left.sym->name);
>> +                       sym_calc_value(prop->expr->left.sym);
>> +                       name = sym_get_string_value(prop->expr->left.sym);
>>                         in = zconf_fopen(name);
>>                         if (in) {
>>                                 conf_message(_("using defaults found in %s"),
>> diff --git a/scripts/kconfig/env.c b/scripts/kconfig/env.c
>> new file mode 100644
>> index 0000000..9702f5c
>> --- /dev/null
>> +++ b/scripts/kconfig/env.c
>> @@ -0,0 +1,95 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +//
>> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
>> +
>> +static LIST_HEAD(env_list);
>> +
>> +struct env {
>> +       char *name;
>> +       char *value;
>> +       struct list_head node;
>> +};
>> +
>> +static struct env *env_list_lookup(const char *name)
>> +{
>> +       struct env *e;
>> +
>> +       list_for_each_entry(e, &env_list, node) {
>> +               if (!strcmp(name, e->name))
>> +                       return e;
>> +       }
>> +
>> +       return NULL;
>> +}
>> +
>> +static void env_list_add(const char *name, const char *value)
>> +{
>> +       struct env *e;
>> +
>> +       e = xmalloc(sizeof(*e));
>> +       e->name = xstrdup(name);
>> +       e->value = xstrdup(value);
>> +
>> +       list_add_tail(&e->node, &env_list);
>> +}
>> +
>> +static void env_list_del(struct env *e)
>> +{
>> +       list_del(&e->node);
>> +       free(e->name);
>> +       free(e->value);
>> +       free(e);
>> +}
>> +
>> +/* the returned pointer must be freed when done */
>> +static char *env_expand(const char *name)
>> +{
>> +       struct env *e;
>> +       const char *value;
>> +
>> +       e = env_list_lookup(name);
>> +       if (e)
>> +               return xstrdup(e->value);
>> +
>> +       value = getenv(name);
>> +       if (!value) {
>> +               fprintf(stderr, "environment variable \"%s\" undefined\n", name);
>> +               value = "";
>> +       }
>> +
>> +       /*
>> +        * we need to remember all referenced environments.
>> +        * They will be written out to include/config/auto.conf.cmd
>> +        */
>> +       env_list_add(name, value);
>> +
>> +       return xstrdup(value);
>> +}
>> +
>> +/* works like env_expand, but 'name' does not need to be null-terminated */
>> +char *env_expand_n(const char *name, size_t n)
>> +{
>> +       char *tmp, *res;
>> +
>> +       tmp = xmalloc(n + 1);
>> +       memcpy(tmp, name, n);
>> +       *(tmp + n) = '\0';
>> +
>> +       res = env_expand(tmp);
>> +
>> +       free(tmp);
>> +
>> +       return res;
>> +}
>> +
>> +void env_write_dep(FILE *f, const char *autoconfig_name)
>> +{
>> +       struct env *env, *tmp;
>> +
>> +       list_for_each_entry_safe(env, tmp, &env_list, node) {
>> +               fprintf(f, "ifneq \"$(%s)\" \"%s\"\n", env->name, env->value);
>> +               fprintf(f, "%s: FORCE\n", autoconfig_name);
>> +               fprintf(f, "endif\n");
>> +               env_list_del(env);
>> +       }
>> +}
>> diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
>> index 3ea9c5f..b3e0ea0 100644
>> --- a/scripts/kconfig/kconf_id.c
>> +++ b/scripts/kconfig/kconf_id.c
>> @@ -32,7 +32,6 @@ static struct kconf_id kconf_id_array[] = {
>>         { "on",                 T_ON,                   TF_PARAM },
>>         { "modules",            T_OPT_MODULES,          TF_OPTION },
>>         { "defconfig_list",     T_OPT_DEFCONFIG_LIST,   TF_OPTION },
>> -       { "env",                T_OPT_ENV,              TF_OPTION },
>>         { "allnoconfig_y",      T_OPT_ALLNOCONFIG_Y,    TF_OPTION },
>>  };
>>
>> diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
>> index c8d9e55..03d007f 100644
>> --- a/scripts/kconfig/lkc.h
>> +++ b/scripts/kconfig/lkc.h
>> @@ -58,7 +58,6 @@ enum conf_def_mode {
>>
>>  #define T_OPT_MODULES          1
>>  #define T_OPT_DEFCONFIG_LIST   2
>> -#define T_OPT_ENV              3
>>  #define T_OPT_ALLNOCONFIG_Y    4
>>
>>  struct kconf_id {
>> @@ -95,6 +94,10 @@ static inline void xfwrite(const void *str, size_t len, size_t count, FILE *out)
>>                 fprintf(stderr, "Error in writing or end of file.\n");
>>  }
>>
>> +/* env.c */
>> +char *env_expand_n(const char *name, size_t n);
>> +void env_write_dep(FILE *f, const char *auto_conf_name);
>> +
>>  /* menu.c */
>>  void _menu_init(void);
>>  void menu_warn(struct menu *menu, const char *fmt, ...);
>> @@ -135,9 +138,6 @@ void str_printf(struct gstr *gs, const char *fmt, ...);
>>  const char *str_get(struct gstr *gs);
>>
>>  /* symbol.c */
>> -extern struct expr *sym_env_list;
>> -
>> -void sym_init(void);
>>  void sym_clear_all_valid(void);
>>  struct symbol *sym_choice_default(struct symbol *sym);
>>  const char *sym_get_string_default(struct symbol *sym);
>> diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
>> index 5c5c137..8148305 100644
>> --- a/scripts/kconfig/menu.c
>> +++ b/scripts/kconfig/menu.c
>> @@ -214,9 +214,6 @@ void menu_add_option(int token, char *arg)
>>                         zconf_error("trying to redefine defconfig symbol");
>>                 sym_defconfig_list->flags |= SYMBOL_AUTO;
>>                 break;
>> -       case T_OPT_ENV:
>> -               prop_add_env(arg);
>> -               break;
>>         case T_OPT_ALLNOCONFIG_Y:
>>                 current_entry->sym->flags |= SYMBOL_ALLNOCONFIG_Y;
>>                 break;
>> diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
>> index 03143b2..7c9a88e 100644
>> --- a/scripts/kconfig/symbol.c
>> +++ b/scripts/kconfig/symbol.c
>> @@ -33,33 +33,6 @@ struct symbol *sym_defconfig_list;
>>  struct symbol *modules_sym;
>>  tristate modules_val;
>>
>> -struct expr *sym_env_list;
>> -
>> -static void sym_add_default(struct symbol *sym, const char *def)
>> -{
>> -       struct property *prop = prop_alloc(P_DEFAULT, sym);
>> -
>> -       prop->expr = expr_alloc_symbol(sym_lookup(def, SYMBOL_CONST));
>> -}
>> -
>> -void sym_init(void)
>> -{
>> -       struct symbol *sym;
>> -       struct utsname uts;
>> -       static bool inited = false;
>> -
>> -       if (inited)
>> -               return;
>> -       inited = true;
>> -
>> -       uname(&uts);
>> -
>> -       sym = sym_lookup("UNAME_RELEASE", 0);
>> -       sym->type = S_STRING;
>> -       sym->flags |= SYMBOL_AUTO;
>> -       sym_add_default(sym, uts.release);
>> -}
>> -
>>  enum symbol_type sym_get_type(struct symbol *sym)
>>  {
>>         enum symbol_type type = sym->type;
>> @@ -1348,32 +1321,3 @@ const char *prop_get_type_name(enum prop_type type)
>>         }
>>         return "unknown";
>>  }
>> -
>> -static void prop_add_env(const char *env)
>> -{
>> -       struct symbol *sym, *sym2;
>> -       struct property *prop;
>> -       char *p;
>> -
>> -       sym = current_entry->sym;
>> -       sym->flags |= SYMBOL_AUTO;
>> -       for_all_properties(sym, prop, P_ENV) {
>> -               sym2 = prop_get_symbol(prop);
>> -               if (strcmp(sym2->name, env))
>> -                       menu_warn(current_entry, "redefining environment symbol from %s",
>> -                                 sym2->name);
>> -               return;
>> -       }
>> -
>> -       prop = prop_alloc(P_ENV, sym);
>> -       prop->expr = expr_alloc_symbol(sym_lookup(env, SYMBOL_CONST));
>> -
>> -       sym_env_list = expr_alloc_one(E_LIST, sym_env_list);
>> -       sym_env_list->right.sym = sym;
>> -
>> -       p = getenv(env);
>> -       if (p)
>> -               sym_add_default(sym, p);
>> -       else
>> -               menu_warn(current_entry, "environment variable %s undefined", env);
>> -}
>> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
>> index 22201a4..136e497 100644
>> --- a/scripts/kconfig/util.c
>> +++ b/scripts/kconfig/util.c
>> @@ -8,16 +8,18 @@
>>  #include <stdarg.h>
>>  #include <stdlib.h>
>>  #include <string.h>
>> +
>> +#include "list.h"
>>  #include "lkc.h"
>>
>>  /*
>> - * Expand symbol's names embedded in the string given in argument. Symbols'
>> - * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
>> + * Expand environments embedded in the string given in argument. Environments
>> + * to be expanded shall be prefixed by a '$'. Unknown environment expands to
>>   * the empty string.
>>   */
>>  char *expand_string_value(const char *in)
>>  {
>> -       const char *src;
>> +       const char *p, *q;
>>         char *res;
>>         size_t reslen;
>>
>> @@ -25,39 +27,28 @@ char *expand_string_value(const char *in)
>>          * Note: 'in' might come from a token that's about to be
>>          * freed, so make sure to always allocate a new string
>>          */
>> -       reslen = strlen(in) + 1;
>> -       res = xmalloc(reslen);
>> -       res[0] = '\0';
>> -
>> -       while ((src = strchr(in, '$'))) {
>> -               char *p, name[SYMBOL_MAXLENGTH];
>> -               const char *symval = "";
>> -               struct symbol *sym;
>> -               size_t newlen;
>> -
>> -               strncat(res, in, src - in);
>> -               src++;
>> -
>> -               p = name;
>> -               while (isalnum(*src) || *src == '_')
>> -                       *p++ = *src++;
>> -               *p = '\0';
>> -
>> -               sym = sym_find(name);
>> -               if (sym != NULL) {
>> -                       sym_calc_value(sym);
>> -                       symval = sym_get_string_value(sym);
>> -               }
>> +       res = xmalloc(1);
>> +       *res = '\0';
>>
>> -               newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
>> -               if (newlen > reslen) {
>> -                       reslen = newlen;
>> -                       res = xrealloc(res, reslen);
>> -               }
>> +       while ((p = strchr(in, '$'))) {
>> +               char *new;
>> +
>> +               q = p + 1;
>> +               while (isalnum(*q) || *q == '_')
>> +                       q++;
>>
>> -               strcat(res, symval);
>> -               in = src;
>> +               new = env_expand_n(p + 1, q - p - 1);
>> +
>> +               reslen = strlen(res) + (p - in) + strlen(new) + 1;
>> +               res = xrealloc(res, reslen);
>> +               strncat(res, in, p - in);
>> +               strcat(res, new);
>> +               free(new);
>> +               in = q;
>>         }
>> +
>> +       reslen = strlen(res) + strlen(in) + 1;
>> +       res = xrealloc(res, reslen);
>>         strcat(res, in);
>>
>>         return res;
>> @@ -87,8 +78,6 @@ struct file *file_lookup(const char *name)
>>  /* write a dependency file as used by kbuild to track dependencies */
>>  int file_write_dep(const char *name)
>>  {
>> -       struct symbol *sym, *env_sym;
>> -       struct expr *e;
>>         struct file *file;
>>         FILE *out;
>>
>> @@ -107,21 +96,7 @@ int file_write_dep(const char *name)
>>         fprintf(out, "\n%s: \\\n"
>>                      "\t$(deps_config)\n\n", conf_get_autoconfig_name());
>>
>> -       expr_list_for_each_sym(sym_env_list, e, sym) {
>> -               struct property *prop;
>> -               const char *value;
>> -
>> -               prop = sym_get_env_prop(sym);
>> -               env_sym = prop_get_symbol(prop);
>> -               if (!env_sym)
>> -                       continue;
>> -               value = getenv(env_sym->name);
>> -               if (!value)
>> -                       value = "";
>> -               fprintf(out, "ifneq \"$(%s)\" \"%s\"\n", env_sym->name, value);
>> -               fprintf(out, "%s: FORCE\n", conf_get_autoconfig_name());
>> -               fprintf(out, "endif\n");
>> -       }
>> +       env_write_dep(out, conf_get_autoconfig_name());
>>
>>         fprintf(out, "\n$(deps_config): ;\n");
>>         fclose(out);
>> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
>> index 045093d..551ca47 100644
>> --- a/scripts/kconfig/zconf.l
>> +++ b/scripts/kconfig/zconf.l
>> @@ -35,6 +35,7 @@ struct buffer *current_buf;
>>
>>  static int last_ts, first_ts;
>>
>> +static void expand_string(const char *in);
>>  static void zconf_endhelp(void);
>>  static void zconf_endfile(void);
>>
>> @@ -120,6 +121,7 @@ n   [A-Za-z0-9_-]
>>  }
>>
>>  <PARAM>{
>> +       "$".*   expand_string(yytext);
>>         "&&"    return T_AND;
>>         "||"    return T_OR;
>>         "("     return T_OPEN_PAREN;
>> @@ -157,12 +159,13 @@ n [A-Za-z0-9_-]
>>  }
>>
>>  <STRING>{
>> -       [^'"\\\n]+/\n   {
>> +       "$".*   expand_string(yytext);
>> +       [^$'"\\\n]+/\n  {
>>                 append_string(yytext, yyleng);
>>                 yylval.string = text;
>>                 return T_WORD_QUOTE;
>>         }
>> -       [^'"\\\n]+      {
>> +       [^$'"\\\n]+     {
>>                 append_string(yytext, yyleng);
>>         }
>>         \\.?/\n {
>> @@ -249,6 +252,19 @@ n  [A-Za-z0-9_-]
>>  }
>>
>>  %%
>> +static void expand_string(const char *in)
>> +{
>> +       char *p, *q;
>> +
>> +       p = expand_string_value(in);
>> +
>> +       q = p + strlen(p);
>> +       while (q > p)
>> +               unput(*--q);
>> +
>> +       free(p);
>> +}
>> +
>
> I like the simplicity of this approach, but I suspect it might be too simple.
>
> For example, the following breaks with a syntax error if $ENV has any
> double quotes in its value:
>
>     config FOO
>         string "foo"
>         default "$ENV"
>
> The following will only work as expected if $ENV expands to a valid
> Kconfig symbol name. If it doesn't, random stuff will happen (most
> likely a syntax error).
>
>     config FOO
>         string "foo"
>         default $ENV
>
> The reason it works if $ENV expands to a valid symbol name is that
> undefined symbols get their name as their (string) value. If the
> symbol happens to be defined, it will be referenced, which seems
> confusing too.
>
> In general, that reinterpretation of expanded values feels a bit icky
> to me, and as something that might add complexity to Kconfig for
> little value. If $ENV outside of quotes absolutely must be supported,
> I think it should be a shorthand for "$ENV" (which means "constant
> value" in Kconfig speak).

If you want something as general as the C preprocessor (which I think
would be overkill and complexity land), then it seems kinda weird to
limit it to certain Kconfig contexts as well: Right now you'd be able
to output arbitrary tokens inside an expression, but you couldn't do
something like generate a 'default X if Y'.

Cheers,
Ulf

Re: [PATCH v2 04/21] kconfig: reference environments directly and remove 'option env=' syntax

[Ulf Magnusson]

On Thu, Mar 29, 2018 at 4:56 AM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
> On Thu, Mar 29, 2018 at 4:19 AM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
>> I've been kinda busy lately, so that's why I disappeared.
>>
>> I'll try to go over this patchset in more detail over the weekend.
>>
>> On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
>> <yamada.masahiro@socionext.com> wrote:
>>> To get an environment value, Kconfig needs to define a symbol using
>>> "option env=" syntax.  It is tedious to add a config entry for each
>>> environment given that we need more environments such as 'CC', 'AS',
>>> 'srctree' etc. to evaluate the compiler capability in Kconfig.
>>>
>>> Adding '$' to symbols is weird.  Kconfig can reference symbols directly
>>> like this:
>>>
>>>   config FOO
>>>           string
>>>           default BAR
>>>
>>> So, I want to use the following syntax to get environment 'BAR' from
>>> the system:
>>>
>>>   config FOO
>>>           string
>>>           default $BAR
>>>
>>> Looking at the code, the symbols prefixed with 'S' are expanded by:
>>>  - conf_expand_value()
>>>    This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
>>>  - expand_string_value()
>>>    This is used to expand strings in 'source' and 'mainmenu'
>>>
>>> All of them are fixed values independent of user configuration.  So,
>>> this kind of syntax should be moved to simply take the environment.
>>>
>>> This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
>>> 'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.
>>>
>>> sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
>>> should be be given from the environment.
>>>
>>> ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
>>> by 'default ARCH_DEFCONFIG'.
>>>
>>> The environments are expanding in the lexer; when '$' is encountered,
>>> it is expanded, and resulted strings are pushed back to the input
>>> stream.  This makes the implementation simpler.
>>>
>>> For example, the following code works.
>>>
>>> [Example code]
>>>
>>>   config TOOLCHAIN_LIST
>>>           string
>>>           default "My tools: CC=$CC, AS=$AS, CPP=$CPP"
>>>
>>> [Result]
>>>
>>>   $ make -s alldefconfig && tail -n 1 .config
>>>   CONFIG_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"
>>>
>>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>>> ---
>>>
>>> I tested all 'make *config' for arch architectures.
>>> I confirmed this commit still produced the same result
>>> (by my kconfig test tool).
>>>
>>>
>>> Changes in v2:
>>>   - Move the string expansion to the lexer phase.
>>>   - Split environment helpers to env.c
>>>
>>>  Documentation/kbuild/kconfig-language.txt |  8 ---
>>>  Kconfig                                   |  4 --
>>>  Makefile                                  |  3 +-
>>>  arch/sh/Kconfig                           |  4 +-
>>>  arch/sparc/Kconfig                        |  4 +-
>>>  arch/tile/Kconfig                         |  2 +-
>>>  arch/um/Kconfig.common                    |  4 --
>>>  arch/x86/Kconfig                          |  4 +-
>>>  arch/x86/um/Kconfig                       |  4 +-
>>>  init/Kconfig                              | 10 +---
>>>  scripts/kconfig/confdata.c                | 31 +---------
>>>  scripts/kconfig/env.c                     | 95 +++++++++++++++++++++++++++++++
>>>  scripts/kconfig/kconf_id.c                |  1 -
>>>  scripts/kconfig/lkc.h                     |  8 +--
>>>  scripts/kconfig/menu.c                    |  3 -
>>>  scripts/kconfig/symbol.c                  | 56 ------------------
>>>  scripts/kconfig/util.c                    | 75 ++++++++----------------
>>>  scripts/kconfig/zconf.l                   | 20 ++++++-
>>>  scripts/kconfig/zconf.y                   |  2 +-
>>>  19 files changed, 158 insertions(+), 180 deletions(-)
>>>  create mode 100644 scripts/kconfig/env.c
>>>
>>> diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
>>> index f5b9493..0e966e8 100644
>>> --- a/Documentation/kbuild/kconfig-language.txt
>>> +++ b/Documentation/kbuild/kconfig-language.txt
>>> @@ -198,14 +198,6 @@ applicable everywhere (see syntax).
>>>      enables the third modular state for all config symbols.
>>>      At most one symbol may have the "modules" option set.
>>>
>>> -  - "env"=<value>
>>> -    This imports the environment variable into Kconfig. It behaves like
>>> -    a default, except that the value comes from the environment, this
>>> -    also means that the behaviour when mixing it with normal defaults is
>>> -    undefined at this point. The symbol is currently not exported back
>>> -    to the build environment (if this is desired, it can be done via
>>> -    another symbol).
>>> -
>>>    - "allnoconfig_y"
>>>      This declares the symbol as one that should have the value y when
>>>      using "allnoconfig". Used for symbols that hide other symbols.
>>> diff --git a/Kconfig b/Kconfig
>>> index 8c4c1cb..e6ece5b 100644
>>> --- a/Kconfig
>>> +++ b/Kconfig
>>> @@ -5,8 +5,4 @@
>>>  #
>>>  mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
>>>
>>> -config SRCARCH
>>> -       string
>>> -       option env="SRCARCH"
>>> -
>>>  source "arch/$SRCARCH/Kconfig"
>>> diff --git a/Makefile b/Makefile
>>> index 5c395ed..4ae1486 100644
>>> --- a/Makefile
>>> +++ b/Makefile
>>> @@ -284,7 +284,8 @@ include scripts/Kbuild.include
>>>  # Read KERNELRELEASE from include/config/kernel.release (if it exists)
>>>  KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
>>>  KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
>>> -export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
>>> +UNAME_RELEASE := $(shell uname --release)
>>> +export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
>>>
>>>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>>>  # first, and if a usermode build is happening, the "ARCH=um" on the command
>>> diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
>>> index 97fe293..14f3ef1 100644
>>> --- a/arch/sh/Kconfig
>>> +++ b/arch/sh/Kconfig
>>> @@ -57,7 +57,7 @@ config SUPERH
>>>           <http://www.linux-sh.org/>.
>>>
>>>  config SUPERH32
>>> -       def_bool ARCH = "sh"
>>> +       def_bool "$ARCH" = "sh"
>>>         select HAVE_KPROBES
>>>         select HAVE_KRETPROBES
>>>         select HAVE_IOREMAP_PROT if MMU && !X2TLB
>>> @@ -76,7 +76,7 @@ config SUPERH32
>>>         select HAVE_CC_STACKPROTECTOR
>>>
>>>  config SUPERH64
>>> -       def_bool ARCH = "sh64"
>>> +       def_bool "$ARCH" = "sh64"
>>>         select HAVE_EXIT_THREAD
>>>         select KALLSYMS
>>>
>>> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
>>> index 8767e45..86b852e 100644
>>> --- a/arch/sparc/Kconfig
>>> +++ b/arch/sparc/Kconfig
>>> @@ -1,6 +1,6 @@
>>>  config 64BIT
>>> -       bool "64-bit kernel" if ARCH = "sparc"
>>> -       default ARCH = "sparc64"
>>> +       bool "64-bit kernel" if "$ARCH" = "sparc"
>>> +       default "$ARCH" = "sparc64"
>>>         help
>>>           SPARC is a family of RISC microprocessors designed and marketed by
>>>           Sun Microsystems, incorporated.  They are very widely found in Sun
>>> diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
>>> index ef9d403..acc2182 100644
>>> --- a/arch/tile/Kconfig
>>> +++ b/arch/tile/Kconfig
>>> @@ -119,7 +119,7 @@ config HVC_TILE
>>>  # Building with ARCH=tilegx (or ARCH=tile) implies using the
>>>  # 64-bit TILE-Gx toolchain, so force CONFIG_TILEGX on.
>>>  config TILEGX
>>> -       def_bool ARCH != "tilepro"
>>> +       def_bool "$ARCH" != "tilepro"
>>>         select ARCH_SUPPORTS_ATOMIC_RMW
>>>         select GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
>>>         select HAVE_ARCH_JUMP_LABEL
>>> diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
>>> index c68add8..07f84c8 100644
>>> --- a/arch/um/Kconfig.common
>>> +++ b/arch/um/Kconfig.common
>>> @@ -54,10 +54,6 @@ config HZ
>>>         int
>>>         default 100
>>>
>>> -config SUBARCH
>>> -       string
>>> -       option env="SUBARCH"
>>> -
>>>  config NR_CPUS
>>>         int
>>>         range 1 1
>>> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>>> index 0fa71a7..986fb0a 100644
>>> --- a/arch/x86/Kconfig
>>> +++ b/arch/x86/Kconfig
>>> @@ -1,8 +1,8 @@
>>>  # SPDX-License-Identifier: GPL-2.0
>>>  # Select 32 or 64 bit
>>>  config 64BIT
>>> -       bool "64-bit kernel" if ARCH = "x86"
>>> -       default ARCH != "i386"
>>> +       bool "64-bit kernel" if "$ARCH" = "x86"
>>> +       default "$ARCH" != "i386"
>>>         ---help---
>>>           Say yes to build a 64-bit kernel - formerly known as x86_64
>>>           Say no to build a 32-bit kernel - formerly known as i386
>>> diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
>>> index 13ed827..d355413 100644
>>> --- a/arch/x86/um/Kconfig
>>> +++ b/arch/x86/um/Kconfig
>>> @@ -16,8 +16,8 @@ config UML_X86
>>>         select GENERIC_FIND_FIRST_BIT
>>>
>>>  config 64BIT
>>> -       bool "64-bit kernel" if SUBARCH = "x86"
>>> -       default SUBARCH != "i386"
>>> +       bool "64-bit kernel" if $SUBARCH = "x86"
>>> +       default $SUBARCH != "i386"
>>>
>>>  config X86_32
>>>         def_bool !64BIT
>>> diff --git a/init/Kconfig b/init/Kconfig
>>> index df18492..b4814e6 100644
>>> --- a/init/Kconfig
>>> +++ b/init/Kconfig
>>> @@ -1,11 +1,3 @@
>>> -config ARCH
>>> -       string
>>> -       option env="ARCH"
>>> -
>>> -config KERNELVERSION
>>> -       string
>>> -       option env="KERNELVERSION"
>>> -
>>>  config DEFCONFIG_LIST
>>>         string
>>>         depends on !UML
>>> @@ -13,7 +5,7 @@ config DEFCONFIG_LIST
>>>         default "/lib/modules/$UNAME_RELEASE/.config"
>>>         default "/etc/kernel-config"
>>>         default "/boot/config-$UNAME_RELEASE"
>>> -       default "$ARCH_DEFCONFIG"
>>> +       default ARCH_DEFCONFIG
>>>         default "arch/$ARCH/defconfig"
>>>
>>>  config CONSTRUCTORS
>>> diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
>>> index df26c7b..98c2014 100644
>>> --- a/scripts/kconfig/confdata.c
>>> +++ b/scripts/kconfig/confdata.c
>>> @@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
>>>         return name ? name : "include/config/auto.conf";
>>>  }
>>>
>>> -static char *conf_expand_value(const char *in)
>>> -{
>>> -       struct symbol *sym;
>>> -       const char *src;
>>> -       static char res_value[SYMBOL_MAXLENGTH];
>>> -       char *dst, name[SYMBOL_MAXLENGTH];
>>> -
>>> -       res_value[0] = 0;
>>> -       dst = name;
>>> -       while ((src = strchr(in, '$'))) {
>>> -               strncat(res_value, in, src - in);
>>> -               src++;
>>> -               dst = name;
>>> -               while (isalnum(*src) || *src == '_')
>>> -                       *dst++ = *src++;
>>> -               *dst = 0;
>>> -               sym = sym_lookup(name, 0);
>>> -               sym_calc_value(sym);
>>> -               strcat(res_value, sym_get_string_value(sym));
>>> -               in = src;
>>> -       }
>>> -       strcat(res_value, in);
>>> -
>>> -       return res_value;
>>> -}
>>> -
>>>  char *conf_get_default_confname(void)
>>>  {
>>>         struct stat buf;
>>>         static char fullname[PATH_MAX+1];
>>>         char *env, *name;
>>>
>>> -       name = conf_expand_value(conf_defname);
>>> +       name = expand_string_value(conf_defname);
>>>         env = getenv(SRCTREE);
>>>         if (env) {
>>>                 sprintf(fullname, "%s/%s", env, name);
>>> @@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
>>>                         if (expr_calc_value(prop->visible.expr) == no ||
>>>                             prop->expr->type != E_SYMBOL)
>>>                                 continue;
>>> -                       name = conf_expand_value(prop->expr->left.sym->name);
>>> +                       sym_calc_value(prop->expr->left.sym);
>>> +                       name = sym_get_string_value(prop->expr->left.sym);
>>>                         in = zconf_fopen(name);
>>>                         if (in) {
>>>                                 conf_message(_("using defaults found in %s"),
>>> diff --git a/scripts/kconfig/env.c b/scripts/kconfig/env.c
>>> new file mode 100644
>>> index 0000000..9702f5c
>>> --- /dev/null
>>> +++ b/scripts/kconfig/env.c
>>> @@ -0,0 +1,95 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +//
>>> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
>>> +
>>> +static LIST_HEAD(env_list);
>>> +
>>> +struct env {
>>> +       char *name;
>>> +       char *value;
>>> +       struct list_head node;
>>> +};
>>> +
>>> +static struct env *env_list_lookup(const char *name)
>>> +{
>>> +       struct env *e;
>>> +
>>> +       list_for_each_entry(e, &env_list, node) {
>>> +               if (!strcmp(name, e->name))
>>> +                       return e;
>>> +       }
>>> +
>>> +       return NULL;
>>> +}
>>> +
>>> +static void env_list_add(const char *name, const char *value)
>>> +{
>>> +       struct env *e;
>>> +
>>> +       e = xmalloc(sizeof(*e));
>>> +       e->name = xstrdup(name);
>>> +       e->value = xstrdup(value);
>>> +
>>> +       list_add_tail(&e->node, &env_list);
>>> +}
>>> +
>>> +static void env_list_del(struct env *e)
>>> +{
>>> +       list_del(&e->node);
>>> +       free(e->name);
>>> +       free(e->value);
>>> +       free(e);
>>> +}
>>> +
>>> +/* the returned pointer must be freed when done */
>>> +static char *env_expand(const char *name)
>>> +{
>>> +       struct env *e;
>>> +       const char *value;
>>> +
>>> +       e = env_list_lookup(name);
>>> +       if (e)
>>> +               return xstrdup(e->value);
>>> +
>>> +       value = getenv(name);
>>> +       if (!value) {
>>> +               fprintf(stderr, "environment variable \"%s\" undefined\n", name);
>>> +               value = "";
>>> +       }
>>> +
>>> +       /*
>>> +        * we need to remember all referenced environments.
>>> +        * They will be written out to include/config/auto.conf.cmd
>>> +        */
>>> +       env_list_add(name, value);
>>> +
>>> +       return xstrdup(value);
>>> +}
>>> +
>>> +/* works like env_expand, but 'name' does not need to be null-terminated */
>>> +char *env_expand_n(const char *name, size_t n)
>>> +{
>>> +       char *tmp, *res;
>>> +
>>> +       tmp = xmalloc(n + 1);
>>> +       memcpy(tmp, name, n);
>>> +       *(tmp + n) = '\0';
>>> +
>>> +       res = env_expand(tmp);
>>> +
>>> +       free(tmp);
>>> +
>>> +       return res;
>>> +}
>>> +
>>> +void env_write_dep(FILE *f, const char *autoconfig_name)
>>> +{
>>> +       struct env *env, *tmp;
>>> +
>>> +       list_for_each_entry_safe(env, tmp, &env_list, node) {
>>> +               fprintf(f, "ifneq \"$(%s)\" \"%s\"\n", env->name, env->value);
>>> +               fprintf(f, "%s: FORCE\n", autoconfig_name);
>>> +               fprintf(f, "endif\n");
>>> +               env_list_del(env);
>>> +       }
>>> +}
>>> diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
>>> index 3ea9c5f..b3e0ea0 100644
>>> --- a/scripts/kconfig/kconf_id.c
>>> +++ b/scripts/kconfig/kconf_id.c
>>> @@ -32,7 +32,6 @@ static struct kconf_id kconf_id_array[] = {
>>>         { "on",                 T_ON,                   TF_PARAM },
>>>         { "modules",            T_OPT_MODULES,          TF_OPTION },
>>>         { "defconfig_list",     T_OPT_DEFCONFIG_LIST,   TF_OPTION },
>>> -       { "env",                T_OPT_ENV,              TF_OPTION },
>>>         { "allnoconfig_y",      T_OPT_ALLNOCONFIG_Y,    TF_OPTION },
>>>  };
>>>
>>> diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
>>> index c8d9e55..03d007f 100644
>>> --- a/scripts/kconfig/lkc.h
>>> +++ b/scripts/kconfig/lkc.h
>>> @@ -58,7 +58,6 @@ enum conf_def_mode {
>>>
>>>  #define T_OPT_MODULES          1
>>>  #define T_OPT_DEFCONFIG_LIST   2
>>> -#define T_OPT_ENV              3
>>>  #define T_OPT_ALLNOCONFIG_Y    4
>>>
>>>  struct kconf_id {
>>> @@ -95,6 +94,10 @@ static inline void xfwrite(const void *str, size_t len, size_t count, FILE *out)
>>>                 fprintf(stderr, "Error in writing or end of file.\n");
>>>  }
>>>
>>> +/* env.c */
>>> +char *env_expand_n(const char *name, size_t n);
>>> +void env_write_dep(FILE *f, const char *auto_conf_name);
>>> +
>>>  /* menu.c */
>>>  void _menu_init(void);
>>>  void menu_warn(struct menu *menu, const char *fmt, ...);
>>> @@ -135,9 +138,6 @@ void str_printf(struct gstr *gs, const char *fmt, ...);
>>>  const char *str_get(struct gstr *gs);
>>>
>>>  /* symbol.c */
>>> -extern struct expr *sym_env_list;
>>> -
>>> -void sym_init(void);
>>>  void sym_clear_all_valid(void);
>>>  struct symbol *sym_choice_default(struct symbol *sym);
>>>  const char *sym_get_string_default(struct symbol *sym);
>>> diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
>>> index 5c5c137..8148305 100644
>>> --- a/scripts/kconfig/menu.c
>>> +++ b/scripts/kconfig/menu.c
>>> @@ -214,9 +214,6 @@ void menu_add_option(int token, char *arg)
>>>                         zconf_error("trying to redefine defconfig symbol");
>>>                 sym_defconfig_list->flags |= SYMBOL_AUTO;
>>>                 break;
>>> -       case T_OPT_ENV:
>>> -               prop_add_env(arg);
>>> -               break;
>>>         case T_OPT_ALLNOCONFIG_Y:
>>>                 current_entry->sym->flags |= SYMBOL_ALLNOCONFIG_Y;
>>>                 break;
>>> diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
>>> index 03143b2..7c9a88e 100644
>>> --- a/scripts/kconfig/symbol.c
>>> +++ b/scripts/kconfig/symbol.c
>>> @@ -33,33 +33,6 @@ struct symbol *sym_defconfig_list;
>>>  struct symbol *modules_sym;
>>>  tristate modules_val;
>>>
>>> -struct expr *sym_env_list;
>>> -
>>> -static void sym_add_default(struct symbol *sym, const char *def)
>>> -{
>>> -       struct property *prop = prop_alloc(P_DEFAULT, sym);
>>> -
>>> -       prop->expr = expr_alloc_symbol(sym_lookup(def, SYMBOL_CONST));
>>> -}
>>> -
>>> -void sym_init(void)
>>> -{
>>> -       struct symbol *sym;
>>> -       struct utsname uts;
>>> -       static bool inited = false;
>>> -
>>> -       if (inited)
>>> -               return;
>>> -       inited = true;
>>> -
>>> -       uname(&uts);
>>> -
>>> -       sym = sym_lookup("UNAME_RELEASE", 0);
>>> -       sym->type = S_STRING;
>>> -       sym->flags |= SYMBOL_AUTO;
>>> -       sym_add_default(sym, uts.release);
>>> -}
>>> -
>>>  enum symbol_type sym_get_type(struct symbol *sym)
>>>  {
>>>         enum symbol_type type = sym->type;
>>> @@ -1348,32 +1321,3 @@ const char *prop_get_type_name(enum prop_type type)
>>>         }
>>>         return "unknown";
>>>  }
>>> -
>>> -static void prop_add_env(const char *env)
>>> -{
>>> -       struct symbol *sym, *sym2;
>>> -       struct property *prop;
>>> -       char *p;
>>> -
>>> -       sym = current_entry->sym;
>>> -       sym->flags |= SYMBOL_AUTO;
>>> -       for_all_properties(sym, prop, P_ENV) {
>>> -               sym2 = prop_get_symbol(prop);
>>> -               if (strcmp(sym2->name, env))
>>> -                       menu_warn(current_entry, "redefining environment symbol from %s",
>>> -                                 sym2->name);
>>> -               return;
>>> -       }
>>> -
>>> -       prop = prop_alloc(P_ENV, sym);
>>> -       prop->expr = expr_alloc_symbol(sym_lookup(env, SYMBOL_CONST));
>>> -
>>> -       sym_env_list = expr_alloc_one(E_LIST, sym_env_list);
>>> -       sym_env_list->right.sym = sym;
>>> -
>>> -       p = getenv(env);
>>> -       if (p)
>>> -               sym_add_default(sym, p);
>>> -       else
>>> -               menu_warn(current_entry, "environment variable %s undefined", env);
>>> -}
>>> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
>>> index 22201a4..136e497 100644
>>> --- a/scripts/kconfig/util.c
>>> +++ b/scripts/kconfig/util.c
>>> @@ -8,16 +8,18 @@
>>>  #include <stdarg.h>
>>>  #include <stdlib.h>
>>>  #include <string.h>
>>> +
>>> +#include "list.h"
>>>  #include "lkc.h"
>>>
>>>  /*
>>> - * Expand symbol's names embedded in the string given in argument. Symbols'
>>> - * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
>>> + * Expand environments embedded in the string given in argument. Environments
>>> + * to be expanded shall be prefixed by a '$'. Unknown environment expands to
>>>   * the empty string.
>>>   */
>>>  char *expand_string_value(const char *in)
>>>  {
>>> -       const char *src;
>>> +       const char *p, *q;
>>>         char *res;
>>>         size_t reslen;
>>>
>>> @@ -25,39 +27,28 @@ char *expand_string_value(const char *in)
>>>          * Note: 'in' might come from a token that's about to be
>>>          * freed, so make sure to always allocate a new string
>>>          */
>>> -       reslen = strlen(in) + 1;
>>> -       res = xmalloc(reslen);
>>> -       res[0] = '\0';
>>> -
>>> -       while ((src = strchr(in, '$'))) {
>>> -               char *p, name[SYMBOL_MAXLENGTH];
>>> -               const char *symval = "";
>>> -               struct symbol *sym;
>>> -               size_t newlen;
>>> -
>>> -               strncat(res, in, src - in);
>>> -               src++;
>>> -
>>> -               p = name;
>>> -               while (isalnum(*src) || *src == '_')
>>> -                       *p++ = *src++;
>>> -               *p = '\0';
>>> -
>>> -               sym = sym_find(name);
>>> -               if (sym != NULL) {
>>> -                       sym_calc_value(sym);
>>> -                       symval = sym_get_string_value(sym);
>>> -               }
>>> +       res = xmalloc(1);
>>> +       *res = '\0';
>>>
>>> -               newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
>>> -               if (newlen > reslen) {
>>> -                       reslen = newlen;
>>> -                       res = xrealloc(res, reslen);
>>> -               }
>>> +       while ((p = strchr(in, '$'))) {
>>> +               char *new;
>>> +
>>> +               q = p + 1;
>>> +               while (isalnum(*q) || *q == '_')
>>> +                       q++;
>>>
>>> -               strcat(res, symval);
>>> -               in = src;
>>> +               new = env_expand_n(p + 1, q - p - 1);
>>> +
>>> +               reslen = strlen(res) + (p - in) + strlen(new) + 1;
>>> +               res = xrealloc(res, reslen);
>>> +               strncat(res, in, p - in);
>>> +               strcat(res, new);
>>> +               free(new);
>>> +               in = q;
>>>         }
>>> +
>>> +       reslen = strlen(res) + strlen(in) + 1;
>>> +       res = xrealloc(res, reslen);
>>>         strcat(res, in);
>>>
>>>         return res;
>>> @@ -87,8 +78,6 @@ struct file *file_lookup(const char *name)
>>>  /* write a dependency file as used by kbuild to track dependencies */
>>>  int file_write_dep(const char *name)
>>>  {
>>> -       struct symbol *sym, *env_sym;
>>> -       struct expr *e;
>>>         struct file *file;
>>>         FILE *out;
>>>
>>> @@ -107,21 +96,7 @@ int file_write_dep(const char *name)
>>>         fprintf(out, "\n%s: \\\n"
>>>                      "\t$(deps_config)\n\n", conf_get_autoconfig_name());
>>>
>>> -       expr_list_for_each_sym(sym_env_list, e, sym) {
>>> -               struct property *prop;
>>> -               const char *value;
>>> -
>>> -               prop = sym_get_env_prop(sym);
>>> -               env_sym = prop_get_symbol(prop);
>>> -               if (!env_sym)
>>> -                       continue;
>>> -               value = getenv(env_sym->name);
>>> -               if (!value)
>>> -                       value = "";
>>> -               fprintf(out, "ifneq \"$(%s)\" \"%s\"\n", env_sym->name, value);
>>> -               fprintf(out, "%s: FORCE\n", conf_get_autoconfig_name());
>>> -               fprintf(out, "endif\n");
>>> -       }
>>> +       env_write_dep(out, conf_get_autoconfig_name());
>>>
>>>         fprintf(out, "\n$(deps_config): ;\n");
>>>         fclose(out);
>>> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
>>> index 045093d..551ca47 100644
>>> --- a/scripts/kconfig/zconf.l
>>> +++ b/scripts/kconfig/zconf.l
>>> @@ -35,6 +35,7 @@ struct buffer *current_buf;
>>>
>>>  static int last_ts, first_ts;
>>>
>>> +static void expand_string(const char *in);
>>>  static void zconf_endhelp(void);
>>>  static void zconf_endfile(void);
>>>
>>> @@ -120,6 +121,7 @@ n   [A-Za-z0-9_-]
>>>  }
>>>
>>>  <PARAM>{
>>> +       "$".*   expand_string(yytext);
>>>         "&&"    return T_AND;
>>>         "||"    return T_OR;
>>>         "("     return T_OPEN_PAREN;
>>> @@ -157,12 +159,13 @@ n [A-Za-z0-9_-]
>>>  }
>>>
>>>  <STRING>{
>>> -       [^'"\\\n]+/\n   {
>>> +       "$".*   expand_string(yytext);
>>> +       [^$'"\\\n]+/\n  {
>>>                 append_string(yytext, yyleng);
>>>                 yylval.string = text;
>>>                 return T_WORD_QUOTE;
>>>         }
>>> -       [^'"\\\n]+      {
>>> +       [^$'"\\\n]+     {
>>>                 append_string(yytext, yyleng);
>>>         }
>>>         \\.?/\n {
>>> @@ -249,6 +252,19 @@ n  [A-Za-z0-9_-]
>>>  }
>>>
>>>  %%
>>> +static void expand_string(const char *in)
>>> +{
>>> +       char *p, *q;
>>> +
>>> +       p = expand_string_value(in);
>>> +
>>> +       q = p + strlen(p);
>>> +       while (q > p)
>>> +               unput(*--q);
>>> +
>>> +       free(p);
>>> +}
>>> +
>>
>> I like the simplicity of this approach, but I suspect it might be too simple.
>>
>> For example, the following breaks with a syntax error if $ENV has any
>> double quotes in its value:
>>
>>     config FOO
>>         string "foo"
>>         default "$ENV"
>>
>> The following will only work as expected if $ENV expands to a valid
>> Kconfig symbol name. If it doesn't, random stuff will happen (most
>> likely a syntax error).
>>
>>     config FOO
>>         string "foo"
>>         default $ENV
>>
>> The reason it works if $ENV expands to a valid symbol name is that
>> undefined symbols get their name as their (string) value. If the
>> symbol happens to be defined, it will be referenced, which seems
>> confusing too.
>>
>> In general, that reinterpretation of expanded values feels a bit icky
>> to me, and as something that might add complexity to Kconfig for
>> little value. If $ENV outside of quotes absolutely must be supported,
>> I think it should be a shorthand for "$ENV" (which means "constant
>> value" in Kconfig speak).
>
> If you want something as general as the C preprocessor (which I think
> would be overkill and complexity land), then it seems kinda weird to
> limit it to certain Kconfig contexts as well: Right now you'd be able
> to output arbitrary tokens inside an expression, but you couldn't do
> something like generate a 'default X if Y'.
>
> Cheers,
> Ulf

I know you're not a fan, but if expansion was limited to within
constant values (quotes), then you'd be able to handle it all by just
expanding the text before a T_WORD_QUOTE is returned. Extremely
simple, and no gotchas. ;)

Cheers,
Ulf

Re: [PATCH v2 04/21] kconfig: reference environments directly and remove 'option env=' syntax

[Masahiro Yamada]

2018-03-29 11:19 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:.

>>  %%
>> +static void expand_string(const char *in)
>> +{
>> +       char *p, *q;
>> +
>> +       p = expand_string_value(in);
>> +
>> +       q = p + strlen(p);
>> +       while (q > p)
>> +               unput(*--q);
>> +
>> +       free(p);
>> +}
>> +
>
> I like the simplicity of this approach, but I suspect it might be too simple.
>
> For example, the following breaks with a syntax error if $ENV has any
> double quotes in its value:
>
>     config FOO
>         string "foo"
>         default "$ENV"
>
> The following will only work as expected if $ENV expands to a valid
> Kconfig symbol name. If it doesn't, random stuff will happen (most
> likely a syntax error).
>
>     config FOO
>         string "foo"
>         default $ENV
>
> The reason it works if $ENV expands to a valid symbol name is that
> undefined symbols get their name as their (string) value. If the
> symbol happens to be defined, it will be referenced, which seems
> confusing too.
>
> In general, that reinterpretation of expanded values feels a bit icky
> to me, and as something that might add complexity to Kconfig for
> little value. If $ENV outside of quotes absolutely must be supported,
> I think it should be a shorthand for "$ENV" (which means "constant
> value" in Kconfig speak).
>

You are right.  This implementation is too lazy, and bad.
I will change the implementation.

Thanks.




-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 04/21] kconfig: reference environments directly and remove 'option env=' syntax

[Ulf Magnusson]

Here's a more in-depth review:

On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> To get an environment value, Kconfig needs to define a symbol using
> "option env=" syntax.  It is tedious to add a config entry for each
> environment given that we need more environments such as 'CC', 'AS',

"Environment variables" would be clearer. I've never seen them called
"environments".

> 'srctree' etc. to evaluate the compiler capability in Kconfig.
>
> Adding '$' to symbols is weird.  Kconfig can reference symbols directly
> like this:
>
>   config FOO
>           string
>           default BAR
>
> So, I want to use the following syntax to get environment 'BAR' from
> the system:
>
>   config FOO
>           string
>           default $BAR
>
> Looking at the code, the symbols prefixed with 'S' are expanded by:
>  - conf_expand_value()
>    This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
>  - expand_string_value()
>    This is used to expand strings in 'source' and 'mainmenu'
>
> All of them are fixed values independent of user configuration.  So,
> this kind of syntax should be moved to simply take the environment.
>
> This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
> 'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.
>
> sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
> should be be given from the environment.
>
> ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
> by 'default ARCH_DEFCONFIG'.
>
> The environments are expanding in the lexer; when '$' is encountered,

s/environments/environment variables/

> it is expanded, and resulted strings are pushed back to the input
> stream.  This makes the implementation simpler.
>
> For example, the following code works.
>
> [Example code]
>
>   config TOOLCHAIN_LIST
>           string
>           default "My tools: CC=$CC, AS=$AS, CPP=$CPP"
>
> [Result]
>
>   $ make -s alldefconfig && tail -n 1 .config
>   CONFIG_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> I tested all 'make *config' for arch architectures.
> I confirmed this commit still produced the same result
> (by my kconfig test tool).
>
>
> Changes in v2:
>   - Move the string expansion to the lexer phase.
>   - Split environment helpers to env.c
>
>  Documentation/kbuild/kconfig-language.txt |  8 ---
>  Kconfig                                   |  4 --
>  Makefile                                  |  3 +-
>  arch/sh/Kconfig                           |  4 +-
>  arch/sparc/Kconfig                        |  4 +-
>  arch/tile/Kconfig                         |  2 +-
>  arch/um/Kconfig.common                    |  4 --
>  arch/x86/Kconfig                          |  4 +-
>  arch/x86/um/Kconfig                       |  4 +-
>  init/Kconfig                              | 10 +---
>  scripts/kconfig/confdata.c                | 31 +---------
>  scripts/kconfig/env.c                     | 95 +++++++++++++++++++++++++++++++
>  scripts/kconfig/kconf_id.c                |  1 -
>  scripts/kconfig/lkc.h                     |  8 +--
>  scripts/kconfig/menu.c                    |  3 -
>  scripts/kconfig/symbol.c                  | 56 ------------------
>  scripts/kconfig/util.c                    | 75 ++++++++----------------
>  scripts/kconfig/zconf.l                   | 20 ++++++-
>  scripts/kconfig/zconf.y                   |  2 +-
>  19 files changed, 158 insertions(+), 180 deletions(-)
>  create mode 100644 scripts/kconfig/env.c
>
> diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
> index f5b9493..0e966e8 100644
> --- a/Documentation/kbuild/kconfig-language.txt
> +++ b/Documentation/kbuild/kconfig-language.txt
> @@ -198,14 +198,6 @@ applicable everywhere (see syntax).
>      enables the third modular state for all config symbols.
>      At most one symbol may have the "modules" option set.
>
> -  - "env"=<value>
> -    This imports the environment variable into Kconfig. It behaves like
> -    a default, except that the value comes from the environment, this
> -    also means that the behaviour when mixing it with normal defaults is
> -    undefined at this point. The symbol is currently not exported back
> -    to the build environment (if this is desired, it can be done via
> -    another symbol).
> -

The new behavior needs to be documented later as well (but iirc you
already mentioned that somewhere else).

>    - "allnoconfig_y"
>      This declares the symbol as one that should have the value y when
>      using "allnoconfig". Used for symbols that hide other symbols.
> diff --git a/Kconfig b/Kconfig
> index 8c4c1cb..e6ece5b 100644
> --- a/Kconfig
> +++ b/Kconfig
> @@ -5,8 +5,4 @@
>  #
>  mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
>
> -config SRCARCH
> -       string
> -       option env="SRCARCH"
> -
>  source "arch/$SRCARCH/Kconfig"
> diff --git a/Makefile b/Makefile
> index 5c395ed..4ae1486 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -284,7 +284,8 @@ include scripts/Kbuild.include
>  # Read KERNELRELEASE from include/config/kernel.release (if it exists)
>  KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
>  KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
> -export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
> +UNAME_RELEASE := $(shell uname --release)
> +export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
>
>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>  # first, and if a usermode build is happening, the "ARCH=um" on the command
> diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
> index 97fe293..14f3ef1 100644
> --- a/arch/sh/Kconfig
> +++ b/arch/sh/Kconfig
> @@ -57,7 +57,7 @@ config SUPERH
>           <http://www.linux-sh.org/>.
>
>  config SUPERH32
> -       def_bool ARCH = "sh"
> +       def_bool "$ARCH" = "sh"
>         select HAVE_KPROBES
>         select HAVE_KRETPROBES
>         select HAVE_IOREMAP_PROT if MMU && !X2TLB
> @@ -76,7 +76,7 @@ config SUPERH32
>         select HAVE_CC_STACKPROTECTOR
>
>  config SUPERH64
> -       def_bool ARCH = "sh64"
> +       def_bool "$ARCH" = "sh64"
>         select HAVE_EXIT_THREAD
>         select KALLSYMS
>
> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
> index 8767e45..86b852e 100644
> --- a/arch/sparc/Kconfig
> +++ b/arch/sparc/Kconfig
> @@ -1,6 +1,6 @@
>  config 64BIT
> -       bool "64-bit kernel" if ARCH = "sparc"
> -       default ARCH = "sparc64"
> +       bool "64-bit kernel" if "$ARCH" = "sparc"
> +       default "$ARCH" = "sparc64"
>         help
>           SPARC is a family of RISC microprocessors designed and marketed by
>           Sun Microsystems, incorporated.  They are very widely found in Sun
> diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
> index ef9d403..acc2182 100644
> --- a/arch/tile/Kconfig
> +++ b/arch/tile/Kconfig
> @@ -119,7 +119,7 @@ config HVC_TILE
>  # Building with ARCH=tilegx (or ARCH=tile) implies using the
>  # 64-bit TILE-Gx toolchain, so force CONFIG_TILEGX on.
>  config TILEGX
> -       def_bool ARCH != "tilepro"
> +       def_bool "$ARCH" != "tilepro"
>         select ARCH_SUPPORTS_ATOMIC_RMW
>         select GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
>         select HAVE_ARCH_JUMP_LABEL
> diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
> index c68add8..07f84c8 100644
> --- a/arch/um/Kconfig.common
> +++ b/arch/um/Kconfig.common
> @@ -54,10 +54,6 @@ config HZ
>         int
>         default 100
>
> -config SUBARCH
> -       string
> -       option env="SUBARCH"
> -
>  config NR_CPUS
>         int
>         range 1 1
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 0fa71a7..986fb0a 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1,8 +1,8 @@
>  # SPDX-License-Identifier: GPL-2.0
>  # Select 32 or 64 bit
>  config 64BIT
> -       bool "64-bit kernel" if ARCH = "x86"
> -       default ARCH != "i386"
> +       bool "64-bit kernel" if "$ARCH" = "x86"
> +       default "$ARCH" != "i386"
>         ---help---
>           Say yes to build a 64-bit kernel - formerly known as x86_64
>           Say no to build a 32-bit kernel - formerly known as i386
> diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
> index 13ed827..d355413 100644
> --- a/arch/x86/um/Kconfig
> +++ b/arch/x86/um/Kconfig
> @@ -16,8 +16,8 @@ config UML_X86
>         select GENERIC_FIND_FIRST_BIT
>
>  config 64BIT
> -       bool "64-bit kernel" if SUBARCH = "x86"
> -       default SUBARCH != "i386"
> +       bool "64-bit kernel" if $SUBARCH = "x86"
> +       default $SUBARCH != "i386"
>
>  config X86_32
>         def_bool !64BIT
> diff --git a/init/Kconfig b/init/Kconfig
> index df18492..b4814e6 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1,11 +1,3 @@
> -config ARCH
> -       string
> -       option env="ARCH"
> -
> -config KERNELVERSION
> -       string
> -       option env="KERNELVERSION"
> -
>  config DEFCONFIG_LIST
>         string
>         depends on !UML
> @@ -13,7 +5,7 @@ config DEFCONFIG_LIST
>         default "/lib/modules/$UNAME_RELEASE/.config"
>         default "/etc/kernel-config"
>         default "/boot/config-$UNAME_RELEASE"
> -       default "$ARCH_DEFCONFIG"
> +       default ARCH_DEFCONFIG
>         default "arch/$ARCH/defconfig"
>
>  config CONSTRUCTORS
> diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
> index df26c7b..98c2014 100644
> --- a/scripts/kconfig/confdata.c
> +++ b/scripts/kconfig/confdata.c
> @@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
>         return name ? name : "include/config/auto.conf";
>  }
>
> -static char *conf_expand_value(const char *in)
> -{
> -       struct symbol *sym;
> -       const char *src;
> -       static char res_value[SYMBOL_MAXLENGTH];
> -       char *dst, name[SYMBOL_MAXLENGTH];
> -
> -       res_value[0] = 0;
> -       dst = name;
> -       while ((src = strchr(in, '$'))) {
> -               strncat(res_value, in, src - in);
> -               src++;
> -               dst = name;
> -               while (isalnum(*src) || *src == '_')
> -                       *dst++ = *src++;
> -               *dst = 0;
> -               sym = sym_lookup(name, 0);
> -               sym_calc_value(sym);
> -               strcat(res_value, sym_get_string_value(sym));
> -               in = src;
> -       }
> -       strcat(res_value, in);
> -
> -       return res_value;
> -}
> -
>  char *conf_get_default_confname(void)
>  {
>         struct stat buf;
>         static char fullname[PATH_MAX+1];
>         char *env, *name;
>
> -       name = conf_expand_value(conf_defname);
> +       name = expand_string_value(conf_defname);
>         env = getenv(SRCTREE);
>         if (env) {
>                 sprintf(fullname, "%s/%s", env, name);
> @@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
>                         if (expr_calc_value(prop->visible.expr) == no ||
>                             prop->expr->type != E_SYMBOL)
>                                 continue;
> -                       name = conf_expand_value(prop->expr->left.sym->name);
> +                       sym_calc_value(prop->expr->left.sym);
> +                       name = sym_get_string_value(prop->expr->left.sym);
>                         in = zconf_fopen(name);
>                         if (in) {
>                                 conf_message(_("using defaults found in %s"),
> diff --git a/scripts/kconfig/env.c b/scripts/kconfig/env.c
> new file mode 100644
> index 0000000..9702f5c
> --- /dev/null
> +++ b/scripts/kconfig/env.c
> @@ -0,0 +1,95 @@
> +// SPDX-License-Identifier: GPL-2.0
> +//
> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
> +
> +static LIST_HEAD(env_list);
> +
> +struct env {
> +       char *name;
> +       char *value;
> +       struct list_head node;
> +};
> +
> +static struct env *env_list_lookup(const char *name)
> +{
> +       struct env *e;
> +
> +       list_for_each_entry(e, &env_list, node) {
> +               if (!strcmp(name, e->name))
> +                       return e;
> +       }
> +
> +       return NULL;
> +}

This function might be unnecessary -- see below.

> +
> +static void env_list_add(const char *name, const char *value)
> +{
> +       struct env *e;
> +
> +       e = xmalloc(sizeof(*e));
> +       e->name = xstrdup(name);
> +       e->value = xstrdup(value);
> +
> +       list_add_tail(&e->node, &env_list);
> +}
> +
> +static void env_list_del(struct env *e)
> +{
> +       list_del(&e->node);
> +       free(e->name);
> +       free(e->value);
> +       free(e);
> +}
> +
> +/* the returned pointer must be freed when done */
> +static char *env_expand(const char *name)

This function is basically just getenv() with some dependency
housekeeping added. A name like env_get() or env_lookup() would be
clearer I think.

> +{
> +       struct env *e;
> +       const char *value;
> +

> +       e = env_list_lookup(name);
> +       if (e)
> +               return xstrdup(e->value);

Not sure this bit is needed. It is always safe to get the value from
getenv(). See below.

> +
> +       value = getenv(name);
> +       if (!value) {
> +               fprintf(stderr, "environment variable \"%s\" undefined\n", name);
> +               value = "";
> +       }
> +
> +       /*
> +        * we need to remember all referenced environments.

s/environments/environment variables/

> +        * They will be written out to include/config/auto.conf.cmd
> +        */
> +       env_list_add(name, value);

AFAICS, we only need the following functionality:

    1. Record referenced environment variables along with their value

    2. Go through all the recorded environment variables and write
dependency information

For (1), I think it would be better to simply have env_list_add() (or
some other suitable name) add the variable to the list if it isn't
already there (like a kind of set). It is always safe to get the value
of the environment variable from getenv(), and that seems less
confusing.

> +
> +       return xstrdup(value);

The returned string is never modified, and the result from getenv()
never gets stale, so I think the strdup() is unnecessary. The function
could return a const char * to avoid accidental modification.

> +}
> +
> +/* works like env_expand, but 'name' does not need to be null-terminated */
> +char *env_expand_n(const char *name, size_t n)

Could be renamed to something like env_get_n(), and could return a
const char * if the above modification is made.

> +{
> +       char *tmp, *res;
> +
> +       tmp = xmalloc(n + 1);
> +       memcpy(tmp, name, n);
> +       *(tmp + n) = '\0';
> +
> +       res = env_expand(tmp);
> +
> +       free(tmp);
> +
> +       return res;
> +}
> +
> +void env_write_dep(FILE *f, const char *autoconfig_name)
> +{
> +       struct env *env, *tmp;
> +
> +       list_for_each_entry_safe(env, tmp, &env_list, node) {
> +               fprintf(f, "ifneq \"$(%s)\" \"%s\"\n", env->name, env->value);
> +               fprintf(f, "%s: FORCE\n", autoconfig_name);
> +               fprintf(f, "endif\n");
> +               env_list_del(env);
> +       }
> +}
> diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
> index 3ea9c5f..b3e0ea0 100644
> --- a/scripts/kconfig/kconf_id.c
> +++ b/scripts/kconfig/kconf_id.c
> @@ -32,7 +32,6 @@ static struct kconf_id kconf_id_array[] = {
>         { "on",                 T_ON,                   TF_PARAM },
>         { "modules",            T_OPT_MODULES,          TF_OPTION },
>         { "defconfig_list",     T_OPT_DEFCONFIG_LIST,   TF_OPTION },
> -       { "env",                T_OPT_ENV,              TF_OPTION },
>         { "allnoconfig_y",      T_OPT_ALLNOCONFIG_Y,    TF_OPTION },
>  };
>
> diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
> index c8d9e55..03d007f 100644
> --- a/scripts/kconfig/lkc.h
> +++ b/scripts/kconfig/lkc.h
> @@ -58,7 +58,6 @@ enum conf_def_mode {
>
>  #define T_OPT_MODULES          1
>  #define T_OPT_DEFCONFIG_LIST   2
> -#define T_OPT_ENV              3
>  #define T_OPT_ALLNOCONFIG_Y    4

Renumber to 3? Looks like a bitmask to me otherwise.

>
>  struct kconf_id {
> @@ -95,6 +94,10 @@ static inline void xfwrite(const void *str, size_t len, size_t count, FILE *out)
>                 fprintf(stderr, "Error in writing or end of file.\n");
>  }
>
> +/* env.c */
> +char *env_expand_n(const char *name, size_t n);
> +void env_write_dep(FILE *f, const char *auto_conf_name);
> +
>  /* menu.c */
>  void _menu_init(void);
>  void menu_warn(struct menu *menu, const char *fmt, ...);
> @@ -135,9 +138,6 @@ void str_printf(struct gstr *gs, const char *fmt, ...);
>  const char *str_get(struct gstr *gs);
>
>  /* symbol.c */
> -extern struct expr *sym_env_list;
> -
> -void sym_init(void);
>  void sym_clear_all_valid(void);
>  struct symbol *sym_choice_default(struct symbol *sym);
>  const char *sym_get_string_default(struct symbol *sym);
> diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
> index 5c5c137..8148305 100644
> --- a/scripts/kconfig/menu.c
> +++ b/scripts/kconfig/menu.c
> @@ -214,9 +214,6 @@ void menu_add_option(int token, char *arg)
>                         zconf_error("trying to redefine defconfig symbol");
>                 sym_defconfig_list->flags |= SYMBOL_AUTO;
>                 break;
> -       case T_OPT_ENV:
> -               prop_add_env(arg);
> -               break;
>         case T_OPT_ALLNOCONFIG_Y:
>                 current_entry->sym->flags |= SYMBOL_ALLNOCONFIG_Y;
>                 break;
> diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
> index 03143b2..7c9a88e 100644
> --- a/scripts/kconfig/symbol.c
> +++ b/scripts/kconfig/symbol.c
> @@ -33,33 +33,6 @@ struct symbol *sym_defconfig_list;
>  struct symbol *modules_sym;
>  tristate modules_val;
>
> -struct expr *sym_env_list;
> -
> -static void sym_add_default(struct symbol *sym, const char *def)
> -{
> -       struct property *prop = prop_alloc(P_DEFAULT, sym);
> -
> -       prop->expr = expr_alloc_symbol(sym_lookup(def, SYMBOL_CONST));
> -}
> -
> -void sym_init(void)
> -{
> -       struct symbol *sym;
> -       struct utsname uts;
> -       static bool inited = false;
> -
> -       if (inited)
> -               return;
> -       inited = true;
> -
> -       uname(&uts);
> -
> -       sym = sym_lookup("UNAME_RELEASE", 0);
> -       sym->type = S_STRING;
> -       sym->flags |= SYMBOL_AUTO;
> -       sym_add_default(sym, uts.release);
> -}
> -
>  enum symbol_type sym_get_type(struct symbol *sym)
>  {
>         enum symbol_type type = sym->type;
> @@ -1348,32 +1321,3 @@ const char *prop_get_type_name(enum prop_type type)
>         }
>         return "unknown";
>  }
> -
> -static void prop_add_env(const char *env)
> -{
> -       struct symbol *sym, *sym2;
> -       struct property *prop;
> -       char *p;
> -
> -       sym = current_entry->sym;
> -       sym->flags |= SYMBOL_AUTO;
> -       for_all_properties(sym, prop, P_ENV) {
> -               sym2 = prop_get_symbol(prop);
> -               if (strcmp(sym2->name, env))
> -                       menu_warn(current_entry, "redefining environment symbol from %s",
> -                                 sym2->name);
> -               return;
> -       }
> -
> -       prop = prop_alloc(P_ENV, sym);
> -       prop->expr = expr_alloc_symbol(sym_lookup(env, SYMBOL_CONST));
> -
> -       sym_env_list = expr_alloc_one(E_LIST, sym_env_list);
> -       sym_env_list->right.sym = sym;
> -
> -       p = getenv(env);
> -       if (p)
> -               sym_add_default(sym, p);
> -       else
> -               menu_warn(current_entry, "environment variable %s undefined", env);
> -}
> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
> index 22201a4..136e497 100644
> --- a/scripts/kconfig/util.c
> +++ b/scripts/kconfig/util.c
> @@ -8,16 +8,18 @@
>  #include <stdarg.h>
>  #include <stdlib.h>
>  #include <string.h>
> +
> +#include "list.h"
>  #include "lkc.h"
>
>  /*
> - * Expand symbol's names embedded in the string given in argument. Symbols'
> - * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
> + * Expand environments embedded in the string given in argument. Environments

s/environments/environment variables/

> + * to be expanded shall be prefixed by a '$'. Unknown environment expands to
>   * the empty string.
>   */
>  char *expand_string_value(const char *in)
>  {
> -       const char *src;
> +       const char *p, *q;
>         char *res;
>         size_t reslen;
>
> @@ -25,39 +27,28 @@ char *expand_string_value(const char *in)
>          * Note: 'in' might come from a token that's about to be
>          * freed, so make sure to always allocate a new string
>          */
> -       reslen = strlen(in) + 1;
> -       res = xmalloc(reslen);
> -       res[0] = '\0';
> -
> -       while ((src = strchr(in, '$'))) {
> -               char *p, name[SYMBOL_MAXLENGTH];
> -               const char *symval = "";
> -               struct symbol *sym;
> -               size_t newlen;
> -
> -               strncat(res, in, src - in);
> -               src++;
> -
> -               p = name;
> -               while (isalnum(*src) || *src == '_')
> -                       *p++ = *src++;
> -               *p = '\0';
> -
> -               sym = sym_find(name);
> -               if (sym != NULL) {
> -                       sym_calc_value(sym);
> -                       symval = sym_get_string_value(sym);
> -               }
> +       res = xmalloc(1);
> +       *res = '\0';
>
> -               newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
> -               if (newlen > reslen) {
> -                       reslen = newlen;
> -                       res = xrealloc(res, reslen);
> -               }
> +       while ((p = strchr(in, '$'))) {
> +               char *new;
> +
> +               q = p + 1;
> +               while (isalnum(*q) || *q == '_')
> +                       q++;
>
> -               strcat(res, symval);
> -               in = src;
> +               new = env_expand_n(p + 1, q - p - 1);

'value' might be a clearer name than 'new'.

> +
> +               reslen = strlen(res) + (p - in) + strlen(new) + 1;
> +               res = xrealloc(res, reslen);
> +               strncat(res, in, p - in);
> +               strcat(res, new);
> +               free(new);

Not needed it env_expand_n() (env_get_n()) is modified to return the
value from getenv() directly. 'new' is never modified either.

> +               in = q;
>         }
> +
> +       reslen = strlen(res) + strlen(in) + 1;
> +       res = xrealloc(res, reslen);
>         strcat(res, in);
>
>         return res;
> @@ -87,8 +78,6 @@ struct file *file_lookup(const char *name)
>  /* write a dependency file as used by kbuild to track dependencies */
>  int file_write_dep(const char *name)
>  {
> -       struct symbol *sym, *env_sym;
> -       struct expr *e;
>         struct file *file;
>         FILE *out;
>
> @@ -107,21 +96,7 @@ int file_write_dep(const char *name)
>         fprintf(out, "\n%s: \\\n"
>                      "\t$(deps_config)\n\n", conf_get_autoconfig_name());
>
> -       expr_list_for_each_sym(sym_env_list, e, sym) {
> -               struct property *prop;
> -               const char *value;
> -
> -               prop = sym_get_env_prop(sym);
> -               env_sym = prop_get_symbol(prop);
> -               if (!env_sym)
> -                       continue;
> -               value = getenv(env_sym->name);
> -               if (!value)
> -                       value = "";
> -               fprintf(out, "ifneq \"$(%s)\" \"%s\"\n", env_sym->name, value);
> -               fprintf(out, "%s: FORCE\n", conf_get_autoconfig_name());
> -               fprintf(out, "endif\n");
> -       }
> +       env_write_dep(out, conf_get_autoconfig_name());
>
>         fprintf(out, "\n$(deps_config): ;\n");
>         fclose(out);
> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
> index 045093d..551ca47 100644
> --- a/scripts/kconfig/zconf.l
> +++ b/scripts/kconfig/zconf.l
> @@ -35,6 +35,7 @@ struct buffer *current_buf;
>
>  static int last_ts, first_ts;
>
> +static void expand_string(const char *in);
>  static void zconf_endhelp(void);
>  static void zconf_endfile(void);
>
> @@ -120,6 +121,7 @@ n   [A-Za-z0-9_-]
>  }
>
>  <PARAM>{
> +       "$".*   expand_string(yytext);
>         "&&"    return T_AND;
>         "||"    return T_OR;
>         "("     return T_OPEN_PAREN;
> @@ -157,12 +159,13 @@ n [A-Za-z0-9_-]
>  }
>
>  <STRING>{
> -       [^'"\\\n]+/\n   {
> +       "$".*   expand_string(yytext);
> +       [^$'"\\\n]+/\n  {
>                 append_string(yytext, yyleng);
>                 yylval.string = text;
>                 return T_WORD_QUOTE;
>         }
> -       [^'"\\\n]+      {
> +       [^$'"\\\n]+     {
>                 append_string(yytext, yyleng);
>         }
>         \\.?/\n {
> @@ -249,6 +252,19 @@ n  [A-Za-z0-9_-]
>  }
>
>  %%
> +static void expand_string(const char *in)
> +{
> +       char *p, *q;
> +
> +       p = expand_string_value(in);
> +
> +       q = p + strlen(p);
> +       while (q > p)
> +               unput(*--q);
> +
> +       free(p);
> +}

(In case any other reviewers jump in -- see earlier messages in this
thread for some gotchas related to this.)

> +
>  void zconf_starthelp(void)
>  {
>         new_string();
> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
> index 262c464..4ff4ac9 100644
> --- a/scripts/kconfig/zconf.y
> +++ b/scripts/kconfig/zconf.y
> @@ -534,7 +534,6 @@ void conf_parse(const char *name)
>
>         zconf_initscan(name);
>
> -       sym_init();
>         _menu_init();
>
>         if (getenv("ZCONF_DEBUG"))
> @@ -775,6 +774,7 @@ void zconfdump(FILE *out)
>  }
>
>  #include "zconf.lex.c"
> +#include "env.c"
>  #include "util.c"
>  #include "confdata.c"
>  #include "expr.c"
> --
> 2.7.4
>

Cheers,
Ulf

Re: [PATCH v2 04/21] kconfig: reference environments directly and remove 'option env=' syntax

[Ulf Magnusson]

On Sun, Apr 1, 2018 at 4:27 AM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
>
> AFAICS, we only need the following functionality:
>
>     1. Record referenced environment variables along with their value
>
>     2. Go through all the recorded environment variables and write
> dependency information
>
> For (1), I think it would be better to simply have env_list_add() (or
> some other suitable name) add the variable to the list if it isn't
> already there (like a kind of set). It is always safe to get the value
> of the environment variable from getenv(), and that seems less
> confusing.
>

env_list_lookup() could be removed then as well.

Cheers,
Ulf

Re: [PATCH v2 05/21] kconfig: remove string expansion in file_lookup()

[Ulf Magnusson]

On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> There are two callers of file_lookup().
>
> [1] zconf_initscan()
>     This is used to open the first Kconfig.  However, it cannot
>     contain environments in the file path because zconf_fopen() is
>     called before file_lookup().  By swapping the call order,
>     KBUILD_KCONFIG would be able to contain environments, but I do
>     not see practical benefits to support it.
>
> [2] zconf_nextfile()
>     This is used to open the next file from 'source' statement like
>         source "arch/$SRCARCH/Kconfig"
>     but this has already been expanded in the lexer phase.
>
> So, file_lookup() does not need to expand the given path.
>
> By the way, file_lookup() was already buggy; it expanded a given path,
> but it used the path before expansion for look-up:
>         if (!strcmp(name, file->name)) {
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

s/environments/environment variables/

Reviewed-by: Ulf Magnusson <ulfalizer@gmail.com>

Re: [PATCH v2 06/21] kconfig: remove string expansion for mainmenu after yyparse()

[Ulf Magnusson]

On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Now that environments are expanded in the lexer, conf_parse() does
> not need to explicitly expand $ARCH and $KERNELVERSION in the mainmenu.
>
> The hack introduced by commit 0724a7c32a54 ("kconfig: Don't leak
> main menus during parsing") can go away.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Ulf Magnusson <ulfalizer@gmail.com>

Re: [PATCH v2 07/21] kconfig: add function support and implement 'shell' function

[Ulf Magnusson]

On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> This commit adds a new concept 'function' to do more text processing
> in Kconfig.
>
> A function call looks like this:
>
>   $(function arg1, arg2, arg3, ...)
>
> (Actually, this syntax was inspired by make.)
>
> Real examples will look like this:
>
>   $(shell echo hello world)
>   $(cc-option -fstackprotector)
>
> This commit adds the basic infrastructure to add, delete, evaluate
> functions, and also the first built-in function $(shell ...).  This
> accepts a single command to execute.  It returns the standard output
> from it.
>
> [Example code]
>
>   config HELLO
>           string
>           default "$(shell echo hello world)"
>
>   config Y
>           def_bool $(shell echo y)
>
> [Result]
>
>   $ make -s alldefconfig && tail -n 2 .config
>   CONFIG_HELLO="hello world"
>   CONFIG_Y=y
>
> Caveat:
> Like environments, functions are expanded in the lexer.  You cannot
> pass symbols to function arguments.  This is a limitation to simplify
> the implementation.  I want to avoid the dynamic function evaluation,
> which would introduce much more complexity.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Reminder for myself:
> Update Documentation/kbuild/kconfig-language.txt
>
>
> Changes in v2:
>   - Use 'shell' for getting stdout from the comment.
>     It was 'shell-stdout' in the previous version.
>   - Symplify the implementation since the expansion has been moved to
>     lexer.
>
>  scripts/kconfig/function.c  | 170 ++++++++++++++++++++++++++++++++++++++++++++
>  scripts/kconfig/lkc_proto.h |   5 ++
>  scripts/kconfig/util.c      |  46 +++++++++---
>  scripts/kconfig/zconf.y     |   9 +++
>  4 files changed, 222 insertions(+), 8 deletions(-)
>  create mode 100644 scripts/kconfig/function.c
>
> diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
> new file mode 100644
> index 0000000..913685f
> --- /dev/null
> +++ b/scripts/kconfig/function.c
> @@ -0,0 +1,170 @@
> +// SPDX-License-Identifier: GPL-2.0
> +//
> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +
> +#include "list.h"
> +
> +#define FUNCTION_MAX_ARGS              10
> +
> +static LIST_HEAD(function_list);
> +
> +struct function {
> +       char *name;
> +       char *(*func)(struct function *f, int argc, char *argv[]);
> +       struct list_head node;
> +};
> +
> +static struct function *func_lookup(const char *name)
> +{
> +       struct function *f;
> +
> +       list_for_each_entry(f, &function_list, node) {
> +               if (!strcmp(name, f->name))
> +                       return f;
> +       }
> +
> +       return NULL;
> +}
> +
> +static void func_add(const char *name,
> +                    char *(*func)(struct function *f, int argc, char *argv[]))
> +{
> +       struct function *f;
> +
> +       f = func_lookup(name);
> +       if (f) {
> +               fprintf(stderr, "%s: function already exists. ignored.\n", name);
> +               return;
> +       }
> +
> +       f = xmalloc(sizeof(*f));
> +       f->name = xstrdup(name);
> +       f->func = func;
> +
> +       list_add_tail(&f->node, &function_list);
> +}
> +
> +static void func_del(struct function *f)
> +{
> +       list_del(&f->node);
> +       free(f->name);
> +       free(f);
> +}
> +
> +static char *func_call(int argc, char *argv[])
> +{
> +       struct function *f;
> +
> +       f = func_lookup(argv[0]);
> +       if (!f) {
> +               fprintf(stderr, "%s: function not found\n", argv[0]);
> +               return NULL;
> +       }
> +
> +       return f->func(f, argc, argv);
> +}
> +
> +static char *func_eval(const char *func)
> +{
> +       char *expanded, *saveptr, *str, *token, *res;
> +       const char *delim;
> +       int argc = 0;
> +       char *argv[FUNCTION_MAX_ARGS];
> +
> +       expanded = expand_string_value(func);
> +
> +       str = expanded;
> +       delim = " ";
> +
> +       while ((token = strtok_r(str, delim, &saveptr))) {
> +               argv[argc++] = token;

Would be nice to error out if the array is overstepped.

> +               str = NULL;
> +               delim = ",";
> +       }
> +
> +       res = func_call(argc, argv);
> +
> +       free(expanded);
> +
> +       return res ?: xstrdup("");
> +}

Since only 'macro' will take multiple parameters, I wonder if it might
be better to implement the argument parsing there, and simply pass the
string (minus the function name) as-is to functions.

You would then be able to have ',' in shell commands, which might be
required -- think gcc -Wl,option and the like.

> +
> +char *func_eval_n(const char *func, size_t n)
> +{
> +       char *tmp, *res;
> +
> +       tmp = xmalloc(n + 1);
> +       memcpy(tmp, func, n);
> +       *(tmp + n) = '\0';
> +
> +       res = func_eval(tmp);
> +
> +       free(tmp);
> +
> +       return res;
> +}
> +
> +/* built-in functions */
> +static char *do_shell(struct function *f, int argc, char *argv[])
> +{
> +       static const char *pre = "(";
> +       static const char *post = ") 2>/dev/null";

Arrays seem neater, since the pointers aren't needed.

> +       FILE *p;
> +       char buf[256];
> +       char *cmd;
> +       int ret;

Could get rid of 'ret' and just do

    if (pclose(p) == -1)
        perror(cmd);

> +
> +       if (argc != 2)
> +               return NULL;
> +
> +       /*
> +        * Surround the command with ( ) in case it is piped commands.
> +        * Also, redirect stderr to /dev/null.
> +        */
> +       cmd = xmalloc(strlen(pre) + strlen(argv[1]) + strlen(post) + 1);
> +       strcpy(cmd, pre);
> +       strcat(cmd, argv[1]);
> +       strcat(cmd, post);
> +
> +       p = popen(cmd, "r");
> +       if (!p) {
> +               perror(cmd);
> +               goto free;
> +       }
> +       if (fgets(buf, sizeof(buf), p)) {
> +               size_t len = strlen(buf);
> +
> +               if (buf[len - 1] == '\n')
> +                       buf[len - 1] = '\0';
> +       } else {
> +               buf[0] = '\0';
> +       }
> +
> +       ret = pclose(p);
> +       if (ret == -1)
> +               perror(cmd);
> +
> +free:
> +       free(cmd);
> +
> +       return xstrdup(buf);
> +}
> +
> +void func_init(void)
> +{
> +       /* register built-in functions */
> +       func_add("shell", do_shell);
> +}
> +
> +void func_exit(void)
> +{
> +       struct function *f, *tmp;
> +
> +       /* unregister all functions */
> +       list_for_each_entry_safe(f, tmp, &function_list, node)
> +               func_del(f);
> +}
> diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
> index 9884adc..09a4f53 100644
> --- a/scripts/kconfig/lkc_proto.h
> +++ b/scripts/kconfig/lkc_proto.h
> @@ -48,5 +48,10 @@ const char * sym_get_string_value(struct symbol *sym);
>
>  const char * prop_get_type_name(enum prop_type type);
>
> +/* function.c */
> +char *func_eval_n(const char *func, size_t n);
> +void func_init(void);
> +void func_exit(void);
> +
>  /* expr.c */
>  void expr_print(struct expr *e, void (*fn)(void *, struct symbol *, const char *), void *data, int prevtoken);
> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
> index 3d27c49..218b051 100644
> --- a/scripts/kconfig/util.c
> +++ b/scripts/kconfig/util.c
> @@ -13,9 +13,10 @@
>  #include "lkc.h"
>
>  /*
> - * Expand environments embedded in the string given in argument. Environments
> - * to be expanded shall be prefixed by a '$'. Unknown environment expands to
> - * the empty string.
> + * Expand environments and functions embedded in the string given in argument.
> + * Environments to be expanded shall be prefixed by a '$'. Functions to be
> + * evaluated shall be surrounded by $(). Unknown environment/function expands
> + * to the empty string.
>   */
>  char *expand_string_value(const char *in)
>  {
> @@ -33,11 +34,40 @@ char *expand_string_value(const char *in)
>         while ((p = strchr(in, '$'))) {
>                 char *new;
>
> -               q = p + 1;
> -               while (isalnum(*q) || *q == '_')
> -                       q++;
> -
> -               new = env_expand_n(p + 1, q - p - 1);
> +               /*
> +                * If the next character is '(', it is a function.
> +                * Otherwise, environment.
> +                */
> +               if (*(p + 1) == '(') {
> +                       int nest = 0;
> +
> +                       q = p + 2;
> +                       while (1) {
> +                               if (*q == '\0') {
> +                                       fprintf(stderr,
> +                                               "unterminated function: %s\n",
> +                                               p);
> +                                       new = xstrdup("");
> +                                       break;
> +                               } else if (*q == '(') {
> +                                       nest++;
> +                               } else if (*q == ')') {
> +                                       if (nest-- == 0) {
> +                                               new = func_eval_n(p + 2,
> +                                                                 q - p - 2);
> +                                               q++;
> +                                               break;
> +                                       }
> +                               }
> +                               q++;
> +                       }

A loop like this might work too:

    q = p + 1;
    do {
        if (*q == '\0') {
            *error*
            val = ...
            goto error;
        }

        if (*q == '(')
            nest++;
        if (*q == ')')
            nest--;
        q++;
    } while (nest > 0);

    val = func_eval_n(...)
error:

> +               } else {
> +                       q = p + 1;
> +                       while (isalnum(*q) || *q == '_')
> +                               q++;
> +
> +                       new = env_expand_n(p + 1, q - p - 1);
> +               }
>
>                 reslen = strlen(res) + (p - in) + strlen(new) + 1;
>                 res = xrealloc(res, reslen);
> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
> index d8120c7..feaea18 100644
> --- a/scripts/kconfig/zconf.y
> +++ b/scripts/kconfig/zconf.y
> @@ -520,11 +520,19 @@ void conf_parse(const char *name)
>
>         zconf_initscan(name);
>
> +       func_init();
>         _menu_init();
>
>         if (getenv("ZCONF_DEBUG"))
>                 yydebug = 1;
>         yyparse();
> +
> +       /*
> +        * Currently, functions are evaluated only when Kconfig files are
> +        * parsed. We can free functions here.
> +        */
> +       func_exit();
> +
>         if (yynerrs)
>                 exit(1);
>         if (!modules_sym)
> @@ -765,4 +773,5 @@ void zconfdump(FILE *out)
>  #include "confdata.c"
>  #include "expr.c"
>  #include "symbol.c"
> +#include "function.c"
>  #include "menu.c"
> --
> 2.7.4
>

Cheers,
Ulf

Re: [PATCH v2 08/21] kconfig: replace $UNAME_RELEASE with function call

[Ulf Magnusson]

On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Now that 'shell' function is supported, this can be self-contained in
> Kconfig.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Ulf Magnusson <ulfalizer@gmail.com>

Re: [PATCH v2 09/21] kconfig: add 'macro' keyword to support user-defined function

[Ulf Magnusson]

On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Now, we got a basic ability to test compiler capability in Kconfig.
>
> config CC_HAS_STACKPROTECTOR
>         def_bool $(shell (($CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null) && echo y) || echo n)
>
> This works, but it is ugly to repeat this long boilerplate.
>
> We want to describe like this:
>
> config CC_HAS_STACKPROTECTOR
>         bool
>         default $(cc-option -fstack-protector)
>
> It is straight-forward to add a new function, but I do not like to
> hard-code specialized functions like this.  Hence, here is another
> feature to add functions from Kconfig files.
>
> A user-defined function is defined with a special keyword 'macro'.
> It can be referenced in the same way as built-in functions.  This
> feature was also inspired by Makefile where user-defined functions
> are referenced by $(call func-name, args...), but I omitted the 'call'
> to makes it shorter.
>
> The macro definition can contain $(1), $(2), ... which will be replaced
> with arguments from the caller.  The macro works just as a textual
> shorthand, which is also expanded in the lexer phase.
>
> [Example Code]
>
>   macro success $(shell ($(1) && echo y) || echo n)
>
>   config TRUE
>           bool "true"
>           default $(success true)
>
>   config FALSE
>           bool "false"
>           default $(success false)
>
> [Result]
>
>   $ make -s alldefconfig
>   $ tail -n 2 .config
>   CONFIG_TRUE=y
>   # CONFIG_FALSE is not set
>
> [Example Code]
>
>   macro success $(shell ($(1) && echo y) || echo n)
>
>   macro cc-option $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
>
>   config CC_HAS_STACKPROTECTOR
>           def_bool $(cc-option -fstack-protector)
>
> [Result]
>   $ make -s alldefconfig
>   $ tail -n 1 .config
>   CONFIG_CC_HAS_STACKPROTECTOR=y
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Reminder for myself:
> Update Documentation/kbuild/kconfig-language.txt
>
> Changes in v2:
>   - Use 'macro' directly instead of inside the string type symbol.
>
>  scripts/kconfig/function.c  | 59 +++++++++++++++++++++++++++++++++++++++++++--
>  scripts/kconfig/lkc_proto.h |  1 +
>  scripts/kconfig/zconf.l     | 31 ++++++++++++++++++++++++
>  3 files changed, 89 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
> index 913685f..389bb44 100644
> --- a/scripts/kconfig/function.c
> +++ b/scripts/kconfig/function.c
> @@ -15,6 +15,7 @@ static LIST_HEAD(function_list);
>  struct function {
>         char *name;
>         char *(*func)(struct function *f, int argc, char *argv[]);
> +       char *macro;
>         struct list_head node;
>  };
>
> @@ -31,7 +32,8 @@ static struct function *func_lookup(const char *name)
>  }
>
>  static void func_add(const char *name,
> -                    char *(*func)(struct function *f, int argc, char *argv[]))
> +                    char *(*func)(struct function *f, int argc, char *argv[]),
> +                    const char *macro)
>  {
>         struct function *f;
>
> @@ -44,6 +46,7 @@ static void func_add(const char *name,
>         f = xmalloc(sizeof(*f));
>         f->name = xstrdup(name);
>         f->func = func;
> +       f->macro = macro ? xstrdup(macro) : NULL;
>
>         list_add_tail(&f->node, &function_list);
>  }
> @@ -51,6 +54,7 @@ static void func_add(const char *name,
>  static void func_del(struct function *f)
>  {
>         list_del(&f->node);
> +       free(f->macro);
>         free(f->name);
>         free(f);
>  }
> @@ -108,6 +112,57 @@ char *func_eval_n(const char *func, size_t n)
>         return res;
>  }
>
> +/* run user-defined function */
> +static char *do_macro(struct function *f, int argc, char *argv[])
> +{
> +       char *new;
> +       char *src, *p, *res;
> +       size_t newlen;
> +       int n;
> +
> +       new = xmalloc(1);
> +       *new = 0;

new = '\0' would be consistent with the rest of the code.

> +
> +       /*
> +        * This is a format string. $(1), $(2), ... must be replaced with
> +        * function arguments.
> +        */
> +       src = f->macro;
> +       p = src;
> +
> +       while ((p = strstr(p, "$("))) {
> +               if (isdigit(p[2]) && p[3] == ')') {
> +                       n = p[2] - '0';
> +                       if (n < argc) {
> +                               newlen = strlen(new) + (p - src) +
> +                                                       strlen(argv[n]) + 1;
> +                               new = xrealloc(new, newlen);
> +                               strncat(new, src, p - src);
> +                               strcat(new, argv[n]);
> +                               src = p + 4;
> +                       }

Might be nice to warn when a macro call has missing arguments.

> +                       p += 2;
> +               }
> +               p += 2;
> +       }

I had to stare at this for a while to see how it worked. What do you
think of this tweak?

while ((p = strstr(p, "$("))) {
        if (isdigit(p[2]) && p[3] == ')') {
                n = p[2] - '0';
                if (n < argc) {
                        newlen = strlen(new) + (p - src) +
                                                strlen(argv[n]) + 1;
                        new = xrealloc(new, newlen);
                        strncat(new, src, p - src);
                        strcat(new, argv[n]);

                        /*
                         * Jump past macro parameter ("$(n)") and remember the
                         * position
                         */
                        p += 4;
                        src = p;

                        continue;
                }
        }

        /* Jump past "$(" that isn't from a macro parameter */
        p += 2;
}

> +
> +       newlen = strlen(new) + strlen(src) + 1;
> +       new = xrealloc(new, newlen);
> +       strcat(new, src);
> +
> +       res = expand_string_value(new);
> +
> +       free(new);
> +
> +       return res;
> +}
> +
> +/* add user-defined function (macro) */
> +void func_add_macro(const char *name, const char *macro)
> +{
> +       func_add(name, do_macro, macro);
> +}
> +
>  /* built-in functions */
>  static char *do_shell(struct function *f, int argc, char *argv[])
>  {
> @@ -157,7 +212,7 @@ static char *do_shell(struct function *f, int argc, char *argv[])
>  void func_init(void)
>  {
>         /* register built-in functions */
> -       func_add("shell", do_shell);
> +       func_add("shell", do_shell, NULL);
>  }
>
>  void func_exit(void)
> diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
> index 09a4f53..48699c0 100644
> --- a/scripts/kconfig/lkc_proto.h
> +++ b/scripts/kconfig/lkc_proto.h
> @@ -50,6 +50,7 @@ const char * prop_get_type_name(enum prop_type type);
>
>  /* function.c */
>  char *func_eval_n(const char *func, size_t n);
> +void func_add_macro(const char *name, const char *macro);
>  void func_init(void);
>  void func_exit(void);
>
> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
> index 551ca47..6a18c68 100644
> --- a/scripts/kconfig/zconf.l
> +++ b/scripts/kconfig/zconf.l
> @@ -74,6 +74,36 @@ static void warn_ignored_character(char chr)
>                 "%s:%d:warning: ignoring unsupported character '%c'\n",
>                 zconf_curname(), zconf_lineno(), chr);
>  }
> +
> +static void handle_macro(const char *text)
> +{
> +       char *p, *q;
> +
> +       while (isspace(*text))
> +               text++;
> +
> +       p = xstrdup(text);
> +
> +       q = p;
> +       while (isalnum(*q) || *q == '_' || *q == '-')
> +               q++;
> +
> +       if (q == p || !*q) {
> +               yyerror("invalid\n");
> +               goto free;
> +       }
> +
> +       *q = '\0';
> +
> +       q++;
> +       while (isspace(*q))
> +               q++;
> +
> +       func_add_macro(p, q);
> +free:
> +       free(p);
> +}
> +
>  %}
>
>  n      [A-Za-z0-9_-]
> @@ -82,6 +112,7 @@ n    [A-Za-z0-9_-]
>         int str = 0;
>         int ts, i;
>
> +"macro"[ \t].* handle_macro(yytext + 6);
>  [ \t]*#.*\n    |
>  [ \t]*\n       {
>         return T_EOL;
> --
> 2.7.4
>

Cheers,
Ulf

Re: [PATCH v2 10/21] kconfig: add 'success' and 'cc-option' macros

[Ulf Magnusson]

On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 'cc-option' will be the most frequently used macro.  It evaluates to 'y'
> if the given argument is supported by the compiler, or 'n' otherwise.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Changes in v2:
>   - Implement 'success' as a macro
>
>  init/Kconfig | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/init/Kconfig b/init/Kconfig
> index 0d6fe20..c456030 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -8,6 +8,12 @@ config DEFCONFIG_LIST
>         default ARCH_DEFCONFIG
>         default "arch/$ARCH/defconfig"
>
> +# expanded to y if the command exits with 0, n otherwise
> +macro success $(shell ($(1) && echo y) || echo n)
> +
> +# expanded to y if the given compiler flag is supported, n otherwise
> +macro cc-option $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
> +
>  config CONSTRUCTORS
>         bool
>         depends on !UML
> --
> 2.7.4
>

Reviewed-by: Ulf Magnusson <ulfalizer@gmail.com>

Some other tests (e.g. scripts/*-stack-protector.sh) also pass -S and
-O0 and stuff to speed things up. Feels like that could lead to
gotchas though.

-S does help a bit at least:

    $ time gcc -c -x c /dev/null -o /dev/null
    real 0m0.020s
    user 0m0.013s
    sys 0m0.007s

    $ time gcc -S -x c /dev/null -o /dev/null
    real 0m0.012s
    user 0m0.009s
    sys 0m0.003s

Re: [PATCH v2 09/21] kconfig: add 'macro' keyword to support user-defined function

[Ulf Magnusson]

On Sun, Apr 1, 2018 at 8:05 AM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
> On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> Now, we got a basic ability to test compiler capability in Kconfig.
>>
>> config CC_HAS_STACKPROTECTOR
>>         def_bool $(shell (($CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null) && echo y) || echo n)
>>
>> This works, but it is ugly to repeat this long boilerplate.
>>
>> We want to describe like this:
>>
>> config CC_HAS_STACKPROTECTOR
>>         bool
>>         default $(cc-option -fstack-protector)
>>
>> It is straight-forward to add a new function, but I do not like to
>> hard-code specialized functions like this.  Hence, here is another
>> feature to add functions from Kconfig files.
>>
>> A user-defined function is defined with a special keyword 'macro'.
>> It can be referenced in the same way as built-in functions.  This
>> feature was also inspired by Makefile where user-defined functions
>> are referenced by $(call func-name, args...), but I omitted the 'call'
>> to makes it shorter.
>>
>> The macro definition can contain $(1), $(2), ... which will be replaced
>> with arguments from the caller.  The macro works just as a textual
>> shorthand, which is also expanded in the lexer phase.
>>
>> [Example Code]
>>
>>   macro success $(shell ($(1) && echo y) || echo n)
>>
>>   config TRUE
>>           bool "true"
>>           default $(success true)
>>
>>   config FALSE
>>           bool "false"
>>           default $(success false)
>>
>> [Result]
>>
>>   $ make -s alldefconfig
>>   $ tail -n 2 .config
>>   CONFIG_TRUE=y
>>   # CONFIG_FALSE is not set
>>
>> [Example Code]
>>
>>   macro success $(shell ($(1) && echo y) || echo n)
>>
>>   macro cc-option $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
>>
>>   config CC_HAS_STACKPROTECTOR
>>           def_bool $(cc-option -fstack-protector)
>>
>> [Result]
>>   $ make -s alldefconfig
>>   $ tail -n 1 .config
>>   CONFIG_CC_HAS_STACKPROTECTOR=y
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> Reminder for myself:
>> Update Documentation/kbuild/kconfig-language.txt
>>
>> Changes in v2:
>>   - Use 'macro' directly instead of inside the string type symbol.
>>
>>  scripts/kconfig/function.c  | 59 +++++++++++++++++++++++++++++++++++++++++++--
>>  scripts/kconfig/lkc_proto.h |  1 +
>>  scripts/kconfig/zconf.l     | 31 ++++++++++++++++++++++++
>>  3 files changed, 89 insertions(+), 2 deletions(-)
>>
>> diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
>> index 913685f..389bb44 100644
>> --- a/scripts/kconfig/function.c
>> +++ b/scripts/kconfig/function.c
>> @@ -15,6 +15,7 @@ static LIST_HEAD(function_list);
>>  struct function {
>>         char *name;
>>         char *(*func)(struct function *f, int argc, char *argv[]);
>> +       char *macro;
>>         struct list_head node;
>>  };
>>
>> @@ -31,7 +32,8 @@ static struct function *func_lookup(const char *name)
>>  }
>>
>>  static void func_add(const char *name,
>> -                    char *(*func)(struct function *f, int argc, char *argv[]))
>> +                    char *(*func)(struct function *f, int argc, char *argv[]),
>> +                    const char *macro)
>>  {
>>         struct function *f;
>>
>> @@ -44,6 +46,7 @@ static void func_add(const char *name,
>>         f = xmalloc(sizeof(*f));
>>         f->name = xstrdup(name);
>>         f->func = func;
>> +       f->macro = macro ? xstrdup(macro) : NULL;
>>
>>         list_add_tail(&f->node, &function_list);
>>  }
>> @@ -51,6 +54,7 @@ static void func_add(const char *name,
>>  static void func_del(struct function *f)
>>  {
>>         list_del(&f->node);
>> +       free(f->macro);
>>         free(f->name);
>>         free(f);
>>  }
>> @@ -108,6 +112,57 @@ char *func_eval_n(const char *func, size_t n)
>>         return res;
>>  }
>>
>> +/* run user-defined function */
>> +static char *do_macro(struct function *f, int argc, char *argv[])
>> +{
>> +       char *new;
>> +       char *src, *p, *res;
>> +       size_t newlen;
>> +       int n;
>> +
>> +       new = xmalloc(1);
>> +       *new = 0;
>
> new = '\0' would be consistent with the rest of the code.
>
>> +
>> +       /*
>> +        * This is a format string. $(1), $(2), ... must be replaced with
>> +        * function arguments.
>> +        */
>> +       src = f->macro;
>> +       p = src;
>> +
>> +       while ((p = strstr(p, "$("))) {
>> +               if (isdigit(p[2]) && p[3] == ')') {
>> +                       n = p[2] - '0';
>> +                       if (n < argc) {
>> +                               newlen = strlen(new) + (p - src) +
>> +                                                       strlen(argv[n]) + 1;
>> +                               new = xrealloc(new, newlen);
>> +                               strncat(new, src, p - src);
>> +                               strcat(new, argv[n]);
>> +                               src = p + 4;
>> +                       }
>
> Might be nice to warn when a macro call has missing arguments.

Or just error out. There isn't even any backwards compatibility to
think of, and that'd make the code even simpler. Something like this:

while ((p = strstr(p, "$("))) {
        if (isdigit(p[2]) && p[3] == ')') {
                n = p[2] - '0';
                if (n >= argc)
                        *ERROR*

                newlen = strlen(new) + (p - src) + strlen(argv[n]) + 1;
                new = xrealloc(new, newlen);
                strncat(new, src, p - src);
                strcat(new, argv[n]);

                /*
                 * Jump past macro parameter ("$(n)") and remember the new
                 * position
                 */
                p += 4;
                src = p;
        }
        else {
                /* Jump past "$(" that isn't from a macro parameter */
                p += 2;
        }
}

>
>> +                       p += 2;
>> +               }
>> +               p += 2;
>> +       }
>
> I had to stare at this for a while to see how it worked. What do you
> think of this tweak?
>
> while ((p = strstr(p, "$("))) {
>         if (isdigit(p[2]) && p[3] == ')') {
>                 n = p[2] - '0';
>                 if (n < argc) {
>                         newlen = strlen(new) + (p - src) +
>                                                 strlen(argv[n]) + 1;
>                         new = xrealloc(new, newlen);
>                         strncat(new, src, p - src);
>                         strcat(new, argv[n]);
>
>                         /*
>                          * Jump past macro parameter ("$(n)") and remember the
>                          * position
>                          */
>                         p += 4;
>                         src = p;
>
>                         continue;
>                 }
>         }
>
>         /* Jump past "$(" that isn't from a macro parameter */
>         p += 2;
> }
>
>> +
>> +       newlen = strlen(new) + strlen(src) + 1;
>> +       new = xrealloc(new, newlen);
>> +       strcat(new, src);
>> +
>> +       res = expand_string_value(new);
>> +
>> +       free(new);
>> +
>> +       return res;
>> +}
>> +
>> +/* add user-defined function (macro) */
>> +void func_add_macro(const char *name, const char *macro)
>> +{
>> +       func_add(name, do_macro, macro);
>> +}
>> +
>>  /* built-in functions */
>>  static char *do_shell(struct function *f, int argc, char *argv[])
>>  {
>> @@ -157,7 +212,7 @@ static char *do_shell(struct function *f, int argc, char *argv[])
>>  void func_init(void)
>>  {
>>         /* register built-in functions */
>> -       func_add("shell", do_shell);
>> +       func_add("shell", do_shell, NULL);
>>  }
>>
>>  void func_exit(void)
>> diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
>> index 09a4f53..48699c0 100644
>> --- a/scripts/kconfig/lkc_proto.h
>> +++ b/scripts/kconfig/lkc_proto.h
>> @@ -50,6 +50,7 @@ const char * prop_get_type_name(enum prop_type type);
>>
>>  /* function.c */
>>  char *func_eval_n(const char *func, size_t n);
>> +void func_add_macro(const char *name, const char *macro);
>>  void func_init(void);
>>  void func_exit(void);
>>
>> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
>> index 551ca47..6a18c68 100644
>> --- a/scripts/kconfig/zconf.l
>> +++ b/scripts/kconfig/zconf.l
>> @@ -74,6 +74,36 @@ static void warn_ignored_character(char chr)
>>                 "%s:%d:warning: ignoring unsupported character '%c'\n",
>>                 zconf_curname(), zconf_lineno(), chr);
>>  }
>> +
>> +static void handle_macro(const char *text)
>> +{
>> +       char *p, *q;
>> +
>> +       while (isspace(*text))
>> +               text++;
>> +
>> +       p = xstrdup(text);
>> +
>> +       q = p;
>> +       while (isalnum(*q) || *q == '_' || *q == '-')
>> +               q++;
>> +
>> +       if (q == p || !*q) {
>> +               yyerror("invalid\n");
>> +               goto free;
>> +       }
>> +
>> +       *q = '\0';
>> +
>> +       q++;
>> +       while (isspace(*q))
>> +               q++;
>> +
>> +       func_add_macro(p, q);
>> +free:
>> +       free(p);
>> +}
>> +
>>  %}
>>
>>  n      [A-Za-z0-9_-]
>> @@ -82,6 +112,7 @@ n    [A-Za-z0-9_-]
>>         int str = 0;
>>         int ts, i;
>>
>> +"macro"[ \t].* handle_macro(yytext + 6);
>>  [ \t]*#.*\n    |
>>  [ \t]*\n       {
>>         return T_EOL;
>> --
>> 2.7.4
>>
>
> Cheers,
> Ulf

Re: [PATCH v2 11/21] stack-protector: test compiler capability in Kconfig and drop AUTO mode

[Masahiro Yamada]

2018-03-28 20:18 GMT+09:00 Kees Cook <keescook@chromium.org>:
> On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> Move the test for -fstack-protector(-strong) option to Kconfig.
>>
>> If the compiler does not support the option, the corresponding menu
>> is automatically hidden.  If _STRONG is not supported, it will fall
>> back to _REGULAR.  If _REGULAR is not supported, it will be disabled.
>> This means, _AUTO is implicitly handled by the dependency solver of
>> Kconfig, hence removed.
>>
>> I also turned the 'choice' into only two boolean symbols.  The use of
>> 'choice' is not a good idea here, because all of all{yes,mod,no}config
>> would choose the first visible value, while we want allnoconfig to
>> disable as many features as possible.
>>
>> X86 has additional shell scripts in case the compiler supports the
>> option, but generates broken code.  I added CC_HAS_SANE_STACKPROTECTOR
>> to test this.  I had to add -m32 to gcc-x86_32-has-stack-protector.sh
>> to make it work correctly.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>
> This looks really good. Notes below...
>
>> ---
>>
>> Changes in v2:
>>   - Describe $(cc-option ...) directly in depends on context
>>
>>  Makefile                                  | 93 ++-----------------------------
>>  arch/Kconfig                              | 29 +++-------
>>  arch/x86/Kconfig                          |  8 ++-
>>  scripts/gcc-x86_32-has-stack-protector.sh |  7 +--
>>  scripts/gcc-x86_64-has-stack-protector.sh |  5 --
>>  5 files changed, 22 insertions(+), 120 deletions(-)
>>
>> diff --git a/Makefile b/Makefile
>> index 5c395ed..5cadffa 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -675,55 +675,11 @@ ifneq ($(CONFIG_FRAME_WARN),0)
>>  KBUILD_CFLAGS += $(call cc-option,-Wframe-larger-than=${CONFIG_FRAME_WARN})
>>  endif
>>
>> -# This selects the stack protector compiler flag. Testing it is delayed
>> -# until after .config has been reprocessed, in the prepare-compiler-check
>> -# target.
>> -ifdef CONFIG_CC_STACKPROTECTOR_AUTO
>> -  stackp-flag := $(call cc-option,-fstack-protector-strong,$(call cc-option,-fstack-protector))
>> -  stackp-name := AUTO
>> -else
>> -ifdef CONFIG_CC_STACKPROTECTOR_REGULAR
>> -  stackp-flag := -fstack-protector
>> -  stackp-name := REGULAR
>> -else
>> -ifdef CONFIG_CC_STACKPROTECTOR_STRONG
>> -  stackp-flag := -fstack-protector-strong
>> -  stackp-name := STRONG
>> -else
>> -  # If either there is no stack protector for this architecture or
>> -  # CONFIG_CC_STACKPROTECTOR_NONE is selected, we're done, and $(stackp-name)
>> -  # is empty, skipping all remaining stack protector tests.
>> -  #
>> -  # Force off for distro compilers that enable stack protector by default.
>> -  KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
>> -endif
>> -endif
>> -endif
>> -# Find arch-specific stack protector compiler sanity-checking script.
>> -ifdef stackp-name
>> -ifneq ($(stackp-flag),)
>> -  stackp-path := $(srctree)/scripts/gcc-$(SRCARCH)_$(BITS)-has-stack-protector.sh
>> -  stackp-check := $(wildcard $(stackp-path))
>> -  # If the wildcard test matches a test script, run it to check functionality.
>> -  ifdef stackp-check
>> -    ifneq ($(shell $(CONFIG_SHELL) $(stackp-check) $(CC) $(KBUILD_CPPFLAGS) $(biarch)),y)
>> -      stackp-broken := y
>> -    endif
>> -  endif
>> -  ifndef stackp-broken
>> -    # If the stack protector is functional, enable code that depends on it.
>> -    KBUILD_CPPFLAGS += -DCONFIG_CC_STACKPROTECTOR
>> -    # Either we've already detected the flag (for AUTO) or we'll fail the
>> -    # build in the prepare-compiler-check rule (for specific flag).
>> -    KBUILD_CFLAGS += $(stackp-flag)
>> -  else
>> -    # We have to make sure stack protector is unconditionally disabled if
>> -    # the compiler is broken (in case we're going to continue the build in
>> -    # AUTO mode).
>
> Let's keep this comment (slightly rewritten) since the reason for
> setting this flag isn't obvious.

Will move this to help of arch/x86/Kconfig


>> -    KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
>> -  endif
>> -endif
>> -endif
>> +stackp-flags-y                                 := -fno-stack-protector
>
> This is a (minor?) regression in my testing. Making this unconditional
> may break for a compiler built without stack-protector. It should be
> rare, but it's technically possible. Perhaps:
>
> stackp-flags-y := ($call cc-option, -fno-stack-protector)

Will add CONFIG_CC_HAS_STACKPROTECTOR_NONE


>> +stackp-flags-$(CONFIG_CC_STACKPROTECTOR)       := -fstack-protector
>> +stackp-flags-$(CONFIG_CC_STACKPROTECTOR_STRONG)        := -fstack-protector-strong
>> +
>> +KBUILD_CFLAGS += $(stackp-flags-y)
>> [...]
>> diff --git a/arch/Kconfig b/arch/Kconfig
>> index 8e0d665..b42378d 100644
>> --- a/arch/Kconfig
>> +++ b/arch/Kconfig
>> @@ -535,13 +535,13 @@ config HAVE_CC_STACKPROTECTOR
>>         bool
>>         help
>>           An arch should select this symbol if:
>> -         - its compiler supports the -fstack-protector option
>
> Please leave this note: it's still valid. An arch must still have
> compiler support for this to be sensible.
>

No.

"its compiler supports the -fstack-protector option"
is tested by $(cc-option -fstack-protector)

ARCH does not need to know the GCC support level.




>>           - it has implemented a stack canary (e.g. __stack_chk_guard)

-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 11/21] stack-protector: test compiler capability in Kconfig and drop AUTO mode

[Kees Cook]

On Mon, Apr 9, 2018 at 1:54 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2018-03-28 20:18 GMT+09:00 Kees Cook <keescook@chromium.org>:
>> On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
>> <yamada.masahiro@socionext.com> wrote:
>>> diff --git a/arch/Kconfig b/arch/Kconfig
>>> index 8e0d665..b42378d 100644
>>> --- a/arch/Kconfig
>>> +++ b/arch/Kconfig
>>> @@ -535,13 +535,13 @@ config HAVE_CC_STACKPROTECTOR
>>>         bool
>>>         help
>>>           An arch should select this symbol if:
>>> -         - its compiler supports the -fstack-protector option
>>
>> Please leave this note: it's still valid. An arch must still have
>> compiler support for this to be sensible.
>>
>
> No.
>
> "its compiler supports the -fstack-protector option"
> is tested by $(cc-option -fstack-protector)
>
> ARCH does not need to know the GCC support level.

That's not correct: if you enable stack protector for a kernel
architecture that doesn't having it enabled, it's unlikely for the
resulting kernel to boot. An architecture must handle the changes that
the compiler introduces when adding -fstack-protector (for example,
having the stack protector canary value defined, having the failure
function defined, handling context switches changing canaries, etc).

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 11/21] stack-protector: test compiler capability in Kconfig and drop AUTO mode

[Masahiro Yamada]

2018-04-10 0:04 GMT+09:00 Kees Cook <keescook@chromium.org>:
> On Mon, Apr 9, 2018 at 1:54 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> 2018-03-28 20:18 GMT+09:00 Kees Cook <keescook@chromium.org>:
>>> On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
>>> <yamada.masahiro@socionext.com> wrote:
>>>> diff --git a/arch/Kconfig b/arch/Kconfig
>>>> index 8e0d665..b42378d 100644
>>>> --- a/arch/Kconfig
>>>> +++ b/arch/Kconfig
>>>> @@ -535,13 +535,13 @@ config HAVE_CC_STACKPROTECTOR
>>>>         bool
>>>>         help
>>>>           An arch should select this symbol if:
>>>> -         - its compiler supports the -fstack-protector option
>>>
>>> Please leave this note: it's still valid. An arch must still have
>>> compiler support for this to be sensible.
>>>
>>
>> No.
>>
>> "its compiler supports the -fstack-protector option"
>> is tested by $(cc-option -fstack-protector)
>>
>> ARCH does not need to know the GCC support level.
>
> That's not correct: if you enable stack protector for a kernel
> architecture that doesn't having it enabled, it's unlikely for the
> resulting kernel to boot. An architecture must handle the changes that
> the compiler introduces when adding -fstack-protector (for example,
> having the stack protector canary value defined, having the failure
> function defined, handling context switches changing canaries, etc).
>



It is still hard to understand this.


When we "its compiler supports the -fstack-protector option",
we have two meanings

[1] the stack protector feature is implemented in GCC source code.

[2] -fstack-protector is recognized as a valid option in the GCC being used.
    This can be tested by $(cc-option -fstack-protector)

I guess you were talking about [1], where as I [2].
Is this correct?


Does [2] happen only after [1] happens?
Or, are they independent?

If there is a case where GCC recognizes -fstack-protector,
but not implemented?


For x86, there are cases where the option is recognized but not working.
That's why we have
scripts/gcc-x86_{32,64}-has-stack-protector.sh

Generally, if GCC accepts -fstack-protector as a valid option,
we expect "it is working".

I wonder why we need additional information about the compiler
even after $(cc-option -fstack-protector) succeeds.


This is just a matter of comment.

Can you clarify your problem?




> resulting kernel to boot. An architecture must handle the changes that
> the compiler introduces when adding -fstack-protector (for example,
> having the stack protector canary value defined, having the failure
> function defined, handling context switches changing canaries, etc).
>

All of these are talking about the kernel side implementation.
So, it is included in the following comment I am still keeping.

  - it has implemented a stack canary (e.g. __stack_chk_guard)



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 20/21] gcc-plugins: enable GCC_PLUGINS for COMPILE_TEST

[Masahiro Yamada]

2018-03-28 20:47 GMT+09:00 Kees Cook <keescook@chromium.org>:
> On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> The plugin availability is checked in Kconfig, so all{yes,mod}config
>> will not be bothered.  Remove 'depends on !COMPILE_TEST'.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> Changes in v2:
>>   - Remove more 'depends on'
>>
>>  arch/Kconfig | 4 ----
>>  1 file changed, 4 deletions(-)
>>
>> diff --git a/arch/Kconfig b/arch/Kconfig
>> index 88cc925..b07094c 100644
>> --- a/arch/Kconfig
>> +++ b/arch/Kconfig
>> @@ -414,7 +414,6 @@ menuconfig GCC_PLUGINS
>>         bool "GCC plugins"
>>         depends on HAVE_GCC_PLUGINS
>>         depends on $(success $srctree/scripts/gcc-plugin.sh $HOSTCXX $CC)
>> -       depends on !COMPILE_TEST
>>         help
>>           GCC plugins are loadable modules that provide extra features to the
>>           compiler. They are useful for runtime instrumentation and static analysis.
>
> This one is nice, yes! We get coverage for anyone with plugin support.
>
>> @@ -424,7 +423,6 @@ menuconfig GCC_PLUGINS
>>  config GCC_PLUGIN_CYC_COMPLEXITY
>>         bool "Compute the cyclomatic complexity of a function" if EXPERT
>>         depends on GCC_PLUGINS
>> -       depends on !COMPILE_TEST
>>         help
>>           The complexity M of a function's control flow graph is defined as:
>>            M = E - N + 2P
>> @@ -487,7 +485,6 @@ config GCC_PLUGIN_STRUCTLEAK_BYREF_ALL
>>  config GCC_PLUGIN_STRUCTLEAK_VERBOSE
>>         bool "Report forcefully initialized variables"
>>         depends on GCC_PLUGIN_STRUCTLEAK
>> -       depends on !COMPILE_TEST
>>         help
>>           This option will cause a warning to be printed each time the
>>           structleak plugin finds a variable it thinks needs to be
>> @@ -527,7 +524,6 @@ config GCC_PLUGIN_RANDSTRUCT
>>  config GCC_PLUGIN_RANDSTRUCT_PERFORMANCE
>>         bool "Use cacheline-aware structure randomization"
>>         depends on GCC_PLUGIN_RANDSTRUCT
>> -       depends on !COMPILE_TEST
>>         help
>>           If you say Y here, the RANDSTRUCT randomization will make a
>>           best effort at restricting randomization to cacheline-sized
>
> All three of these, though, please leave as they were. They're either
> needlessly noisy for COMPILE_TEST (GCC_PLUGIN_CYC_COMPLEXITY and
> *_VERBOSE),


CONFIG_GCC_PLUGIN_CYC_COMPLEXITY is crazily annoying.
I wonder if the right thing might be to dump the log to
a separate report file instead of messing up the console,
but this is a different issue.
So I will keep this as-is.




CONFIG_GCC_PLUGIN_STRUCTLEAK_VERBOSE prinkles
 'userspace variable will be forcibly initialized'

If this is a matter of warning level,
we generally control this by W= option
as you see in scripts/Makefile.extrawarn

But, maybe I am wrong.  I will keep this as-is
until I figure out what is the right thing to do
for warning verbosity.




> or reduce the COMPILE_TEST coverage
> (GCC_PLUGIN_RANDSTRUCT_PERFORMANCE).
>

Can you explain a bit more about GCC_PLUGIN_RANDSTRUCT_PERFORMANCE?

IIUC, this option reduces the randomization.

Is it related to COMPILE_TEST coverage?






> (And perhaps a comment is needed to explain these to avoid confusion
> in the future?)
>
> -Kees
>
> --
> Kees Cook
> Pixel Security
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 20/21] gcc-plugins: enable GCC_PLUGINS for COMPILE_TEST

[Kees Cook]

On Mon, Apr 9, 2018 at 11:15 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Can you explain a bit more about GCC_PLUGIN_RANDSTRUCT_PERFORMANCE?
>
> IIUC, this option reduces the randomization.

Correct: I don't want to do COMPILE_TEST buids with
GCC_PLUGIN_RANDSTRUCT_PERFORMANCE since it is effectively a subset of
the GCC_PLUGIN_RANDSTRUCT possibilities. So, ..._PERFORMANCE shouldn't
be enabled for COMPILE_TEST, as that would reduce the coverage for
buid tests.

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 19/21] gcc-plugins: test GCC plugin support in Kconfig

[Masahiro Yamada]

2018-03-28 20:44 GMT+09:00 Kees Cook <keescook@chromium.org>:
> On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> Run scripts/gcc-plugin.sh from Kconfig.  Users can enable GCC_PLUGINS
>> only when it is supported.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> Changes in v2: None
>>
>>  arch/Kconfig                 |  4 +++
>>  scripts/Makefile.gcc-plugins | 82 ++++++++++++++++----------------------------
>>  scripts/gcc-plugin.sh        |  1 -
>>  3 files changed, 33 insertions(+), 54 deletions(-)
>>
>> diff --git a/arch/Kconfig b/arch/Kconfig
>> index b42378d..88cc925 100644
>> --- a/arch/Kconfig
>> +++ b/arch/Kconfig
>> @@ -407,9 +407,13 @@ config HAVE_GCC_PLUGINS
>>           An arch should select this symbol if it supports building with
>>           GCC plugins.
>>
>> +config CC_HAS_GCC_PLUGINS
>> +       bool
>> +
>
> This doesn't seem used anywhere?
>
>>  menuconfig GCC_PLUGINS
>>         bool "GCC plugins"
>>         depends on HAVE_GCC_PLUGINS
>> +       depends on $(success $srctree/scripts/gcc-plugin.sh $HOSTCXX $CC)
>>         depends on !COMPILE_TEST
>>         help
>>           GCC plugins are loadable modules that provide extra features to the
>> diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
>> index 25da4c0..19d0d5b 100644
>> --- a/scripts/Makefile.gcc-plugins
>> +++ b/scripts/Makefile.gcc-plugins
>> [...]
>> -# If plugins aren't supported, abort the build before hard-to-read compiler
>> -# errors start getting spewed by the main build.
>> -PHONY += gcc-plugins-check
>> -gcc-plugins-check: FORCE
>> -ifdef CONFIG_GCC_PLUGINS
>> -  ifeq ($(PLUGINCC),)
>> -    ifneq ($(GCC_PLUGINS_CFLAGS),)
>> -      $(Q)$(srctree)/scripts/gcc-plugin.sh --show-error $(HOSTCXX) $(CC) || true
>> -      @echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
>
> As mentioned in the other email, we lose the error reporting. Now the
> lack of plugins is just a silent =n in menuconfig.

This is the right thing to do.
Features unsupported by the compiler should be silently disable.


> Keeping
> --show-error in the Kconfig call and retaining stderr would be nice.

No.
There is no problem to use a compiler without plugin support.

If a user does not want to use plugin in the first place,
why does he/she need to be bothered by such information in stderr?


> I need to do some further testing with SANCOV, but otherwise this all
> looks correct, and my testing shows it behaving correctly.
>
> -Kees
>
> --
> Kees Cook
> Pixel Security
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 19/21] gcc-plugins: test GCC plugin support in Kconfig

[Kees Cook]

On Wed, Apr 11, 2018 at 8:55 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> No.
> There is no problem to use a compiler without plugin support.
>
> If a user does not want to use plugin in the first place,
> why does he/she need to be bothered by such information in stderr?

So, I don't think it's needed for the first version of this, but it'd
be nice to have a way for end users to discover _why_ some option they
want is unavailable. Having them dig through the Kconfigs, edit
scripts to see stderr again, running those by hand, etc, is really
unfriendly. If, instead, stderr was visible from the menuconfig "help"
for a Kconfig, it could serve as automatic documentation for why
something wasn't enabled. I don't mean for stderr to get sprayed out
during a regular "make"; I agree: that would be ugly and pointless.
I'd just like some way for the reason behind an option being disabled
to be discoverable in some way.

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v2 07/21] kconfig: add function support and implement 'shell' function

[Masahiro Yamada]

2018-03-28 12:41 GMT+09:00 Kees Cook <keescook@chromium.org>:
> On Mon, Mar 26, 2018 at 10:29 PM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> This commit adds a new concept 'function' to do more text processing
>> in Kconfig.
>>
>> A function call looks like this:
>>
>>   $(function arg1, arg2, arg3, ...)
>>
>> (Actually, this syntax was inspired by make.)
>>
>> Real examples will look like this:
>>
>>   $(shell echo hello world)
>>   $(cc-option -fstackprotector)
>>
>> This commit adds the basic infrastructure to add, delete, evaluate
>> functions, and also the first built-in function $(shell ...).  This
>> accepts a single command to execute.  It returns the standard output
>> from it.
>>
>> [Example code]
>>
>>   config HELLO
>>           string
>>           default "$(shell echo hello world)"
>>
>>   config Y
>>           def_bool $(shell echo y)
>>
>> [Result]
>>
>>   $ make -s alldefconfig && tail -n 2 .config
>>   CONFIG_HELLO="hello world"
>>   CONFIG_Y=y
>>
>> Caveat:
>> Like environments, functions are expanded in the lexer.  You cannot
>> pass symbols to function arguments.  This is a limitation to simplify
>> the implementation.  I want to avoid the dynamic function evaluation,
>> which would introduce much more complexity.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> Reminder for myself:
>> Update Documentation/kbuild/kconfig-language.txt
>
> Yeah, this needs to be included here, especially given the "cannot
> pass symbols" aspect which might surprise people.
>
>> [...]
>> +/* built-in functions */
>> +static char *do_shell(struct function *f, int argc, char *argv[])
>> +{
>> +       static const char *pre = "(";
>> +       static const char *post = ") 2>/dev/null";
>
> Right now the search and help screens in menuconfig just show the line
> a config is defined and nothing more. I think it would be extremely
> handy to include shell output here in some way.


The current implementation cannot do this.

The $(shell ...) has already expanded
before the parser receives tokens.

There is no way to know whether a token came from a source file as-is,
or it was derived from textual substitution.




> Especially when trying
> to answer questions like "why aren't GCC plugins available?" it's got
> quite a bit harder to debug.
> Could we capture the output (especially stderr) for these kinds of hints?


For example, it would be possible to dump the result of $(shell ...)
evaluation into the console in debug mode.



> Beyond that, looks good!
>
> -Kees
>
> --
> Kees Cook
> Pixel Security
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 07/21] kconfig: add function support and implement 'shell' function

[Masahiro Yamada]

2018-04-01 13:19 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> This commit adds a new concept 'function' to do more text processing
>> in Kconfig.
>>
>> A function call looks like this:
>>
>>   $(function arg1, arg2, arg3, ...)
>>
>> (Actually, this syntax was inspired by make.)
>>
>> Real examples will look like this:
>>
>>   $(shell echo hello world)
>>   $(cc-option -fstackprotector)
>>
>> This commit adds the basic infrastructure to add, delete, evaluate
>> functions, and also the first built-in function $(shell ...).  This
>> accepts a single command to execute.  It returns the standard output
>> from it.
>>
>> [Example code]
>>
>>   config HELLO
>>           string
>>           default "$(shell echo hello world)"
>>
>>   config Y
>>           def_bool $(shell echo y)
>>
>> [Result]
>>
>>   $ make -s alldefconfig && tail -n 2 .config
>>   CONFIG_HELLO="hello world"
>>   CONFIG_Y=y
>>
>> Caveat:
>> Like environments, functions are expanded in the lexer.  You cannot
>> pass symbols to function arguments.  This is a limitation to simplify
>> the implementation.  I want to avoid the dynamic function evaluation,
>> which would introduce much more complexity.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> Reminder for myself:
>> Update Documentation/kbuild/kconfig-language.txt
>>
>>
>> Changes in v2:
>>   - Use 'shell' for getting stdout from the comment.
>>     It was 'shell-stdout' in the previous version.
>>   - Symplify the implementation since the expansion has been moved to
>>     lexer.
>>
>>  scripts/kconfig/function.c  | 170 ++++++++++++++++++++++++++++++++++++++++++++
>>  scripts/kconfig/lkc_proto.h |   5 ++
>>  scripts/kconfig/util.c      |  46 +++++++++---
>>  scripts/kconfig/zconf.y     |   9 +++
>>  4 files changed, 222 insertions(+), 8 deletions(-)
>>  create mode 100644 scripts/kconfig/function.c
>>
>> diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
>> new file mode 100644
>> index 0000000..913685f
>> --- /dev/null
>> +++ b/scripts/kconfig/function.c
>> @@ -0,0 +1,170 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +//
>> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
>> +
>> +#include <stdio.h>
>> +#include <stdlib.h>
>> +#include <string.h>
>> +
>> +#include "list.h"
>> +
>> +#define FUNCTION_MAX_ARGS              10
>> +
>> +static LIST_HEAD(function_list);
>> +
>> +struct function {
>> +       char *name;
>> +       char *(*func)(struct function *f, int argc, char *argv[]);
>> +       struct list_head node;
>> +};
>> +
>> +static struct function *func_lookup(const char *name)
>> +{
>> +       struct function *f;
>> +
>> +       list_for_each_entry(f, &function_list, node) {
>> +               if (!strcmp(name, f->name))
>> +                       return f;
>> +       }
>> +
>> +       return NULL;
>> +}
>> +
>> +static void func_add(const char *name,
>> +                    char *(*func)(struct function *f, int argc, char *argv[]))
>> +{
>> +       struct function *f;
>> +
>> +       f = func_lookup(name);
>> +       if (f) {
>> +               fprintf(stderr, "%s: function already exists. ignored.\n", name);
>> +               return;
>> +       }
>> +
>> +       f = xmalloc(sizeof(*f));
>> +       f->name = xstrdup(name);
>> +       f->func = func;
>> +
>> +       list_add_tail(&f->node, &function_list);
>> +}
>> +
>> +static void func_del(struct function *f)
>> +{
>> +       list_del(&f->node);
>> +       free(f->name);
>> +       free(f);
>> +}
>> +
>> +static char *func_call(int argc, char *argv[])
>> +{
>> +       struct function *f;
>> +
>> +       f = func_lookup(argv[0]);
>> +       if (!f) {
>> +               fprintf(stderr, "%s: function not found\n", argv[0]);
>> +               return NULL;
>> +       }
>> +
>> +       return f->func(f, argc, argv);
>> +}
>> +
>> +static char *func_eval(const char *func)
>> +{
>> +       char *expanded, *saveptr, *str, *token, *res;
>> +       const char *delim;
>> +       int argc = 0;
>> +       char *argv[FUNCTION_MAX_ARGS];
>> +
>> +       expanded = expand_string_value(func);
>> +
>> +       str = expanded;
>> +       delim = " ";
>> +
>> +       while ((token = strtok_r(str, delim, &saveptr))) {
>> +               argv[argc++] = token;
>
> Would be nice to error out if the array is overstepped.

In V3, I made this a feature.


>> +               str = NULL;
>> +               delim = ",";
>> +       }
>> +
>> +       res = func_call(argc, argv);
>> +
>> +       free(expanded);
>> +
>> +       return res ?: xstrdup("");
>> +}
>
> Since only 'macro' will take multiple parameters, I wonder if it might
> be better to implement the argument parsing there, and simply pass the
> string (minus the function name) as-is to functions.

I may add more built-in functions in the future.
For example, I want to add 'if' function, which takes two or three arguments.


> You would then be able to have ',' in shell commands, which might be
> required -- think gcc -Wl,option and the like.

I fixed 'shell' function in v3.


>> +
>> +char *func_eval_n(const char *func, size_t n)
>> +{
>> +       char *tmp, *res;
>> +
>> +       tmp = xmalloc(n + 1);
>> +       memcpy(tmp, func, n);
>> +       *(tmp + n) = '\0';
>> +
>> +       res = func_eval(tmp);
>> +
>> +       free(tmp);
>> +
>> +       return res;
>> +}
>> +
>> +/* built-in functions */
>> +static char *do_shell(struct function *f, int argc, char *argv[])
>> +{
>> +       static const char *pre = "(";
>> +       static const char *post = ") 2>/dev/null";
>
> Arrays seem neater, since the pointers aren't needed.
>
>> +       FILE *p;
>> +       char buf[256];
>> +       char *cmd;
>> +       int ret;
>
> Could get rid of 'ret' and just do
>
>     if (pclose(p) == -1)
>         perror(cmd);

Done.


>> +
>> +       if (argc != 2)
>> +               return NULL;
>> +
>> +       /*
>> +        * Surround the command with ( ) in case it is piped commands.
>> +        * Also, redirect stderr to /dev/null.
>> +        */
>> +       cmd = xmalloc(strlen(pre) + strlen(argv[1]) + strlen(post) + 1);
>> +       strcpy(cmd, pre);
>> +       strcat(cmd, argv[1]);
>> +       strcat(cmd, post);
>> +
>> +       p = popen(cmd, "r");
>> +       if (!p) {
>> +               perror(cmd);
>> +               goto free;
>> +       }
>> +       if (fgets(buf, sizeof(buf), p)) {
>> +               size_t len = strlen(buf);
>> +
>> +               if (buf[len - 1] == '\n')
>> +                       buf[len - 1] = '\0';
>> +       } else {
>> +               buf[0] = '\0';
>> +       }
>> +
>> +       ret = pclose(p);
>> +       if (ret == -1)
>> +               perror(cmd);
>> +
>> +free:
>> +       free(cmd);
>> +
>> +       return xstrdup(buf);
>> +}
>> +
>> +void func_init(void)
>> +{
>> +       /* register built-in functions */
>> +       func_add("shell", do_shell);
>> +}
>> +
>> +void func_exit(void)
>> +{
>> +       struct function *f, *tmp;
>> +
>> +       /* unregister all functions */
>> +       list_for_each_entry_safe(f, tmp, &function_list, node)
>> +               func_del(f);
>> +}
>> diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
>> index 9884adc..09a4f53 100644
>> --- a/scripts/kconfig/lkc_proto.h
>> +++ b/scripts/kconfig/lkc_proto.h
>> @@ -48,5 +48,10 @@ const char * sym_get_string_value(struct symbol *sym);
>>
>>  const char * prop_get_type_name(enum prop_type type);
>>
>> +/* function.c */
>> +char *func_eval_n(const char *func, size_t n);
>> +void func_init(void);
>> +void func_exit(void);
>> +
>>  /* expr.c */
>>  void expr_print(struct expr *e, void (*fn)(void *, struct symbol *, const char *), void *data, int prevtoken);
>> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
>> index 3d27c49..218b051 100644
>> --- a/scripts/kconfig/util.c
>> +++ b/scripts/kconfig/util.c
>> @@ -13,9 +13,10 @@
>>  #include "lkc.h"
>>
>>  /*
>> - * Expand environments embedded in the string given in argument. Environments
>> - * to be expanded shall be prefixed by a '$'. Unknown environment expands to
>> - * the empty string.
>> + * Expand environments and functions embedded in the string given in argument.
>> + * Environments to be expanded shall be prefixed by a '$'. Functions to be
>> + * evaluated shall be surrounded by $(). Unknown environment/function expands
>> + * to the empty string.
>>   */
>>  char *expand_string_value(const char *in)
>>  {
>> @@ -33,11 +34,40 @@ char *expand_string_value(const char *in)
>>         while ((p = strchr(in, '$'))) {
>>                 char *new;
>>
>> -               q = p + 1;
>> -               while (isalnum(*q) || *q == '_')
>> -                       q++;
>> -
>> -               new = env_expand_n(p + 1, q - p - 1);
>> +               /*
>> +                * If the next character is '(', it is a function.
>> +                * Otherwise, environment.
>> +                */
>> +               if (*(p + 1) == '(') {
>> +                       int nest = 0;
>> +
>> +                       q = p + 2;
>> +                       while (1) {
>> +                               if (*q == '\0') {
>> +                                       fprintf(stderr,
>> +                                               "unterminated function: %s\n",
>> +                                               p);
>> +                                       new = xstrdup("");
>> +                                       break;
>> +                               } else if (*q == '(') {
>> +                                       nest++;
>> +                               } else if (*q == ')') {
>> +                                       if (nest-- == 0) {
>> +                                               new = func_eval_n(p + 2,
>> +                                                                 q - p - 2);
>> +                                               q++;
>> +                                               break;
>> +                                       }
>> +                               }
>> +                               q++;
>> +                       }
>
> A loop like this might work too:
>
>     q = p + 1;
>     do {
>         if (*q == '\0') {
>             *error*
>             val = ...
>             goto error;
>         }
>
>         if (*q == '(')
>             nest++;
>         if (*q == ')')
>             nest--;
>         q++;
>     } while (nest > 0);
>
>     val = func_eval_n(...)
> error:
>
>> +               } else {
>> +                       q = p + 1;
>> +                       while (isalnum(*q) || *q == '_')
>> +                               q++;
>> +
>> +                       new = env_expand_n(p + 1, q - p - 1);
>> +               }
>>
>>                 reslen = strlen(res) + (p - in) + strlen(new) + 1;
>>                 res = xrealloc(res, reslen);
>> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
>> index d8120c7..feaea18 100644
>> --- a/scripts/kconfig/zconf.y
>> +++ b/scripts/kconfig/zconf.y
>> @@ -520,11 +520,19 @@ void conf_parse(const char *name)
>>
>>         zconf_initscan(name);
>>
>> +       func_init();
>>         _menu_init();
>>
>>         if (getenv("ZCONF_DEBUG"))
>>                 yydebug = 1;
>>         yyparse();
>> +
>> +       /*
>> +        * Currently, functions are evaluated only when Kconfig files are
>> +        * parsed. We can free functions here.
>> +        */
>> +       func_exit();
>> +
>>         if (yynerrs)
>>                 exit(1);
>>         if (!modules_sym)
>> @@ -765,4 +773,5 @@ void zconfdump(FILE *out)
>>  #include "confdata.c"
>>  #include "expr.c"
>>  #include "symbol.c"
>> +#include "function.c"
>>  #include "menu.c"
>> --
>> 2.7.4
>>
>
> Cheers,
> Ulf
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 09/21] kconfig: add 'macro' keyword to support user-defined function

[Masahiro Yamada]

2018-04-01 15:05 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> Now, we got a basic ability to test compiler capability in Kconfig.
>>
>> config CC_HAS_STACKPROTECTOR
>>         def_bool $(shell (($CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null) && echo y) || echo n)
>>
>> This works, but it is ugly to repeat this long boilerplate.
>>
>> We want to describe like this:
>>
>> config CC_HAS_STACKPROTECTOR
>>         bool
>>         default $(cc-option -fstack-protector)
>>
>> It is straight-forward to add a new function, but I do not like to
>> hard-code specialized functions like this.  Hence, here is another
>> feature to add functions from Kconfig files.
>>
>> A user-defined function is defined with a special keyword 'macro'.
>> It can be referenced in the same way as built-in functions.  This
>> feature was also inspired by Makefile where user-defined functions
>> are referenced by $(call func-name, args...), but I omitted the 'call'
>> to makes it shorter.
>>
>> The macro definition can contain $(1), $(2), ... which will be replaced
>> with arguments from the caller.  The macro works just as a textual
>> shorthand, which is also expanded in the lexer phase.
>>
>> [Example Code]
>>
>>   macro success $(shell ($(1) && echo y) || echo n)
>>
>>   config TRUE
>>           bool "true"
>>           default $(success true)
>>
>>   config FALSE
>>           bool "false"
>>           default $(success false)
>>
>> [Result]
>>
>>   $ make -s alldefconfig
>>   $ tail -n 2 .config
>>   CONFIG_TRUE=y
>>   # CONFIG_FALSE is not set
>>
>> [Example Code]
>>
>>   macro success $(shell ($(1) && echo y) || echo n)
>>
>>   macro cc-option $(success $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
>>
>>   config CC_HAS_STACKPROTECTOR
>>           def_bool $(cc-option -fstack-protector)
>>
>> [Result]
>>   $ make -s alldefconfig
>>   $ tail -n 1 .config
>>   CONFIG_CC_HAS_STACKPROTECTOR=y
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> Reminder for myself:
>> Update Documentation/kbuild/kconfig-language.txt
>>
>> Changes in v2:
>>   - Use 'macro' directly instead of inside the string type symbol.
>>
>>  scripts/kconfig/function.c  | 59 +++++++++++++++++++++++++++++++++++++++++++--
>>  scripts/kconfig/lkc_proto.h |  1 +
>>  scripts/kconfig/zconf.l     | 31 ++++++++++++++++++++++++
>>  3 files changed, 89 insertions(+), 2 deletions(-)
>>
>> diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
>> index 913685f..389bb44 100644
>> --- a/scripts/kconfig/function.c
>> +++ b/scripts/kconfig/function.c
>> @@ -15,6 +15,7 @@ static LIST_HEAD(function_list);
>>  struct function {
>>         char *name;
>>         char *(*func)(struct function *f, int argc, char *argv[]);
>> +       char *macro;
>>         struct list_head node;
>>  };
>>
>> @@ -31,7 +32,8 @@ static struct function *func_lookup(const char *name)
>>  }
>>
>>  static void func_add(const char *name,
>> -                    char *(*func)(struct function *f, int argc, char *argv[]))
>> +                    char *(*func)(struct function *f, int argc, char *argv[]),
>> +                    const char *macro)
>>  {
>>         struct function *f;
>>
>> @@ -44,6 +46,7 @@ static void func_add(const char *name,
>>         f = xmalloc(sizeof(*f));
>>         f->name = xstrdup(name);
>>         f->func = func;
>> +       f->macro = macro ? xstrdup(macro) : NULL;
>>
>>         list_add_tail(&f->node, &function_list);
>>  }
>> @@ -51,6 +54,7 @@ static void func_add(const char *name,
>>  static void func_del(struct function *f)
>>  {
>>         list_del(&f->node);
>> +       free(f->macro);
>>         free(f->name);
>>         free(f);
>>  }
>> @@ -108,6 +112,57 @@ char *func_eval_n(const char *func, size_t n)
>>         return res;
>>  }
>>
>> +/* run user-defined function */
>> +static char *do_macro(struct function *f, int argc, char *argv[])
>> +{
>> +       char *new;
>> +       char *src, *p, *res;
>> +       size_t newlen;
>> +       int n;
>> +
>> +       new = xmalloc(1);
>> +       *new = 0;
>
> new = '\0' would be consistent with the rest of the code.
>
>> +
>> +       /*
>> +        * This is a format string. $(1), $(2), ... must be replaced with
>> +        * function arguments.
>> +        */
>> +       src = f->macro;
>> +       p = src;
>> +
>> +       while ((p = strstr(p, "$("))) {
>> +               if (isdigit(p[2]) && p[3] == ')') {
>> +                       n = p[2] - '0';
>> +                       if (n < argc) {
>> +                               newlen = strlen(new) + (p - src) +
>> +                                                       strlen(argv[n]) + 1;
>> +                               new = xrealloc(new, newlen);
>> +                               strncat(new, src, p - src);
>> +                               strcat(new, argv[n]);
>> +                               src = p + 4;
>> +                       }
>
> Might be nice to warn when a macro call has missing arguments.


I made this a feature as we see in Make.

In Makefile, it is allowed to pass more or less arguments.

cc-option is often used with one flag.

   $(call cc-option,-fno-PIE)

But, it can take one more as a fall-back compiler option.

   $(call cc-option,-mtune=marvell-f,-mtune=xscale)


Of course, we can describe the first one like follows
just to pass a blank parameter as the last parameter.

   $(call cc-option,-fno-PIE,)




>> +                       p += 2;
>> +               }
>> +               p += 2;
>> +       }
>
> I had to stare at this for a while to see how it worked. What do you
> think of this tweak?
>
> while ((p = strstr(p, "$("))) {
>         if (isdigit(p[2]) && p[3] == ')') {
>                 n = p[2] - '0';
>                 if (n < argc) {
>                         newlen = strlen(new) + (p - src) +
>                                                 strlen(argv[n]) + 1;
>                         new = xrealloc(new, newlen);
>                         strncat(new, src, p - src);
>                         strcat(new, argv[n]);
>
>                         /*
>                          * Jump past macro parameter ("$(n)") and remember the
>                          * position
>                          */
>                         p += 4;
>                         src = p;
>
>                         continue;
>                 }
>         }
>
>         /* Jump past "$(" that isn't from a macro parameter */
>         p += 2;
> }


Will work, but I re-implemented the parser in v3.


>> +
>> +       newlen = strlen(new) + strlen(src) + 1;
>> +       new = xrealloc(new, newlen);
>> +       strcat(new, src);
>> +
>> +       res = expand_string_value(new);
>> +
>> +       free(new);
>> +
>> +       return res;
>> +}
>> +
>> +/* add user-defined function (macro) */
>> +void func_add_macro(const char *name, const char *macro)
>> +{
>> +       func_add(name, do_macro, macro);
>> +}
>> +
>>  /* built-in functions */
>>  static char *do_shell(struct function *f, int argc, char *argv[])
>>  {
>> @@ -157,7 +212,7 @@ static char *do_shell(struct function *f, int argc, char *argv[])
>>  void func_init(void)
>>  {
>>         /* register built-in functions */
>> -       func_add("shell", do_shell);
>> +       func_add("shell", do_shell, NULL);
>>  }
>>
>>  void func_exit(void)
>> diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
>> index 09a4f53..48699c0 100644
>> --- a/scripts/kconfig/lkc_proto.h
>> +++ b/scripts/kconfig/lkc_proto.h
>> @@ -50,6 +50,7 @@ const char * prop_get_type_name(enum prop_type type);
>>
>>  /* function.c */
>>  char *func_eval_n(const char *func, size_t n);
>> +void func_add_macro(const char *name, const char *macro);
>>  void func_init(void);
>>  void func_exit(void);
>>
>> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
>> index 551ca47..6a18c68 100644
>> --- a/scripts/kconfig/zconf.l
>> +++ b/scripts/kconfig/zconf.l
>> @@ -74,6 +74,36 @@ static void warn_ignored_character(char chr)
>>                 "%s:%d:warning: ignoring unsupported character '%c'\n",
>>                 zconf_curname(), zconf_lineno(), chr);
>>  }
>> +
>> +static void handle_macro(const char *text)
>> +{
>> +       char *p, *q;
>> +
>> +       while (isspace(*text))
>> +               text++;
>> +
>> +       p = xstrdup(text);
>> +
>> +       q = p;
>> +       while (isalnum(*q) || *q == '_' || *q == '-')
>> +               q++;
>> +
>> +       if (q == p || !*q) {
>> +               yyerror("invalid\n");
>> +               goto free;
>> +       }
>> +
>> +       *q = '\0';
>> +
>> +       q++;
>> +       while (isspace(*q))
>> +               q++;
>> +
>> +       func_add_macro(p, q);
>> +free:
>> +       free(p);
>> +}
>> +
>>  %}
>>
>>  n      [A-Za-z0-9_-]
>> @@ -82,6 +112,7 @@ n    [A-Za-z0-9_-]
>>         int str = 0;
>>         int ts, i;
>>
>> +"macro"[ \t].* handle_macro(yytext + 6);
>>  [ \t]*#.*\n    |
>>  [ \t]*\n       {
>>         return T_EOL;
>> --
>> 2.7.4
>>
>
> Cheers,
> Ulf
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 04/21] kconfig: reference environments directly and remove 'option env=' syntax

[Masahiro Yamada]

2018-04-01 11:27 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> Here's a more in-depth review:
>
> On Tue, Mar 27, 2018 at 7:29 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> To get an environment value, Kconfig needs to define a symbol using
>> "option env=" syntax.  It is tedious to add a config entry for each
>> environment given that we need more environments such as 'CC', 'AS',
>
> "Environment variables" would be clearer. I've never seen them called
> "environments".
>
>> 'srctree' etc. to evaluate the compiler capability in Kconfig.
>>
>> Adding '$' to symbols is weird.  Kconfig can reference symbols directly
>> like this:
>>
>>   config FOO
>>           string
>>           default BAR
>>
>> So, I want to use the following syntax to get environment 'BAR' from
>> the system:
>>
>>   config FOO
>>           string
>>           default $BAR
>>
>> Looking at the code, the symbols prefixed with 'S' are expanded by:
>>  - conf_expand_value()
>>    This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
>>  - expand_string_value()
>>    This is used to expand strings in 'source' and 'mainmenu'
>>
>> All of them are fixed values independent of user configuration.  So,
>> this kind of syntax should be moved to simply take the environment.
>>
>> This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
>> 'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.
>>
>> sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
>> should be be given from the environment.
>>
>> ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
>> by 'default ARCH_DEFCONFIG'.
>>
>> The environments are expanding in the lexer; when '$' is encountered,
>
> s/environments/environment variables/
>
>> it is expanded, and resulted strings are pushed back to the input
>> stream.  This makes the implementation simpler.
>>
>> For example, the following code works.
>>
>> [Example code]
>>
>>   config TOOLCHAIN_LIST
>>           string
>>           default "My tools: CC=$CC, AS=$AS, CPP=$CPP"
>>
>> [Result]
>>
>>   $ make -s alldefconfig && tail -n 1 .config
>>   CONFIG_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> I tested all 'make *config' for arch architectures.
>> I confirmed this commit still produced the same result
>> (by my kconfig test tool).
>>
>>
>> Changes in v2:
>>   - Move the string expansion to the lexer phase.
>>   - Split environment helpers to env.c
>>
>>  Documentation/kbuild/kconfig-language.txt |  8 ---
>>  Kconfig                                   |  4 --
>>  Makefile                                  |  3 +-
>>  arch/sh/Kconfig                           |  4 +-
>>  arch/sparc/Kconfig                        |  4 +-
>>  arch/tile/Kconfig                         |  2 +-
>>  arch/um/Kconfig.common                    |  4 --
>>  arch/x86/Kconfig                          |  4 +-
>>  arch/x86/um/Kconfig                       |  4 +-
>>  init/Kconfig                              | 10 +---
>>  scripts/kconfig/confdata.c                | 31 +---------
>>  scripts/kconfig/env.c                     | 95 +++++++++++++++++++++++++++++++
>>  scripts/kconfig/kconf_id.c                |  1 -
>>  scripts/kconfig/lkc.h                     |  8 +--
>>  scripts/kconfig/menu.c                    |  3 -
>>  scripts/kconfig/symbol.c                  | 56 ------------------
>>  scripts/kconfig/util.c                    | 75 ++++++++----------------
>>  scripts/kconfig/zconf.l                   | 20 ++++++-
>>  scripts/kconfig/zconf.y                   |  2 +-
>>  19 files changed, 158 insertions(+), 180 deletions(-)
>>  create mode 100644 scripts/kconfig/env.c
>>
>> diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
>> index f5b9493..0e966e8 100644
>> --- a/Documentation/kbuild/kconfig-language.txt
>> +++ b/Documentation/kbuild/kconfig-language.txt
>> @@ -198,14 +198,6 @@ applicable everywhere (see syntax).
>>      enables the third modular state for all config symbols.
>>      At most one symbol may have the "modules" option set.
>>
>> -  - "env"=<value>
>> -    This imports the environment variable into Kconfig. It behaves like
>> -    a default, except that the value comes from the environment, this
>> -    also means that the behaviour when mixing it with normal defaults is
>> -    undefined at this point. The symbol is currently not exported back
>> -    to the build environment (if this is desired, it can be done via
>> -    another symbol).
>> -
>
> The new behavior needs to be documented later as well (but iirc you
> already mentioned that somewhere else).
>
>>    - "allnoconfig_y"
>>      This declares the symbol as one that should have the value y when
>>      using "allnoconfig". Used for symbols that hide other symbols.
>> diff --git a/Kconfig b/Kconfig
>> index 8c4c1cb..e6ece5b 100644
>> --- a/Kconfig
>> +++ b/Kconfig
>> @@ -5,8 +5,4 @@
>>  #
>>  mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
>>
>> -config SRCARCH
>> -       string
>> -       option env="SRCARCH"
>> -
>>  source "arch/$SRCARCH/Kconfig"
>> diff --git a/Makefile b/Makefile
>> index 5c395ed..4ae1486 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -284,7 +284,8 @@ include scripts/Kbuild.include
>>  # Read KERNELRELEASE from include/config/kernel.release (if it exists)
>>  KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
>>  KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
>> -export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
>> +UNAME_RELEASE := $(shell uname --release)
>> +export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
>>
>>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>>  # first, and if a usermode build is happening, the "ARCH=um" on the command
>> diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
>> index 97fe293..14f3ef1 100644
>> --- a/arch/sh/Kconfig
>> +++ b/arch/sh/Kconfig
>> @@ -57,7 +57,7 @@ config SUPERH
>>           <http://www.linux-sh.org/>.
>>
>>  config SUPERH32
>> -       def_bool ARCH = "sh"
>> +       def_bool "$ARCH" = "sh"
>>         select HAVE_KPROBES
>>         select HAVE_KRETPROBES
>>         select HAVE_IOREMAP_PROT if MMU && !X2TLB
>> @@ -76,7 +76,7 @@ config SUPERH32
>>         select HAVE_CC_STACKPROTECTOR
>>
>>  config SUPERH64
>> -       def_bool ARCH = "sh64"
>> +       def_bool "$ARCH" = "sh64"
>>         select HAVE_EXIT_THREAD
>>         select KALLSYMS
>>
>> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
>> index 8767e45..86b852e 100644
>> --- a/arch/sparc/Kconfig
>> +++ b/arch/sparc/Kconfig
>> @@ -1,6 +1,6 @@
>>  config 64BIT
>> -       bool "64-bit kernel" if ARCH = "sparc"
>> -       default ARCH = "sparc64"
>> +       bool "64-bit kernel" if "$ARCH" = "sparc"
>> +       default "$ARCH" = "sparc64"
>>         help
>>           SPARC is a family of RISC microprocessors designed and marketed by
>>           Sun Microsystems, incorporated.  They are very widely found in Sun
>> diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
>> index ef9d403..acc2182 100644
>> --- a/arch/tile/Kconfig
>> +++ b/arch/tile/Kconfig
>> @@ -119,7 +119,7 @@ config HVC_TILE
>>  # Building with ARCH=tilegx (or ARCH=tile) implies using the
>>  # 64-bit TILE-Gx toolchain, so force CONFIG_TILEGX on.
>>  config TILEGX
>> -       def_bool ARCH != "tilepro"
>> +       def_bool "$ARCH" != "tilepro"
>>         select ARCH_SUPPORTS_ATOMIC_RMW
>>         select GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
>>         select HAVE_ARCH_JUMP_LABEL
>> diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
>> index c68add8..07f84c8 100644
>> --- a/arch/um/Kconfig.common
>> +++ b/arch/um/Kconfig.common
>> @@ -54,10 +54,6 @@ config HZ
>>         int
>>         default 100
>>
>> -config SUBARCH
>> -       string
>> -       option env="SUBARCH"
>> -
>>  config NR_CPUS
>>         int
>>         range 1 1
>> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
>> index 0fa71a7..986fb0a 100644
>> --- a/arch/x86/Kconfig
>> +++ b/arch/x86/Kconfig
>> @@ -1,8 +1,8 @@
>>  # SPDX-License-Identifier: GPL-2.0
>>  # Select 32 or 64 bit
>>  config 64BIT
>> -       bool "64-bit kernel" if ARCH = "x86"
>> -       default ARCH != "i386"
>> +       bool "64-bit kernel" if "$ARCH" = "x86"
>> +       default "$ARCH" != "i386"
>>         ---help---
>>           Say yes to build a 64-bit kernel - formerly known as x86_64
>>           Say no to build a 32-bit kernel - formerly known as i386
>> diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
>> index 13ed827..d355413 100644
>> --- a/arch/x86/um/Kconfig
>> +++ b/arch/x86/um/Kconfig
>> @@ -16,8 +16,8 @@ config UML_X86
>>         select GENERIC_FIND_FIRST_BIT
>>
>>  config 64BIT
>> -       bool "64-bit kernel" if SUBARCH = "x86"
>> -       default SUBARCH != "i386"
>> +       bool "64-bit kernel" if $SUBARCH = "x86"
>> +       default $SUBARCH != "i386"
>>
>>  config X86_32
>>         def_bool !64BIT
>> diff --git a/init/Kconfig b/init/Kconfig
>> index df18492..b4814e6 100644
>> --- a/init/Kconfig
>> +++ b/init/Kconfig
>> @@ -1,11 +1,3 @@
>> -config ARCH
>> -       string
>> -       option env="ARCH"
>> -
>> -config KERNELVERSION
>> -       string
>> -       option env="KERNELVERSION"
>> -
>>  config DEFCONFIG_LIST
>>         string
>>         depends on !UML
>> @@ -13,7 +5,7 @@ config DEFCONFIG_LIST
>>         default "/lib/modules/$UNAME_RELEASE/.config"
>>         default "/etc/kernel-config"
>>         default "/boot/config-$UNAME_RELEASE"
>> -       default "$ARCH_DEFCONFIG"
>> +       default ARCH_DEFCONFIG
>>         default "arch/$ARCH/defconfig"
>>
>>  config CONSTRUCTORS
>> diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
>> index df26c7b..98c2014 100644
>> --- a/scripts/kconfig/confdata.c
>> +++ b/scripts/kconfig/confdata.c
>> @@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
>>         return name ? name : "include/config/auto.conf";
>>  }
>>
>> -static char *conf_expand_value(const char *in)
>> -{
>> -       struct symbol *sym;
>> -       const char *src;
>> -       static char res_value[SYMBOL_MAXLENGTH];
>> -       char *dst, name[SYMBOL_MAXLENGTH];
>> -
>> -       res_value[0] = 0;
>> -       dst = name;
>> -       while ((src = strchr(in, '$'))) {
>> -               strncat(res_value, in, src - in);
>> -               src++;
>> -               dst = name;
>> -               while (isalnum(*src) || *src == '_')
>> -                       *dst++ = *src++;
>> -               *dst = 0;
>> -               sym = sym_lookup(name, 0);
>> -               sym_calc_value(sym);
>> -               strcat(res_value, sym_get_string_value(sym));
>> -               in = src;
>> -       }
>> -       strcat(res_value, in);
>> -
>> -       return res_value;
>> -}
>> -
>>  char *conf_get_default_confname(void)
>>  {
>>         struct stat buf;
>>         static char fullname[PATH_MAX+1];
>>         char *env, *name;
>>
>> -       name = conf_expand_value(conf_defname);
>> +       name = expand_string_value(conf_defname);
>>         env = getenv(SRCTREE);
>>         if (env) {
>>                 sprintf(fullname, "%s/%s", env, name);
>> @@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
>>                         if (expr_calc_value(prop->visible.expr) == no ||
>>                             prop->expr->type != E_SYMBOL)
>>                                 continue;
>> -                       name = conf_expand_value(prop->expr->left.sym->name);
>> +                       sym_calc_value(prop->expr->left.sym);
>> +                       name = sym_get_string_value(prop->expr->left.sym);
>>                         in = zconf_fopen(name);
>>                         if (in) {
>>                                 conf_message(_("using defaults found in %s"),
>> diff --git a/scripts/kconfig/env.c b/scripts/kconfig/env.c
>> new file mode 100644
>> index 0000000..9702f5c
>> --- /dev/null
>> +++ b/scripts/kconfig/env.c
>> @@ -0,0 +1,95 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +//
>> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
>> +
>> +static LIST_HEAD(env_list);
>> +
>> +struct env {
>> +       char *name;
>> +       char *value;
>> +       struct list_head node;
>> +};
>> +
>> +static struct env *env_list_lookup(const char *name)
>> +{
>> +       struct env *e;
>> +
>> +       list_for_each_entry(e, &env_list, node) {
>> +               if (!strcmp(name, e->name))
>> +                       return e;
>> +       }
>> +
>> +       return NULL;
>> +}
>
> This function might be unnecessary -- see below.
>
>> +
>> +static void env_list_add(const char *name, const char *value)
>> +{
>> +       struct env *e;
>> +
>> +       e = xmalloc(sizeof(*e));
>> +       e->name = xstrdup(name);
>> +       e->value = xstrdup(value);
>> +
>> +       list_add_tail(&e->node, &env_list);
>> +}
>> +
>> +static void env_list_del(struct env *e)
>> +{
>> +       list_del(&e->node);
>> +       free(e->name);
>> +       free(e->value);
>> +       free(e);
>> +}
>> +
>> +/* the returned pointer must be freed when done */
>> +static char *env_expand(const char *name)
>
> This function is basically just getenv() with some dependency
> housekeeping added. A name like env_get() or env_lookup() would be
> clearer I think.
>
>> +{
>> +       struct env *e;
>> +       const char *value;
>> +
>
>> +       e = env_list_lookup(name);
>> +       if (e)
>> +               return xstrdup(e->value);
>
> Not sure this bit is needed. It is always safe to get the value from
> getenv(). See below.
>
>> +
>> +       value = getenv(name);
>> +       if (!value) {
>> +               fprintf(stderr, "environment variable \"%s\" undefined\n", name);
>> +               value = "";
>> +       }
>> +
>> +       /*
>> +        * we need to remember all referenced environments.
>
> s/environments/environment variables/
>
>> +        * They will be written out to include/config/auto.conf.cmd
>> +        */
>> +       env_list_add(name, value);
>
> AFAICS, we only need the following functionality:
>
>     1. Record referenced environment variables along with their value
>
>     2. Go through all the recorded environment variables and write
> dependency information
>
> For (1), I think it would be better to simply have env_list_add() (or
> some other suitable name) add the variable to the list if it isn't
> already there (like a kind of set). It is always safe to get the value
> of the environment variable from getenv(), and that seems less
> confusing.

I think this is a cached value
for frequently referenced environment variable
such as $(CC), $(srctree).


>> +
>> +       return xstrdup(value);
>
> The returned string is never modified, and the result from getenv()
> never gets stale, so I think the strdup() is unnecessary. The function
> could return a const char * to avoid accidental modification.


The expanded text is always freed when done.
This simplifies the implementation.


If X is defined as

   X = $(shell echo ABC)

The result of $(X) is an allocated string, which must be freed later.

If X is an environment variable, we could handle it as a read-only string,
but we are not allowed to free it.

If we do not do strdup(), we need to remember where it came from.



-- 
Best Regards
Masahiro Yamada