[PATCH v4 00/31] kconfig: move compiler capability tests to Kconfig

[PATCH v4 00/31] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

[Introduction]

The motivation of this work is to move the compiler option tests to
Kconfig from Makefile.  A number of kernel features require the
compiler support.  Enabling such features blindly in Kconfig ends up
with a lot of nasty build-time testing in Makefiles.  If a chosen
feature turns out unsupported by the compiler, what the build system
can do is either to disable it (silently!) or to forcibly break the
build, despite Kconfig has let the user to enable it.  By moving the
compiler capability tests to Kconfig, Kconfig entries will be visible
only when they are available.

[Major Changes in V4]

 - In V4, I slightly change the syntax of a function call.
   I chose grammatical consistency and simplicity.
   Anyway, Kconfig is deviating from Make behavior already.

   In v3, a function call looked like this:

      $(func-name arg1,arg2,arg3)

   In v4, a function is invoked like follows:

      $(func-name,arg1,arg2,arg3)

   The difference is that the function name and the first argument
   are separated by a comma.

 - V3 supported single-letter variable like $X.
   V4 dropped it because we do not need multiple ways to do the
   same thing.
   You must always enclose a variable name like $(X).

 - Support lazy argument expansion.  This is necessary for implementing
   'if', 'and', 'or' functions as in Make.

 - Add more built-in functions:
   'if', 'error', 'filename', 'lineno'

 - Error out if a recursive variable references itself eventually.
   For example,  X = $(X)
   ends up with a circular expansion.  It must be terminated
   since the expansion would continue eternally.

 - Update Documentation and unit-tests, accordingly.

[Major Changes in V3]

This version looks more like Make.

  - Use = operator instead of 'macro' keyword
    to define a user-defined function.

  - 'Recursively expanded variable' is implemented as a side-effect.
     A variable is a function with zero argument.

  - Support simply expanded variable which is defined by := operator

  - Support += operator.
    Probably, this feature will be useful to accumulate compiler flags.
    At least, Clang needs some prerequisite flags such as triplet
    to test other compiler flags.

  - Support $(info ...) and $(warning ...) built-in functions,
    which were useful while I was debugging this.

  - Add documentation

  - Add unit tests

  - Collect helpers to scripts/Kconfig.include

[Old Versions]

V3:  https://lkml.org/lkml/2018/4/13/37
V2:  https://lkml.org/lkml/2018/2/16/610
V1:  https://lkml.org/lkml/2018/2/16/610
RFC: https://lkml.org/lkml/2018/2/8/429



Masahiro Yamada (31):
  kbuild: remove kbuild cache
  kbuild: remove CONFIG_CROSS_COMPILE support
  kconfig: reference environment variables directly and remove 'option
    env='
  kconfig: remove string expansion in file_lookup()
  kconfig: remove string expansion for mainmenu after yyparse()
  kconfig: remove sym_expand_string_value()
  kconfig: add built-in function support
  kconfig: add 'shell' built-in function
  kconfig: replace $(UNAME_RELEASE) with function call
  kconfig: begin PARAM state only when seeing a command keyword
  kconfig: support user-defined function and recursively expanded
    variable
  kconfig: support simply expanded variable
  kconfig: support append assignment operator
  kconfig: expand lefthand side of assignment statement
  kconfig: add 'info', 'warning', and 'error' built-in functions
  kconfig: add 'if' built-in function
  kconfig: add 'filename' and 'lineno' built-in variables
  kconfig: error out if a recursive variable references itself
  Documentation: kconfig: document a new Kconfig macro language
  kconfig: test: add Kconfig macro language tests
  kconfig: show compiler version text in the top comment
  kconfig: add basic helper macros to scripts/Kconfig.include
  stack-protector: test compiler capability in Kconfig and drop AUTO
    mode
  kconfig: add CC_IS_GCC and GCC_VERSION
  kconfig: add CC_IS_CLANG and CLANG_VERSION
  gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT
  kcov: test compiler capability in Kconfig and correct dependency
  gcc-plugins: move GCC version check for PowerPC to Kconfig
  gcc-plugins: test plugin support in Kconfig and clean up Makefile
  gcc-plugins: allow to enable GCC_PLUGINS for COMPILE_TEST
  arm64: move GCC version check for ARCH_SUPPORTS_INT128 to Kconfig

 Documentation/kbuild/kconfig-language.txt          |   8 -
 Documentation/kbuild/kconfig-macro-language.txt    | 252 ++++++++
 Kconfig                                            |  10 +-
 MAINTAINERS                                        |   3 +-
 Makefile                                           | 105 +---
 arch/Kconfig                                       |  49 +-
 arch/arm64/Kconfig                                 |   1 +
 arch/arm64/Makefile                                |   6 -
 arch/powerpc/Kconfig                               |   2 +-
 arch/sh/Kconfig                                    |   4 +-
 arch/sparc/Kconfig                                 |   4 +-
 arch/um/Kconfig.common                             |   4 -
 arch/x86/Kconfig                                   |  15 +-
 arch/x86/um/Kconfig                                |  10 +-
 init/Kconfig                                       |  40 +-
 kernel/gcov/Kconfig                                |  17 +-
 kernel/gcov/Makefile                               |   2 -
 lib/Kconfig.debug                                  |  11 +-
 scripts/Kbuild.include                             | 101 +---
 scripts/Kconfig.include                            |  27 +
 scripts/Makefile.gcc-plugins                       |  91 +--
 scripts/Makefile.kcov                              |  10 +-
 scripts/clang-version.sh                           |  18 +-
 scripts/gcc-plugins/Makefile                       |   1 +
 scripts/gcc-x86_32-has-stack-protector.sh          |   7 +-
 scripts/gcc-x86_64-has-stack-protector.sh          |   5 -
 scripts/kconfig/confdata.c                         |  33 +-
 scripts/kconfig/kconf_id.c                         |   1 -
 scripts/kconfig/lkc.h                              |   4 -
 scripts/kconfig/lkc_proto.h                        |  15 +-
 scripts/kconfig/menu.c                             |   3 -
 scripts/kconfig/preprocess.c                       | 651 +++++++++++++++++++++
 scripts/kconfig/symbol.c                           | 109 ----
 .../kconfig/tests/preprocess/builtin_func/Kconfig  |  29 +
 .../tests/preprocess/builtin_func/__init__.py      |   8 +
 .../tests/preprocess/builtin_func/expected_stderr  |   7 +
 .../tests/preprocess/builtin_func/expected_stdout  |   1 +
 .../tests/preprocess/circular_expansion/Kconfig    |   3 +
 .../preprocess/circular_expansion/__init__.py      |  10 +
 .../preprocess/circular_expansion/expected_stderr  |   1 +
 scripts/kconfig/tests/preprocess/escape/Kconfig    |  21 +
 .../kconfig/tests/preprocess/escape/__init__.py    |   7 +
 .../tests/preprocess/escape/expected_stderr        |   5 +
 scripts/kconfig/tests/preprocess/variable/Kconfig  |  48 ++
 .../kconfig/tests/preprocess/variable/__init__.py  |   7 +
 .../tests/preprocess/variable/expected_stderr      |   9 +
 scripts/kconfig/util.c                             |  22 +-
 scripts/kconfig/zconf.l                            |  95 ++-
 scripts/kconfig/zconf.y                            |  46 +-
 49 files changed, 1366 insertions(+), 572 deletions(-)
 create mode 100644 Documentation/kbuild/kconfig-macro-language.txt
 create mode 100644 scripts/Kconfig.include
 create mode 100644 scripts/kconfig/preprocess.c
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/expected_stderr
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/expected_stdout
 create mode 100644 scripts/kconfig/tests/preprocess/circular_expansion/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/circular_expansion/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/circular_expansion/expected_stderr
 create mode 100644 scripts/kconfig/tests/preprocess/escape/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/escape/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/escape/expected_stderr
 create mode 100644 scripts/kconfig/tests/preprocess/variable/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/variable/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/variable/expected_stderr

-- 
2.7.4

[PATCH v4 01/31] kbuild: remove kbuild cache

[Masahiro Yamada]

The kbuild cache was introduced to remember the result of shell
commands, some of which are expensive to compute, such as
$(call cc-option,...).

However, this turned out not so clever as I had first expected.
Actually, it is problematic.  For example, "$(CC) -print-file-name"
is cached.  If the compiler is updated, the stale search path causes
build error, which is difficult to figure out.  Another problem
scenario is cache files could be touched while install targets are
running under the root permission.  We can patch them if desired,
but the build infrastructure is getting uglier and uglier.

Now, we are going to move compiler flag tests to the configuration
phase.  If this is completed, the result of compiler tests will be
naturally cached in the .config file.  We will not have performance
issues of incremental building since this testing only happens at
Kconfig time.

To start this work with a cleaner code base, remove the kbuild
cache first.

Revert the following commits:
Commit 9a234a2e3843 ("kbuild: create directory for make cache only when necessary")
Commit e17c400ae194 ("kbuild: shrink .cache.mk when it exceeds 1000 lines")
Commit 4e56207130ed ("kbuild: Cache a few more calls to the compiler")
Commit 3298b690b21c ("kbuild: Add a cache for generated variables")

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
---

Changes in v4: None
Changes in v3: None
Changes in v2: None

 Makefile               |   5 +--
 scripts/Kbuild.include | 101 +++++++------------------------------------------
 2 files changed, 16 insertions(+), 90 deletions(-)

diff --git a/Makefile b/Makefile
index ba3106b..e78f73f 100644
--- a/Makefile
+++ b/Makefile
@@ -501,7 +501,7 @@ RETPOLINE_CFLAGS := $(call cc-option,$(RETPOLINE_CFLAGS_GCC),$(call cc-option,$(
 export RETPOLINE_CFLAGS
 
 # check for 'asm goto'
-ifeq ($(call shell-cached,$(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC) $(KBUILD_CFLAGS)), y)
+ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC) $(KBUILD_CFLAGS)), y)
   CC_HAVE_ASM_GOTO := 1
   KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
   KBUILD_AFLAGS += -DCC_HAVE_ASM_GOTO
@@ -804,7 +804,7 @@ KBUILD_CFLAGS	+= $(call cc-option,-fdata-sections,)
 endif
 
 # arch Makefile may override CC so keep this after arch Makefile is included
-NOSTDINC_FLAGS += -nostdinc -isystem $(call shell-cached,$(CC) -print-file-name=include)
+NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)
 CHECKFLAGS     += $(NOSTDINC_FLAGS)
 
 # warn about C99 declaration after statement
@@ -1622,7 +1622,6 @@ clean: $(clean-dirs)
 		-o -name '*.asn1.[ch]' \
 		-o -name '*.symtypes' -o -name 'modules.order' \
 		-o -name modules.builtin -o -name '.tmp_*.o.*' \
-		-o -name .cache.mk \
 		-o -name '*.c.[012]*.*' \
 		-o -name '*.ll' \
 		-o -name '*.gcno' \) -type f -print | xargs rm -f
diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index 50cee53..925a851 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -8,8 +8,6 @@ squote  := '
 empty   :=
 space   := $(empty) $(empty)
 space_escape := _-_SPACE_-_
-right_paren := )
-left_paren := (
 pound := \#
 
 ###
@@ -83,71 +81,6 @@ cc-cross-prefix =  \
 			echo $(c);                                    \
 		fi)))
 
-# Tools for caching Makefile variables that are "expensive" to compute.
-#
-# Here we want to help deal with variables that take a long time to compute
-# by making it easy to store these variables in a cache.
-#
-# The canonical example here is testing for compiler flags.  On a simple system
-# each call to the compiler takes 10 ms, but on a system with a compiler that's
-# called through various wrappers it can take upwards of 100 ms.  If we have
-# 100 calls to the compiler this can take 1 second (on a simple system) or 10
-# seconds (on a complicated system).
-#
-# The "cache" will be in Makefile syntax and can be directly included.
-# Any time we try to reference a variable that's not in the cache we'll
-# calculate it and store it in the cache for next time.
-
-# Include values from last time
-make-cache := $(if $(KBUILD_EXTMOD),$(KBUILD_EXTMOD)/,$(if $(obj),$(obj)/)).cache.mk
-$(make-cache): ;
--include $(make-cache)
-
-cached-data := $(filter __cached_%, $(.VARIABLES))
-
-# If cache exceeds 1000 lines, shrink it down to 500.
-ifneq ($(word 1000,$(cached-data)),)
-$(shell tail -n 500 $(make-cache) > $(make-cache).tmp; \
-	mv $(make-cache).tmp $(make-cache))
-endif
-
-create-cache-dir := $(if $(KBUILD_SRC),$(if $(cache-data),,1))
-
-# Usage: $(call __sanitize-opt,Hello=Hola$(comma)Goodbye Adios)
-#
-# Convert all '$', ')', '(', '\', '=', ' ', ',', ':' to '_'
-__sanitize-opt = $(subst $$,_,$(subst $(right_paren),_,$(subst $(left_paren),_,$(subst \,_,$(subst =,_,$(subst $(space),_,$(subst $(comma),_,$(subst :,_,$(1)))))))))
-
-# Usage:   $(call shell-cached,shell_command)
-# Example: $(call shell-cached,md5sum /usr/bin/gcc)
-#
-# If we've already seen a call to this exact shell command (even in a
-# previous invocation of make!) we'll return the value.  If not, we'll
-# compute it and store the result for future runs.
-#
-# This is a bit of voodoo, but basic explanation is that if the variable
-# was undefined then we'll evaluate the shell command and store the result
-# into the variable.  We'll then store that value in the cache and finally
-# output the value.
-#
-# NOTE: The $$(2) here isn't actually a parameter to __run-and-store.  We
-# happen to know that the caller will have their shell command in $(2) so the
-# result of "call"ing this will produce a reference to that $(2).  The reason
-# for this strangeness is to avoid an extra level of eval (and escaping) of
-# $(2).
-define __run-and-store
-ifeq ($(origin $(1)),undefined)
-  $$(eval $(1) := $$(shell $$(2)))
-ifeq ($(create-cache-dir),1)
-  $$(shell mkdir -p $(dir $(make-cache)))
-  $$(eval create-cache-dir :=)
-endif
-  $$(shell echo '$(1) := $$($(1))' >> $(make-cache))
-endif
-endef
-__shell-cached = $(eval $(call __run-and-store,$(1)))$($(1))
-shell-cached = $(call __shell-cached,__cached_$(call __sanitize-opt,$(1)),$(1))
-
 # output directory for tests below
 TMPOUT := $(if $(KBUILD_EXTMOD),$(firstword $(KBUILD_EXTMOD))/)
 
@@ -155,36 +88,30 @@ TMPOUT := $(if $(KBUILD_EXTMOD),$(firstword $(KBUILD_EXTMOD))/)
 # Usage: option = $(call try-run, $(CC)...-o "$$TMP",option-ok,otherwise)
 # Exit code chooses option. "$$TMP" serves as a temporary file and is
 # automatically cleaned up.
-__try-run = set -e;			\
+try-run = $(shell set -e;		\
 	TMP="$(TMPOUT).$$$$.tmp";	\
 	TMPO="$(TMPOUT).$$$$.o";	\
 	if ($(1)) >/dev/null 2>&1;	\
 	then echo "$(2)";		\
 	else echo "$(3)";		\
 	fi;				\
-	rm -f "$$TMP" "$$TMPO"
-
-try-run = $(shell $(__try-run))
-
-# try-run-cached
-# This works like try-run, but the result is cached.
-try-run-cached = $(call shell-cached,$(__try-run))
+	rm -f "$$TMP" "$$TMPO")
 
 # as-option
 # Usage: cflags-y += $(call as-option,-Wa$(comma)-isa=foo,)
 
-as-option = $(call try-run-cached,\
+as-option = $(call try-run,\
 	$(CC) $(KBUILD_CFLAGS) $(1) -c -x assembler /dev/null -o "$$TMP",$(1),$(2))
 
 # as-instr
 # Usage: cflags-y += $(call as-instr,instr,option1,option2)
 
-as-instr = $(call try-run-cached,\
+as-instr = $(call try-run,\
 	printf "%b\n" "$(1)" | $(CC) $(KBUILD_AFLAGS) -c -x assembler -o "$$TMP" -,$(2),$(3))
 
 # __cc-option
 # Usage: MY_CFLAGS += $(call __cc-option,$(CC),$(MY_CFLAGS),-march=winchip-c6,-march=i586)
-__cc-option = $(call try-run-cached,\
+__cc-option = $(call try-run,\
 	$(1) -Werror $(2) $(3) -c -x c /dev/null -o "$$TMP",$(3),$(4))
 
 # Do not attempt to build with gcc plugins during cc-option tests.
@@ -204,23 +131,23 @@ hostcc-option = $(call __cc-option, $(HOSTCC),\
 
 # cc-option-yn
 # Usage: flag := $(call cc-option-yn,-march=winchip-c6)
-cc-option-yn = $(call try-run-cached,\
+cc-option-yn = $(call try-run,\
 	$(CC) -Werror $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) $(1) -c -x c /dev/null -o "$$TMP",y,n)
 
 # cc-disable-warning
 # Usage: cflags-y += $(call cc-disable-warning,unused-but-set-variable)
-cc-disable-warning = $(call try-run-cached,\
+cc-disable-warning = $(call try-run,\
 	$(CC) -Werror $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) -W$(strip $(1)) -c -x c /dev/null -o "$$TMP",-Wno-$(strip $(1)))
 
 # cc-name
 # Expands to either gcc or clang
-cc-name = $(call shell-cached,$(CC) -v 2>&1 | grep -q "clang version" && echo clang || echo gcc)
+cc-name = $(shell $(CC) -v 2>&1 | grep -q "clang version" && echo clang || echo gcc)
 
 # cc-version
-cc-version = $(call shell-cached,$(CONFIG_SHELL) $(srctree)/scripts/gcc-version.sh $(CC))
+cc-version = $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-version.sh $(CC))
 
 # cc-fullversion
-cc-fullversion = $(call shell-cached,$(CONFIG_SHELL) \
+cc-fullversion = $(shell $(CONFIG_SHELL) \
 	$(srctree)/scripts/gcc-version.sh -p $(CC))
 
 # cc-ifversion
@@ -233,21 +160,21 @@ cc-if-fullversion = $(shell [ $(cc-fullversion) $(1) $(2) ] && echo $(3) || echo
 
 # cc-ldoption
 # Usage: ldflags += $(call cc-ldoption, -Wl$(comma)--hash-style=both)
-cc-ldoption = $(call try-run-cached,\
+cc-ldoption = $(call try-run,\
 	$(CC) $(1) $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) -nostdlib -x c /dev/null -o "$$TMP",$(1),$(2))
 
 # ld-option
 # Usage: LDFLAGS += $(call ld-option, -X)
-ld-option = $(call try-run-cached, $(LD) $(LDFLAGS) $(1) -v,$(1),$(2))
+ld-option = $(call try-run, $(LD) $(LDFLAGS) $(1) -v,$(1),$(2))
 
 # ar-option
 # Usage: KBUILD_ARFLAGS := $(call ar-option,D)
 # Important: no spaces around options
-ar-option = $(call try-run-cached, $(AR) rc$(1) "$$TMP",$(1),$(2))
+ar-option = $(call try-run, $(AR) rc$(1) "$$TMP",$(1),$(2))
 
 # ld-version
 # Note this is mainly for HJ Lu's 3 number binutil versions
-ld-version = $(call shell-cached,$(LD) --version | $(srctree)/scripts/ld-version.sh)
+ld-version = $(shell $(LD) --version | $(srctree)/scripts/ld-version.sh)
 
 # ld-ifversion
 # Usage:  $(call ld-ifversion, -ge, 22252, y)
-- 
2.7.4

[PATCH v4 02/31] kbuild: remove CONFIG_CROSS_COMPILE support

[Masahiro Yamada]

Kbuild provides a couple of ways to specify CROSS_COMPILE:

[1] Command line
[2] Environment
[3] arch/*/Makefile (only some architectures)
[4] CONFIG_CROSS_COMPILE

[4] is problematic for the compiler capability tests in Kconfig.
CONFIG_CROSS_COMPILE allows users to change the compiler prefix from
'make menuconfig', etc.  It means, the compiler options would have
to be all re-calculated everytime CONFIG_CROSS_COMPILE is changed.

To avoid complexity and performance issues, I'd like to evaluate
the shell commands statically, i.e. only parsing Kconfig files.

I guess the majority is [1] or [2].  Currently, there are only
5 defconfig files that specify CONFIG_CROSS_COMPILE.
  arch/arm/configs/lpc18xx_defconfig
  arch/hexagon/configs/comet_defconfig
  arch/nds32/configs/defconfig
  arch/openrisc/configs/or1ksim_defconfig
  arch/openrisc/configs/simple_smp_defconfig

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
---

Changes in v4: None
Changes in v3: None
Changes in v2: None

 Makefile     | 3 ---
 init/Kconfig | 9 ---------
 2 files changed, 12 deletions(-)

diff --git a/Makefile b/Makefile
index e78f73f..01b2211 100644
--- a/Makefile
+++ b/Makefile
@@ -316,12 +316,9 @@ SUBARCH := $(shell uname -m | sed -e s/i.86/x86/ -e s/x86_64/x86/ \
 # CROSS_COMPILE can be set on the command line
 # make CROSS_COMPILE=ia64-linux-
 # Alternatively CROSS_COMPILE can be set in the environment.
-# A third alternative is to store a setting in .config so that plain
-# "make" in the configured kernel build directory always uses that.
 # Default value for CROSS_COMPILE is not to prefix executables
 # Note: Some architectures assign CROSS_COMPILE in their arch/*/Makefile
 ARCH		?= $(SUBARCH)
-CROSS_COMPILE	?= $(CONFIG_CROSS_COMPILE:"%"=%)
 
 # Architecture as present in compile.h
 UTS_MACHINE 	:= $(ARCH)
diff --git a/init/Kconfig b/init/Kconfig
index f013afc..4537962 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -54,15 +54,6 @@ config INIT_ENV_ARG_LIMIT
 	  Maximum of each of the number of arguments and environment
 	  variables passed to init from the kernel command line.
 
-
-config CROSS_COMPILE
-	string "Cross-compiler tool prefix"
-	help
-	  Same as running 'make CROSS_COMPILE=prefix-' but stored for
-	  default make runs in this kernel build directory.  You don't
-	  need to set this unless you want the configured kernel build
-	  directory to select the cross-compiler automatically.
-
 config COMPILE_TEST
 	bool "Compile also drivers which will not load"
 	depends on !UML
-- 
2.7.4

[PATCH v4 03/31] kconfig: reference environment variables directly and remove 'option env='

[Masahiro Yamada]

To get access to environment variables, Kconfig needs to define a
symbol using "option env=" syntax.  It is tedious to add a symbol entry
for each environment variable given that we need to define much more
such as 'CC', 'AS', 'srctree' etc. to evaluate the compiler capability
in Kconfig.

Adding '$' for symbol references is grammatically inconsistent.
Looking at the code, the symbols prefixed with 'S' are expanded by:
 - conf_expand_value()
   This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
 - sym_expand_string_value()
   This is used to expand strings in 'source' and 'mainmenu'

All of them are fixed values independent of user configuration.  So,
they can be changed into the direct expansion instead of symbols.

This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.

sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
should be replaced with an environment variable.

ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
without '$' prefix.

The new syntax is addicted by Make.  The variable reference needs
parentheses, like $(FOO), but you can omit them for single-letter
variables, like $F.  Yet, in Makefiles, people tend to use the
parenthetical form for consistency / clarification.

At this moment, only the environment variable is supported, but I will
extend the concept of 'variable' later on.

The variables are expanded in the lexer so we can simplify the token
handling on the parser side.

For example, the following code works.

[Example code]

  config MY_TOOLCHAIN_LIST
          string
          default "My tools: CC=$(CC), AS=$(AS), CPP=$(CPP)"

[Result]

  $ make -s alldefconfig && tail -n 1 .config
  CONFIG_MY_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
---

Changes in v4:
  - Enclose ARCH in conf_defname
  - Drop single-letter support

Changes in v3:
  - Reimplement
  - Variable reference need parentheses except single-letter variable

Changes in v2:
  - Move the string expansion to the lexer phase.
  - Split environment helpers to env.c

 Documentation/kbuild/kconfig-language.txt |   8 -
 Kconfig                                   |   8 +-
 Makefile                                  |   3 +-
 arch/sh/Kconfig                           |   4 +-
 arch/sparc/Kconfig                        |   4 +-
 arch/um/Kconfig.common                    |   4 -
 arch/x86/Kconfig                          |   4 +-
 arch/x86/um/Kconfig                       |   6 +-
 init/Kconfig                              |  16 +-
 scripts/kconfig/confdata.c                |  33 +---
 scripts/kconfig/kconf_id.c                |   1 -
 scripts/kconfig/lkc.h                     |   4 -
 scripts/kconfig/lkc_proto.h               |   6 +
 scripts/kconfig/menu.c                    |   3 -
 scripts/kconfig/preprocess.c              | 247 ++++++++++++++++++++++++++++++
 scripts/kconfig/symbol.c                  |  56 -------
 scripts/kconfig/util.c                    |  18 +--
 scripts/kconfig/zconf.l                   |  67 +++++++-
 scripts/kconfig/zconf.y                   |   2 +-
 19 files changed, 340 insertions(+), 154 deletions(-)
 create mode 100644 scripts/kconfig/preprocess.c

diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
index f5b9493..0e966e8 100644
--- a/Documentation/kbuild/kconfig-language.txt
+++ b/Documentation/kbuild/kconfig-language.txt
@@ -198,14 +198,6 @@ applicable everywhere (see syntax).
     enables the third modular state for all config symbols.
     At most one symbol may have the "modules" option set.
 
-  - "env"=<value>
-    This imports the environment variable into Kconfig. It behaves like
-    a default, except that the value comes from the environment, this
-    also means that the behaviour when mixing it with normal defaults is
-    undefined at this point. The symbol is currently not exported back
-    to the build environment (if this is desired, it can be done via
-    another symbol).
-
   - "allnoconfig_y"
     This declares the symbol as one that should have the value y when
     using "allnoconfig". Used for symbols that hide other symbols.
diff --git a/Kconfig b/Kconfig
index 8c4c1cb..4af1b42 100644
--- a/Kconfig
+++ b/Kconfig
@@ -3,10 +3,6 @@
 # For a description of the syntax of this configuration file,
 # see Documentation/kbuild/kconfig-language.txt.
 #
-mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
+mainmenu "Linux/$(ARCH) $(KERNELVERSION) Kernel Configuration"
 
-config SRCARCH
-	string
-	option env="SRCARCH"
-
-source "arch/$SRCARCH/Kconfig"
+source "arch/$(SRCARCH)/Kconfig"
diff --git a/Makefile b/Makefile
index 01b2211..c6278e6 100644
--- a/Makefile
+++ b/Makefile
@@ -284,7 +284,8 @@ include scripts/Kbuild.include
 # Read KERNELRELEASE from include/config/kernel.release (if it exists)
 KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
 KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
-export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
+UNAME_RELEASE := $(shell uname --release)
+export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
 
 # SUBARCH tells the usermode build what the underlying arch is.  That is set
 # first, and if a usermode build is happening, the "ARCH=um" on the command
diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
index 1851eae..c8400e3 100644
--- a/arch/sh/Kconfig
+++ b/arch/sh/Kconfig
@@ -58,7 +58,7 @@ config SUPERH
 	  <http://www.linux-sh.org/>.
 
 config SUPERH32
-	def_bool ARCH = "sh"
+	def_bool "$(ARCH)" = "sh"
 	select HAVE_KPROBES
 	select HAVE_KRETPROBES
 	select HAVE_IOREMAP_PROT if MMU && !X2TLB
@@ -77,7 +77,7 @@ config SUPERH32
 	select HAVE_CC_STACKPROTECTOR
 
 config SUPERH64
-	def_bool ARCH = "sh64"
+	def_bool "$(ARCH)" = "sh64"
 	select HAVE_EXIT_THREAD
 	select KALLSYMS
 
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 8767e45..df7410c 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -1,6 +1,6 @@
 config 64BIT
-	bool "64-bit kernel" if ARCH = "sparc"
-	default ARCH = "sparc64"
+	bool "64-bit kernel" if "$(ARCH)" = "sparc"
+	default "$(ARCH)" = "sparc64"
 	help
 	  SPARC is a family of RISC microprocessors designed and marketed by
 	  Sun Microsystems, incorporated.  They are very widely found in Sun
diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
index c68add8..07f84c8 100644
--- a/arch/um/Kconfig.common
+++ b/arch/um/Kconfig.common
@@ -54,10 +54,6 @@ config HZ
 	int
 	default 100
 
-config SUBARCH
-	string
-	option env="SUBARCH"
-
 config NR_CPUS
 	int
 	range 1 1
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index c07f492..2236505 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1,8 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
 # Select 32 or 64 bit
 config 64BIT
-	bool "64-bit kernel" if ARCH = "x86"
-	default ARCH != "i386"
+	bool "64-bit kernel" if "$(ARCH)" = "x86"
+	default "$(ARCH)" != "i386"
 	---help---
 	  Say yes to build a 64-bit kernel - formerly known as x86_64
 	  Say no to build a 32-bit kernel - formerly known as i386
diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
index 13ed827..6a15c4d 100644
--- a/arch/x86/um/Kconfig
+++ b/arch/x86/um/Kconfig
@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
-mainmenu "User Mode Linux/$SUBARCH $KERNELVERSION Kernel Configuration"
+mainmenu "User Mode Linux/$(SUBARCH) $(KERNELVERSION) Kernel Configuration"
 
 source "arch/um/Kconfig.common"
 
@@ -16,8 +16,8 @@ config UML_X86
 	select GENERIC_FIND_FIRST_BIT
 
 config 64BIT
-	bool "64-bit kernel" if SUBARCH = "x86"
-	default SUBARCH != "i386"
+	bool "64-bit kernel" if "$(SUBARCH)" = "x86"
+	default "$(SUBARCH)" != "i386"
 
 config X86_32
 	def_bool !64BIT
diff --git a/init/Kconfig b/init/Kconfig
index 4537962..752816b 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1,20 +1,12 @@
-config ARCH
-	string
-	option env="ARCH"
-
-config KERNELVERSION
-	string
-	option env="KERNELVERSION"
-
 config DEFCONFIG_LIST
 	string
 	depends on !UML
 	option defconfig_list
-	default "/lib/modules/$UNAME_RELEASE/.config"
+	default "/lib/modules/$(UNAME_RELEASE)/.config"
 	default "/etc/kernel-config"
-	default "/boot/config-$UNAME_RELEASE"
-	default "$ARCH_DEFCONFIG"
-	default "arch/$ARCH/defconfig"
+	default "/boot/config-$(UNAME_RELEASE)"
+	default ARCH_DEFCONFIG
+	default "arch/$(ARCH)/defconfig"
 
 config CONSTRUCTORS
 	bool
diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
index df26c7b..f72587c 100644
--- a/scripts/kconfig/confdata.c
+++ b/scripts/kconfig/confdata.c
@@ -30,7 +30,7 @@ static void conf_message(const char *fmt, ...)
 static const char *conf_filename;
 static int conf_lineno, conf_warnings;
 
-const char conf_defname[] = "arch/$ARCH/defconfig";
+const char conf_defname[] = "arch/$(ARCH)/defconfig";
 
 static void conf_warning(const char *fmt, ...)
 {
@@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
 	return name ? name : "include/config/auto.conf";
 }
 
-static char *conf_expand_value(const char *in)
-{
-	struct symbol *sym;
-	const char *src;
-	static char res_value[SYMBOL_MAXLENGTH];
-	char *dst, name[SYMBOL_MAXLENGTH];
-
-	res_value[0] = 0;
-	dst = name;
-	while ((src = strchr(in, '$'))) {
-		strncat(res_value, in, src - in);
-		src++;
-		dst = name;
-		while (isalnum(*src) || *src == '_')
-			*dst++ = *src++;
-		*dst = 0;
-		sym = sym_lookup(name, 0);
-		sym_calc_value(sym);
-		strcat(res_value, sym_get_string_value(sym));
-		in = src;
-	}
-	strcat(res_value, in);
-
-	return res_value;
-}
-
 char *conf_get_default_confname(void)
 {
 	struct stat buf;
 	static char fullname[PATH_MAX+1];
 	char *env, *name;
 
-	name = conf_expand_value(conf_defname);
+	name = expand_string(conf_defname);
 	env = getenv(SRCTREE);
 	if (env) {
 		sprintf(fullname, "%s/%s", env, name);
@@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
 			if (expr_calc_value(prop->visible.expr) == no ||
 			    prop->expr->type != E_SYMBOL)
 				continue;
-			name = conf_expand_value(prop->expr->left.sym->name);
+			sym_calc_value(prop->expr->left.sym);
+			name = sym_get_string_value(prop->expr->left.sym);
 			in = zconf_fopen(name);
 			if (in) {
 				conf_message(_("using defaults found in %s"),
diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
index 3ea9c5f..b3e0ea0 100644
--- a/scripts/kconfig/kconf_id.c
+++ b/scripts/kconfig/kconf_id.c
@@ -32,7 +32,6 @@ static struct kconf_id kconf_id_array[] = {
 	{ "on",			T_ON,			TF_PARAM },
 	{ "modules",		T_OPT_MODULES,		TF_OPTION },
 	{ "defconfig_list",	T_OPT_DEFCONFIG_LIST,	TF_OPTION },
-	{ "env",		T_OPT_ENV,		TF_OPTION },
 	{ "allnoconfig_y",	T_OPT_ALLNOCONFIG_Y,	TF_OPTION },
 };
 
diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
index f4394af..553098a 100644
--- a/scripts/kconfig/lkc.h
+++ b/scripts/kconfig/lkc.h
@@ -58,7 +58,6 @@ enum conf_def_mode {
 
 #define T_OPT_MODULES		1
 #define T_OPT_DEFCONFIG_LIST	2
-#define T_OPT_ENV		3
 #define T_OPT_ALLNOCONFIG_Y	4
 
 struct kconf_id {
@@ -134,9 +133,6 @@ void str_printf(struct gstr *gs, const char *fmt, ...);
 const char *str_get(struct gstr *gs);
 
 /* symbol.c */
-extern struct expr *sym_env_list;
-
-void sym_init(void);
 void sym_clear_all_valid(void);
 struct symbol *sym_choice_default(struct symbol *sym);
 const char *sym_get_string_default(struct symbol *sym);
diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
index 9dc8abf..9f465fe 100644
--- a/scripts/kconfig/lkc_proto.h
+++ b/scripts/kconfig/lkc_proto.h
@@ -49,5 +49,11 @@ const char * sym_get_string_value(struct symbol *sym);
 
 const char * prop_get_type_name(enum prop_type type);
 
+/* preprocess.c */
+void env_write_dep(FILE *f, const char *auto_conf_name);
+char *expand_string(const char *in);
+char *expand_dollar(const char **str);
+char *expand_one_token(const char **str);
+
 /* expr.c */
 void expr_print(struct expr *e, void (*fn)(void *, struct symbol *, const char *), void *data, int prevtoken);
diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
index 5c5c137..8148305 100644
--- a/scripts/kconfig/menu.c
+++ b/scripts/kconfig/menu.c
@@ -214,9 +214,6 @@ void menu_add_option(int token, char *arg)
 			zconf_error("trying to redefine defconfig symbol");
 		sym_defconfig_list->flags |= SYMBOL_AUTO;
 		break;
-	case T_OPT_ENV:
-		prop_add_env(arg);
-		break;
 	case T_OPT_ALLNOCONFIG_Y:
 		current_entry->sym->flags |= SYMBOL_ALLNOCONFIG_Y;
 		break;
diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
new file mode 100644
index 0000000..1bf506c
--- /dev/null
+++ b/scripts/kconfig/preprocess.c
@@ -0,0 +1,247 @@
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
+
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "list.h"
+
+static void __attribute__((noreturn)) pperror(const char *format, ...)
+{
+	va_list ap;
+
+	fprintf(stderr, "%s:%d: ", current_file->name, yylineno);
+	va_start(ap, format);
+	vfprintf(stderr, format, ap);
+	va_end(ap);
+	fprintf(stderr, "\n");
+
+	exit(1);
+}
+
+/*
+ * Environment variables
+ */
+static LIST_HEAD(env_list);
+
+struct env {
+	char *name;
+	char *value;
+	struct list_head node;
+};
+
+static void env_add(const char *name, const char *value)
+{
+	struct env *e;
+
+	e = xmalloc(sizeof(*e));
+	e->name = xstrdup(name);
+	e->value = xstrdup(value);
+
+	list_add_tail(&e->node, &env_list);
+}
+
+static void env_del(struct env *e)
+{
+	list_del(&e->node);
+	free(e->name);
+	free(e->value);
+	free(e);
+}
+
+/* The returned pointer must be freed when done */
+static char *env_expand(const char *name)
+{
+	struct env *e;
+	const char *value;
+
+	list_for_each_entry(e, &env_list, node) {
+		if (!strcmp(name, e->name))
+			return xstrdup(e->value);
+	}
+
+	value = getenv(name);
+	if (!value)
+		value = "";
+
+	/*
+	 * We need to remember all referenced environments.
+	 * They will be written out to include/config/auto.conf.cmd
+	 */
+	env_add(name, value);
+
+	return xstrdup(value);
+}
+
+void env_write_dep(FILE *f, const char *autoconfig_name)
+{
+	struct env *e, *tmp;
+
+	list_for_each_entry_safe(e, tmp, &env_list, node) {
+		fprintf(f, "ifneq \"$(%s)\" \"%s\"\n", e->name, e->value);
+		fprintf(f, "%s: FORCE\n", autoconfig_name);
+		fprintf(f, "endif\n");
+		env_del(e);
+	}
+}
+
+static char *eval_clause(const char *in)
+{
+	char *res, *name;
+
+	/*
+	 * Returns an empty string because '$()' should be evaluated
+	 * to a null string.
+	 */
+	if (!*in)
+		return xstrdup("");
+
+	name = expand_string(in);
+
+	res = env_expand(name);
+	free(name);
+
+	return res;
+}
+
+/*
+ * Expand a string that follows '$'
+ *
+ * For example, if the input string is
+ *     ($(FOO)$($(BAR)))$(BAZ)
+ * this helper evaluates
+ *     $($(FOO)$($(BAR)))
+ * and returns the resulted string, then updates 'str' to point to the next
+ * character after the corresponding closing parenthesis, in this case, *str
+ * will be
+ *     $(BAR)
+ */
+char *expand_dollar(const char **str)
+{
+	const char *p = *str;
+	const char *q;
+	char *tmp, *out;
+	int nest = 0;
+
+	/* '$$' represents an escaped '$' */
+	if (*p == '$') {
+		*str = p + 1;
+		return xstrdup("$");
+	}
+
+	/*
+	 * Kconfig does not support single-letter variable as in $A
+	 * or curly braces as in ${CC}.
+	 */
+	if (*p != '(')
+		pperror("syntax error: '$' not followed by '('", p);
+
+	p++;
+	q = p;
+	while (*q) {
+		if (*q == '(') {
+			nest++;
+		} else if (*q == ')') {
+			if (nest-- == 0)
+				break;
+		}
+		q++;
+	}
+
+	if (!*q)
+		pperror("unterminated reference to '%s': missing ')'", p);
+
+	tmp = xmalloc(q - p + 1);
+	memcpy(tmp, p, q - p);
+	tmp[q - p] = 0;
+	*str = q + 1;
+	out = eval_clause(tmp);
+	free(tmp);
+
+	return out;
+}
+
+/*
+ * Expand variables in the given string.  Undefined variables
+ * expand to an empty string.
+ * The returned string must be freed when done.
+ */
+char *expand_string(const char *in)
+{
+	const char *prev_in, *p;
+	char *new, *out;
+	size_t outlen;
+
+	out = xmalloc(1);
+	*out = 0;
+
+	while ((p = strchr(in, '$'))) {
+		prev_in = in;
+		in = p + 1;
+		new = expand_dollar(&in);
+		outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
+		out = xrealloc(out, outlen);
+		strncat(out, prev_in, p - prev_in);
+		strcat(out, new);
+		free(new);
+	}
+
+	outlen = strlen(out) + strlen(in) + 1;
+	out = xrealloc(out, outlen);
+	strcat(out, in);
+
+	return out;
+}
+
+/*
+ * Expand variables in a token.  The parsing stops when a token separater
+ * (in most cases, it is a whitespace) is encountered.  'str' is updated to
+ * point to the next character.
+ *
+ * The returned string must be freed when done.
+ */
+char *expand_one_token(const char **str)
+{
+	const char *in, *prev_in, *p;
+	char *new, *out;
+	size_t outlen;
+
+	out = xmalloc(1);
+	*out = 0;
+
+	p = in = *str;
+
+	while (1) {
+		if (*p == '$') {
+			prev_in = in;
+			in = p + 1;
+			new = expand_dollar(&in);
+			outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
+			out = xrealloc(out, outlen);
+			strncat(out, prev_in, p - prev_in);
+			strcat(out, new);
+			free(new);
+			p = in;
+			continue;
+		}
+
+		/* Valid characters in a symbol (why '.' and '/' ?) */
+		if (isalnum(*p) || *p == '_' || *p == '-' || *p == '.' || *p == '/') {
+			p++;
+			continue;
+		}
+
+		break;
+	}
+
+	outlen = strlen(out) + (p - in) + 1;
+	out = xrealloc(out, outlen);
+	strncat(out, in, p - in);
+
+	*str = p;
+
+	return out;
+}
diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
index f0b2e3b..2460648 100644
--- a/scripts/kconfig/symbol.c
+++ b/scripts/kconfig/symbol.c
@@ -33,33 +33,6 @@ struct symbol *sym_defconfig_list;
 struct symbol *modules_sym;
 tristate modules_val;
 
-struct expr *sym_env_list;
-
-static void sym_add_default(struct symbol *sym, const char *def)
-{
-	struct property *prop = prop_alloc(P_DEFAULT, sym);
-
-	prop->expr = expr_alloc_symbol(sym_lookup(def, SYMBOL_CONST));
-}
-
-void sym_init(void)
-{
-	struct symbol *sym;
-	struct utsname uts;
-	static bool inited = false;
-
-	if (inited)
-		return;
-	inited = true;
-
-	uname(&uts);
-
-	sym = sym_lookup("UNAME_RELEASE", 0);
-	sym->type = S_STRING;
-	sym->flags |= SYMBOL_AUTO;
-	sym_add_default(sym, uts.release);
-}
-
 enum symbol_type sym_get_type(struct symbol *sym)
 {
 	enum symbol_type type = sym->type;
@@ -1401,32 +1374,3 @@ const char *prop_get_type_name(enum prop_type type)
 	}
 	return "unknown";
 }
-
-static void prop_add_env(const char *env)
-{
-	struct symbol *sym, *sym2;
-	struct property *prop;
-	char *p;
-
-	sym = current_entry->sym;
-	sym->flags |= SYMBOL_AUTO;
-	for_all_properties(sym, prop, P_ENV) {
-		sym2 = prop_get_symbol(prop);
-		if (strcmp(sym2->name, env))
-			menu_warn(current_entry, "redefining environment symbol from %s",
-				  sym2->name);
-		return;
-	}
-
-	prop = prop_alloc(P_ENV, sym);
-	prop->expr = expr_alloc_symbol(sym_lookup(env, SYMBOL_CONST));
-
-	sym_env_list = expr_alloc_one(E_LIST, sym_env_list);
-	sym_env_list->right.sym = sym;
-
-	p = getenv(env);
-	if (p)
-		sym_add_default(sym, p);
-	else
-		menu_warn(current_entry, "environment variable %s undefined", env);
-}
diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index c6f6e21..807147e 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -34,8 +34,6 @@ struct file *file_lookup(const char *name)
 /* write a dependency file as used by kbuild to track dependencies */
 int file_write_dep(const char *name)
 {
-	struct symbol *sym, *env_sym;
-	struct expr *e;
 	struct file *file;
 	FILE *out;
 
@@ -54,21 +52,7 @@ int file_write_dep(const char *name)
 	fprintf(out, "\n%s: \\\n"
 		     "\t$(deps_config)\n\n", conf_get_autoconfig_name());
 
-	expr_list_for_each_sym(sym_env_list, e, sym) {
-		struct property *prop;
-		const char *value;
-
-		prop = sym_get_env_prop(sym);
-		env_sym = prop_get_symbol(prop);
-		if (!env_sym)
-			continue;
-		value = getenv(env_sym->name);
-		if (!value)
-			value = "";
-		fprintf(out, "ifneq \"$(%s)\" \"%s\"\n", env_sym->name, value);
-		fprintf(out, "%s: FORCE\n", conf_get_autoconfig_name());
-		fprintf(out, "endif\n");
-	}
+	env_write_dep(out, conf_get_autoconfig_name());
 
 	fprintf(out, "\n$(deps_config): ;\n");
 	fclose(out);
diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
index 045093d..9dc5fe3 100644
--- a/scripts/kconfig/zconf.l
+++ b/scripts/kconfig/zconf.l
@@ -1,6 +1,5 @@
 %option nostdinit noyywrap never-interactive full ecs
 %option 8bit nodefault yylineno
-%option noinput
 %x COMMAND HELP STRING PARAM
 %{
 /*
@@ -35,6 +34,8 @@ struct buffer *current_buf;
 
 static int last_ts, first_ts;
 
+static char *expand_token(const char *in, size_t n);
+static void append_expanded_string(const char *in);
 static void zconf_endhelp(void);
 static void zconf_endfile(void);
 
@@ -147,6 +148,13 @@ n	[A-Za-z0-9_-]
 		yylval.string = text;
 		return T_WORD;
 	}
+	({n}|[/.$])+	{
+		/* this token includes at least one '$' */
+		yylval.string = expand_token(yytext, yyleng);
+		if (strlen(yylval.string))
+			return T_WORD;
+		free(yylval.string);
+	}
 	#.*	/* comment */
 	\\\n	;
 	[[:blank:]]+
@@ -157,12 +165,13 @@ n	[A-Za-z0-9_-]
 }
 
 <STRING>{
-	[^'"\\\n]+/\n	{
+	"$".*	append_expanded_string(yytext);
+	[^$'"\\\n]+/\n	{
 		append_string(yytext, yyleng);
 		yylval.string = text;
 		return T_WORD_QUOTE;
 	}
-	[^'"\\\n]+	{
+	[^$'"\\\n]+	{
 		append_string(yytext, yyleng);
 	}
 	\\.?/\n	{
@@ -249,6 +258,58 @@ n	[A-Za-z0-9_-]
 }
 
 %%
+static char *expand_token(const char *in, size_t n)
+{
+	char *out;
+	int c;
+	char c2;
+	const char *rest, *end;
+
+	new_string();
+	append_string(in, n);
+
+	/* get the whole the line because we do not know the end of token. */
+	while ((c = input()) != EOF) {
+		if (c == '\n') {
+			unput(c);
+			break;
+		}
+		c2 = c;
+		append_string(&c2, 1);
+	}
+
+	rest = text;
+	out = expand_one_token(&rest);
+
+	/* push back unused characters to the input stream */
+	end = rest + strlen(rest);
+	while (end > rest)
+		unput(*--end);
+
+	free(text);
+
+	return out;
+}
+
+static void append_expanded_string(const char *str)
+{
+	const char *end;
+	char *res;
+
+	str++;
+
+	res = expand_dollar(&str);
+
+	/* push back unused characters to the input stream */
+	end = str + strlen(str);
+	while (end > str)
+		unput(*--end);
+
+	append_string(res, strlen(res));
+
+	free(res);
+}
+
 void zconf_starthelp(void)
 {
 	new_string();
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index ad6305b..3a4a0fa 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -534,7 +534,6 @@ void conf_parse(const char *name)
 
 	zconf_initscan(name);
 
-	sym_init();
 	_menu_init();
 
 	if (getenv("ZCONF_DEBUG"))
@@ -780,3 +779,4 @@ void zconfdump(FILE *out)
 #include "expr.c"
 #include "symbol.c"
 #include "menu.c"
+#include "preprocess.c"
-- 
2.7.4

[PATCH v4 04/31] kconfig: remove string expansion in file_lookup()

[Masahiro Yamada]

There are two callers of file_lookup(), but there is no more reason
to expand the given path.

[1] zconf_initscan()
    This is used to open the first Kconfig.  sym_expand_string_value()
    has never been used in a useful way here; before opening the first
    Kconfig file, obviously there is no symbol to expand.  If you use
    expand_string_value() instead, environments in KBUILD_KCONFIG would
    be expanded, but I do not see practical benefits for that.

[2] zconf_nextfile()
    This is used to open the next file from 'source' statement.
    Symbols in the path like "arch/$SRCARCH/Kconfig" needed expanding,
    but it was replaced with the direct environment expansion.  The
    environment has already been expanded before the token is passed
    to the parser.

By the way, file_lookup() was already buggy; it expanded a given path,
but it used the path before expansion for look-up:
        if (!strcmp(name, file->name)) {

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Ulf Magnusson <ulfalizer@gmail.com>
---

Changes in v4: None
Changes in v3: None
Changes in v2:
  - Simplify the patch.  Just remove text expansion.

 scripts/kconfig/util.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index 807147e..790967df 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -14,18 +14,16 @@
 struct file *file_lookup(const char *name)
 {
 	struct file *file;
-	char *file_name = sym_expand_string_value(name);
 
 	for (file = file_list; file; file = file->next) {
 		if (!strcmp(name, file->name)) {
-			free(file_name);
 			return file;
 		}
 	}
 
 	file = xmalloc(sizeof(*file));
 	memset(file, 0, sizeof(*file));
-	file->name = file_name;
+	file->name = xstrdup(name);
 	file->next = file_list;
 	file_list = file;
 	return file;
-- 
2.7.4

[PATCH v4 05/31] kconfig: remove string expansion for mainmenu after yyparse()

[Masahiro Yamada]

Now that environments are expanded in the lexer, conf_parse() does
not need to expand them explicitly.

The hack introduced by commit 0724a7c32a54 ("kconfig: Don't leak
main menus during parsing") can go away.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Ulf Magnusson <ulfalizer@gmail.com>
---

Changes in v4: None
Changes in v3: None
Changes in v2:
  - Simplify the patch.  Just remove the text expansion.

 scripts/kconfig/zconf.y | 24 +++++-------------------
 1 file changed, 5 insertions(+), 19 deletions(-)

diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index 3a4a0fa..22e318c 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -109,7 +109,7 @@ static struct menu *current_menu, *current_entry;
 %%
 input: nl start | start;
 
-start: mainmenu_stmt stmt_list | no_mainmenu_stmt stmt_list;
+start: mainmenu_stmt stmt_list | stmt_list;
 
 /* mainmenu entry */
 
@@ -118,19 +118,6 @@ mainmenu_stmt: T_MAINMENU prompt nl
 	menu_add_prompt(P_MENU, $2, NULL);
 };
 
-/* Default main menu, if there's no mainmenu entry */
-
-no_mainmenu_stmt: /* empty */
-{
-	/*
-	 * Hack: Keep the main menu title on the heap so we can safely free it
-	 * later regardless of whether it comes from the 'prompt' in
-	 * mainmenu_stmt or here
-	 */
-	menu_add_prompt(P_MENU, xstrdup("Linux Kernel Configuration"), NULL);
-};
-
-
 stmt_list:
 	  /* empty */
 	| stmt_list common_stmt
@@ -528,7 +515,6 @@ word_opt: /* empty */			{ $$ = NULL; }
 
 void conf_parse(const char *name)
 {
-	const char *tmp;
 	struct symbol *sym;
 	int i;
 
@@ -544,10 +530,10 @@ void conf_parse(const char *name)
 	if (!modules_sym)
 		modules_sym = sym_find( "n" );
 
-	tmp = rootmenu.prompt->text;
-	rootmenu.prompt->text = _(rootmenu.prompt->text);
-	rootmenu.prompt->text = sym_expand_string_value(rootmenu.prompt->text);
-	free((char*)tmp);
+	if (!menu_has_prompt(&rootmenu)) {
+		current_entry = &rootmenu;
+		menu_add_prompt(P_MENU, "Linux Kernel Configuration", NULL);
+	}
 
 	menu_finalize(&rootmenu);
 	for_all_symbols(i, sym) {
-- 
2.7.4

[PATCH v4 06/31] kconfig: remove sym_expand_string_value()

[Masahiro Yamada]

There is no more caller of sym_expand_string_value().

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4: None
Changes in v3:
  - newly added

Changes in v2: None

 scripts/kconfig/lkc_proto.h |  1 -
 scripts/kconfig/symbol.c    | 53 ---------------------------------------------
 2 files changed, 54 deletions(-)

diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
index 9f465fe..c46929f 100644
--- a/scripts/kconfig/lkc_proto.h
+++ b/scripts/kconfig/lkc_proto.h
@@ -31,7 +31,6 @@ extern struct symbol * symbol_hash[SYMBOL_HASHSIZE];
 
 struct symbol * sym_lookup(const char *name, int flags);
 struct symbol * sym_find(const char *name);
-char *sym_expand_string_value(const char *in);
 const char * sym_escape_string_value(const char *in);
 struct symbol ** sym_re_search(const char *pattern);
 const char * sym_type_name(enum symbol_type type);
diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
index 2460648..7c9a88e 100644
--- a/scripts/kconfig/symbol.c
+++ b/scripts/kconfig/symbol.c
@@ -879,59 +879,6 @@ struct symbol *sym_find(const char *name)
 	return symbol;
 }
 
-/*
- * Expand symbol's names embedded in the string given in argument. Symbols'
- * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
- * the empty string.
- */
-char *sym_expand_string_value(const char *in)
-{
-	const char *src;
-	char *res;
-	size_t reslen;
-
-	/*
-	 * Note: 'in' might come from a token that's about to be
-	 * freed, so make sure to always allocate a new string
-	 */
-	reslen = strlen(in) + 1;
-	res = xmalloc(reslen);
-	res[0] = '\0';
-
-	while ((src = strchr(in, '$'))) {
-		char *p, name[SYMBOL_MAXLENGTH];
-		const char *symval = "";
-		struct symbol *sym;
-		size_t newlen;
-
-		strncat(res, in, src - in);
-		src++;
-
-		p = name;
-		while (isalnum(*src) || *src == '_')
-			*p++ = *src++;
-		*p = '\0';
-
-		sym = sym_find(name);
-		if (sym != NULL) {
-			sym_calc_value(sym);
-			symval = sym_get_string_value(sym);
-		}
-
-		newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
-		if (newlen > reslen) {
-			reslen = newlen;
-			res = xrealloc(res, reslen);
-		}
-
-		strcat(res, symval);
-		in = src;
-	}
-	strcat(res, in);
-
-	return res;
-}
-
 const char *sym_escape_string_value(const char *in)
 {
 	const char *p;
-- 
2.7.4

[PATCH v4 07/31] kconfig: add built-in function support

[Masahiro Yamada]

This commit adds a new concept 'function' to do more text processing
in Kconfig.

A function call looks like this:

  $(function,arg1,arg2,arg3,...)

This commit adds the basic infrastructure to expand functions.
Change the text expansion helpers to take arguments.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4:
  - Error out if arguments more than FUNCTION_MAX_ARGS are passed
  - Use a comma as a delimiter between the function name and the
    first argument
  - Check the number of arguments accepted by each function
  - Support delayed expansion of arguments.
    This will be needed to implement 'if' function

Changes in v3:
  - Split base infrastructure and 'shell' function
    into separate patches.

Changes in v2:
  - Use 'shell' for getting stdout from the comment.
    It was 'shell-stdout' in the previous version.
  - Simplify the implementation since the expansion has been moved to
    lexer.

 scripts/kconfig/preprocess.c | 168 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 159 insertions(+), 9 deletions(-)

diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
index 1bf506c..5be28ec 100644
--- a/scripts/kconfig/preprocess.c
+++ b/scripts/kconfig/preprocess.c
@@ -3,12 +3,17 @@
 // Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
 
 #include <stdarg.h>
+#include <stdbool.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 
 #include "list.h"
 
+#define ARRAY_SIZE(arr)		(sizeof(arr) / sizeof((arr)[0]))
+
+static char *expand_string_with_args(const char *in, int argc, char *argv[]);
+
 static void __attribute__((noreturn)) pperror(const char *format, ...)
 {
 	va_list ap;
@@ -88,9 +93,85 @@ void env_write_dep(FILE *f, const char *autoconfig_name)
 	}
 }
 
-static char *eval_clause(const char *in)
+/*
+ * Built-in functions
+ */
+struct function {
+	const char *name;
+	unsigned int min_args;
+	unsigned int max_args;
+	bool expand_args;
+	char *(*func)(int argc, char *argv[], int old_argc, char *old_argv[]);
+};
+
+static const struct function function_table[] = {
+	/* Name		MIN	MAX	EXP?	Function */
+};
+
+#define FUNCTION_MAX_ARGS		16
+
+static char *function_expand_arg_and_call(char *(*func)(int argc, char *argv[],
+							int old_argc,
+							char *old_argv[]),
+					  int argc, char *argv[],
+					  int old_argc, char *old_argv[])
+{
+	char *expanded_argv[FUNCTION_MAX_ARGS], *res;
+	int i;
+
+	for (i = 0; i < argc; i++)
+		expanded_argv[i] = expand_string_with_args(argv[i],
+							   old_argc, old_argv);
+
+	res = func(argc, expanded_argv, 0, NULL);
+
+	for (i = 0; i < argc; i++)
+		free(expanded_argv[i]);
+
+	return res;
+}
+
+static char *function_call(const char *name, int argc, char *argv[],
+			   int old_argc, char *old_argv[])
+{
+	const struct function *f;
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(function_table); i++) {
+		f = &function_table[i];
+		if (strcmp(f->name, name))
+			continue;
+
+		if (argc < f->min_args)
+			pperror("too few function arguments passed to '%s'",
+				name);
+
+		if (argc > f->max_args)
+			pperror("too many function arguments passed to '%s'",
+				name);
+
+		if (f->expand_args)
+			return function_expand_arg_and_call(f->func, argc, argv,
+							    old_argc, old_argv);
+		return f->func(argc, argv, old_argc, old_argv);
+	}
+
+	return NULL;
+}
+
+/*
+ * Evaluate a clause with arguments.  argc/argv are arguments from the upper
+ * function call.
+ *
+ * Returned string must be freed when done
+ */
+static char *eval_clause(const char *in, int argc, char *argv[])
 {
-	char *res, *name;
+	char *tmp, *prev, *p, *res, *name;
+	int new_argc = 0;
+	char *new_argv[FUNCTION_MAX_ARGS];
+	int nest = 0;
+	int i;
 
 	/*
 	 * Returns an empty string because '$()' should be evaluated
@@ -99,10 +180,69 @@ static char *eval_clause(const char *in)
 	if (!*in)
 		return xstrdup("");
 
-	name = expand_string(in);
+	tmp = xstrdup(in);
+
+	prev = p = tmp;
+
+	/*
+	 * Split into tokens
+	 * The function name and arguments are separated by a comma.
+	 * For example, if the function call is like this:
+	 *   $(foo,abc,$(x),$(y))
+	 *
+	 * The input string for this helper should be:
+	 *   foo,abc,$(x),$(y)
+	 *
+	 * and split into:
+	 *   new_argv[0] = 'foo'
+	 *   new_argv[1] = 'abc'
+	 *   new_argv[2] = '$(x)'
+	 *   new_argv[3] = '$(y)'
+	 */
+	while (*p) {
+		if (nest == 0 && *p == ',') {
+			*p = 0;
+			if (new_argc >= FUNCTION_MAX_ARGS)
+				pperror("too many function arguments");
+			new_argv[new_argc++] = prev;
+			prev = p + 1;
+		} else if (*p == '(') {
+			nest++;
+		} else if (*p == ')') {
+			nest--;
+		}
+
+		p++;
+	}
+	new_argv[new_argc++] = prev;
 
-	res = env_expand(name);
+	/*
+	 * Shift arguments
+	 * new_argv[0] represents a function name or a variable name.  Put it
+	 * into 'name', then shift the rest of the arguments.  This simplifies
+	 * 'const' handling.
+	 */
+	name = expand_string_with_args(new_argv[0], argc, argv);
+	new_argc--;
+	for (i = 0; i < new_argc; i++)
+		new_argv[i] = new_argv[i + 1];
+
+	/* Look for built-in functions */
+	res = function_call(name, new_argc, new_argv, argc, argv);
+	if (res)
+		goto out;
+
+	/* Last, try environment variable */
+	if (new_argc == 0) {
+		res = env_expand(name);
+		if (res)
+			goto out;
+	}
+
+	res = xstrdup("");
+out:
 	free(name);
+	free(tmp);
 
 	return res;
 }
@@ -119,7 +259,7 @@ static char *eval_clause(const char *in)
  * will be
  *     $(BAR)
  */
-char *expand_dollar(const char **str)
+static char *expand_dollar_with_args(const char **str, int argc, char *argv[])
 {
 	const char *p = *str;
 	const char *q;
@@ -158,18 +298,18 @@ char *expand_dollar(const char **str)
 	memcpy(tmp, p, q - p);
 	tmp[q - p] = 0;
 	*str = q + 1;
-	out = eval_clause(tmp);
+	out = eval_clause(tmp, argc, argv);
 	free(tmp);
 
 	return out;
 }
 
 /*
- * Expand variables in the given string.  Undefined variables
+ * Expand variables, functions, etc. in the given string.  Undefined variables
  * expand to an empty string.
  * The returned string must be freed when done.
  */
-char *expand_string(const char *in)
+static char *expand_string_with_args(const char *in, int argc, char *argv[])
 {
 	const char *prev_in, *p;
 	char *new, *out;
@@ -181,7 +321,7 @@ char *expand_string(const char *in)
 	while ((p = strchr(in, '$'))) {
 		prev_in = in;
 		in = p + 1;
-		new = expand_dollar(&in);
+		new = expand_dollar_with_args(&in, argc, argv);
 		outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
 		out = xrealloc(out, outlen);
 		strncat(out, prev_in, p - prev_in);
@@ -196,6 +336,16 @@ char *expand_string(const char *in)
 	return out;
 }
 
+char *expand_string(const char *in)
+{
+	return expand_string_with_args(in, 0, NULL);
+}
+
+char *expand_dollar(const char **str)
+{
+	return expand_dollar_with_args(str, 0, NULL);
+}
+
 /*
  * Expand variables in a token.  The parsing stops when a token separater
  * (in most cases, it is a whitespace) is encountered.  'str' is updated to
-- 
2.7.4

[PATCH v4 08/31] kconfig: add 'shell' built-in function

[Masahiro Yamada]

This accepts a single command to execute.  It returns the standard
output from it.

[Example code]

  config HELLO
          string
          default "$(shell,echo hello world)"

  config Y
          def_bool $(shell,echo y)

[Result]

  $ make -s alldefconfig && tail -n 2 .config
  CONFIG_HELLO="hello world"
  CONFIG_Y=y

Caveat:
Like environments, functions are expanded in the lexer.  You cannot
pass symbols to function arguments.  This is a limitation to simplify
the implementation.  I want to avoid the dynamic function evaluation,
which would introduce much more complexity.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4:
  - Accept only one argument to simplify the implementation

Changes in v3: None
Changes in v2: None

 scripts/kconfig/preprocess.c | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
index 5be28ec..d8c5f60 100644
--- a/scripts/kconfig/preprocess.c
+++ b/scripts/kconfig/preprocess.c
@@ -104,8 +104,49 @@ struct function {
 	char *(*func)(int argc, char *argv[], int old_argc, char *old_argv[]);
 };
 
+static char *do_shell(int argc, char *argv[], int old_argc, char *old_argv[])
+{
+	FILE *p;
+	char buf[256];
+	char *cmd;
+	size_t nread;
+	int i;
+
+	cmd = argv[0];
+
+	p = popen(cmd, "r");
+	if (!p) {
+		perror(cmd);
+		exit(1);
+	}
+
+	nread = fread(buf, 1, sizeof(buf), p);
+	if (nread == sizeof(buf))
+		nread--;
+
+	/* remove trailing new lines */
+	while (buf[nread - 1] == '\n')
+		nread--;
+
+	buf[nread] = 0;
+
+	/* replace a new line with a space */
+	for (i = 0; i < nread; i++) {
+		if (buf[i] == '\n')
+			buf[i] = ' ';
+	}
+
+	if (pclose(p) == -1) {
+		perror(cmd);
+		exit(1);
+	}
+
+	return xstrdup(buf);
+}
+
 static const struct function function_table[] = {
 	/* Name		MIN	MAX	EXP?	Function */
+	{ "shell",	1,	1,	true,	do_shell },
 };
 
 #define FUNCTION_MAX_ARGS		16
-- 
2.7.4

[PATCH v4 09/31] kconfig: replace $(UNAME_RELEASE) with function call

[Masahiro Yamada]

Now that 'shell' function is supported, this can be self-contained in
Kconfig.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Ulf Magnusson <ulfalizer@gmail.com>
---

Changes in v4: None
Changes in v3: None
Changes in v2: None

 Makefile     | 3 +--
 init/Kconfig | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index c6278e6..01b2211 100644
--- a/Makefile
+++ b/Makefile
@@ -284,8 +284,7 @@ include scripts/Kbuild.include
 # Read KERNELRELEASE from include/config/kernel.release (if it exists)
 KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
 KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
-UNAME_RELEASE := $(shell uname --release)
-export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
+export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
 
 # SUBARCH tells the usermode build what the underlying arch is.  That is set
 # first, and if a usermode build is happening, the "ARCH=um" on the command
diff --git a/init/Kconfig b/init/Kconfig
index 752816b..66f0463 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2,9 +2,9 @@ config DEFCONFIG_LIST
 	string
 	depends on !UML
 	option defconfig_list
-	default "/lib/modules/$(UNAME_RELEASE)/.config"
+	default "/lib/modules/$(shell, uname --release)/.config"
 	default "/etc/kernel-config"
-	default "/boot/config-$(UNAME_RELEASE)"
+	default "/boot/config-$(shell, uname --release)"
 	default ARCH_DEFCONFIG
 	default "arch/$(ARCH)/defconfig"
 
-- 
2.7.4

[PATCH v4 10/31] kconfig: begin PARAM state only when seeing a command keyword

[Masahiro Yamada]

Currently, any statement line starts with a keyword with TF_COMMAND
flag.  So, the following three lines are dead code.

        alloc_string(yytext, yyleng);
        zconflval.string = text;
        return T_WORD;

If a T_WORD token is returned in this context, it will cause syntax
error in the parser anyway.

The next commit will support the assignment statement where a line
starts with an arbitrary identifier.  So, I want the lexer to switch
to the PARAM state only when it sees a command keyword.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4: None
Changes in v3:
 - Newly added

Changes in v2: None

 scripts/kconfig/zconf.l | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
index 9dc5fe3..5e53348 100644
--- a/scripts/kconfig/zconf.l
+++ b/scripts/kconfig/zconf.l
@@ -102,10 +102,10 @@ n	[A-Za-z0-9_-]
 <COMMAND>{
 	{n}+	{
 		const struct kconf_id *id = kconf_id_lookup(yytext, yyleng);
-		BEGIN(PARAM);
 		current_pos.file = current_file;
 		current_pos.lineno = yylineno;
 		if (id && id->flags & TF_COMMAND) {
+			BEGIN(PARAM);
 			yylval.id = id;
 			return id->token;
 		}
-- 
2.7.4

[PATCH v4 11/31] kconfig: support user-defined function and recursively expanded variable

[Masahiro Yamada]

Now, we got a basic ability to test compiler capability in Kconfig.

config CC_HAS_STACKPROTECTOR
        def_bool $(shell, ($(CC) -Werror -fstack-protector -E -x c /dev/null -o /dev/null 2>/dev/null) && echo y || echo n)

This works, but it is ugly to repeat this long boilerplate.

We want to describe like this:

config CC_HAS_STACKPROTECTOR
        bool
        default $(cc-option, -fstack-protector)

It is straight-forward to add a new function, but I do not like to
hard-code specialized functions like that.  Hence, here is another
feature, user-defined function.  This works as a textual shorthand
with parameterization.

A user-defined function is defined by using the = operator, and can
be referenced in the same way as built-in functions.  A user-defined
function in Make is referenced like $(call my-func,arg1,arg2), but I
omitted the 'call' to make the syntax shorter.

The definition of a user-defined function contains $(1), $(2), etc.
in its body to reference the parameters.  It is grammatically valid
to pass more or fewer arguments when calling it.  We already exploit
this feature in our makefiles; scripts/Kbuild.include defines cc-option
which takes two arguments at most, but most of the callers pass only
one argument.

By the way, a variable is supported as a subset of this feature since
a variable is "a user-defined function with zero argument".  In this
context, I mean "variable" as recursively expanded variable.  I will
add a different flavored variable in the next commit.

The code above can be written as follows:

[Example Code]

  success = $(shell, ($(1)) >/dev/null 2>&1 && echo y || echo n)
  cc-option = $(success, $(CC) -Werror $(1) -c -x c /dev/null -o /dev/null)

  config CC_HAS_STACKPROTECTOR
          def_bool $(cc-option, -fstack-protector)

[Result]
  $ make -s alldefconfig && tail -n 1 .config
  CONFIG_CC_HAS_STACKPROTECTOR=y

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4: None
Changes in v3:
  - Re-implement the parse logic
  - Use = operator to define a user-defined function

Changes in v2:
  - Use 'macro' directly instead of inside the string type symbol.

 scripts/kconfig/lkc_proto.h  |   2 +
 scripts/kconfig/preprocess.c | 108 ++++++++++++++++++++++++++++++++++++++++++-
 scripts/kconfig/zconf.l      |  17 ++++++-
 scripts/kconfig/zconf.y      |  19 +++++++-
 4 files changed, 142 insertions(+), 4 deletions(-)

diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
index c46929f..2b16d6e 100644
--- a/scripts/kconfig/lkc_proto.h
+++ b/scripts/kconfig/lkc_proto.h
@@ -50,6 +50,8 @@ const char * prop_get_type_name(enum prop_type type);
 
 /* preprocess.c */
 void env_write_dep(FILE *f, const char *auto_conf_name);
+void variable_add(const char *name, const char *value);
+void variable_all_del(void);
 char *expand_string(const char *in);
 char *expand_dollar(const char **str);
 char *expand_one_token(const char **str);
diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
index d8c5f60..88580c4 100644
--- a/scripts/kconfig/preprocess.c
+++ b/scripts/kconfig/preprocess.c
@@ -201,18 +201,110 @@ static char *function_call(const char *name, int argc, char *argv[],
 }
 
 /*
+ * Variables (and user-defined functions)
+ */
+static LIST_HEAD(variable_list);
+
+struct variable {
+	char *name;
+	char *value;
+	struct list_head node;
+};
+
+static struct variable *variable_lookup(const char *name)
+{
+	struct variable *v;
+
+	list_for_each_entry(v, &variable_list, node) {
+		if (!strcmp(name, v->name))
+			return v;
+	}
+
+	return NULL;
+}
+
+static char *variable_expand(const char *name, int argc, char *argv[],
+			     int old_argc, char *old_argv[])
+{
+	struct variable *v;
+	char *expanded_argv[FUNCTION_MAX_ARGS], *res;
+	int i;
+
+	v = variable_lookup(name);
+	if (!v)
+		return NULL;
+
+	for (i = 0; i < argc; i++)
+		expanded_argv[i] = expand_string_with_args(argv[i],
+							   old_argc, old_argv);
+
+	res = expand_string_with_args(v->value, argc, expanded_argv);
+
+	for (i = 0; i < argc; i++)
+		free(expanded_argv[i]);
+
+	return res;
+}
+
+void variable_add(const char *name, const char *value)
+{
+	struct variable *v;
+
+	v = variable_lookup(name);
+	if (v) {
+		free(v->value);
+	} else {
+		v = xmalloc(sizeof(*v));
+		v->name = xstrdup(name);
+		list_add_tail(&v->node, &variable_list);
+	}
+
+	v->value = xstrdup(value);
+}
+
+static void variable_del(struct variable *v)
+{
+	list_del(&v->node);
+	free(v->name);
+	free(v->value);
+	free(v);
+}
+
+void variable_all_del(void)
+{
+	struct variable *v, *tmp;
+
+	list_for_each_entry_safe(v, tmp, &variable_list, node)
+		variable_del(v);
+}
+
+/*
  * Evaluate a clause with arguments.  argc/argv are arguments from the upper
  * function call.
  *
+ * Let's say 'foo' is defined as:
+ *   foo = ABC$(1)PQR(2)XYZ
+ * and you want to evaluate $(foo x,y)
+ *
+ * First, this helper is called with:
+ *   in  :    foo x,y
+ *   argc:    0
+ * and then, recursively called with:
+ *   in:      ABC$(1)PQR(2)XYZ
+ *   argc:    2
+ *   argv[0]: x
+ *   argv[1]: y
+ *
  * Returned string must be freed when done
  */
 static char *eval_clause(const char *in, int argc, char *argv[])
 {
-	char *tmp, *prev, *p, *res, *name;
+	char *tmp, *prev, *p, *res, *endptr, *name;
 	int new_argc = 0;
 	char *new_argv[FUNCTION_MAX_ARGS];
 	int nest = 0;
 	int i;
+	unsigned long n;
 
 	/*
 	 * Returns an empty string because '$()' should be evaluated
@@ -221,6 +313,15 @@ static char *eval_clause(const char *in, int argc, char *argv[])
 	if (!*in)
 		return xstrdup("");
 
+	/*
+	 * If variable name is '1', '2', etc.  It is generally an argument
+	 * from a user-function call (i.e. local-scope variable).  If not
+	 * available, then look-up global-scope variables.
+	 */
+	n = strtoul(in, &endptr, 10);
+	if (!*endptr && n > 0 && n <= argc)
+		return xstrdup(argv[n - 1]);
+
 	tmp = xstrdup(in);
 
 	prev = p = tmp;
@@ -268,6 +369,11 @@ static char *eval_clause(const char *in, int argc, char *argv[])
 	for (i = 0; i < new_argc; i++)
 		new_argv[i] = new_argv[i + 1];
 
+	/* Search for variables */
+	res = variable_expand(name, new_argc, new_argv, argc, argv);
+	if (res)
+		goto out;
+
 	/* Look for built-in functions */
 	res = function_call(name, new_argc, new_argv, argc, argv);
 	if (res)
diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
index 5e53348..19e5ebf 100644
--- a/scripts/kconfig/zconf.l
+++ b/scripts/kconfig/zconf.l
@@ -1,12 +1,13 @@
 %option nostdinit noyywrap never-interactive full ecs
 %option 8bit nodefault yylineno
-%x COMMAND HELP STRING PARAM
+%x COMMAND HELP STRING PARAM ASSIGN_VAL
 %{
 /*
  * Copyright (C) 2002 Roman Zippel <zippel@linux-m68k.org>
  * Released under the terms of the GNU GPL v2.0.
  */
 
+#include <assert.h>
 #include <limits.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -111,8 +112,10 @@ n	[A-Za-z0-9_-]
 		}
 		alloc_string(yytext, yyleng);
 		yylval.string = text;
-		return T_WORD;
+		return T_VARIABLE;
 	}
+	"="	{ BEGIN(ASSIGN_VAL); return T_ASSIGN; }
+	[[:blank:]]+
 	.	warn_ignored_character(*yytext);
 	\n	{
 		BEGIN(INITIAL);
@@ -120,6 +123,16 @@ n	[A-Za-z0-9_-]
 	}
 }
 
+<ASSIGN_VAL>{
+	[^[:blank:]\n]+.*	{
+		alloc_string(yytext, yyleng);
+		yylval.string = text;
+		return T_ASSIGN_VAL;
+	}
+	\n	{ BEGIN(INITIAL); return T_EOL; }
+	.
+}
+
 <PARAM>{
 	"&&"	return T_AND;
 	"||"	return T_OR;
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index 22e318c..6201119 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -77,6 +77,9 @@ static struct menu *current_menu, *current_entry;
 %token T_CLOSE_PAREN
 %token T_OPEN_PAREN
 %token T_EOL
+%token <string> T_VARIABLE
+%token T_ASSIGN
+%token <string> T_ASSIGN_VAL
 
 %left T_OR
 %left T_AND
@@ -92,7 +95,7 @@ static struct menu *current_menu, *current_entry;
 %type <id> end
 %type <id> option_name
 %type <menu> if_entry menu_entry choice_entry
-%type <string> symbol_option_arg word_opt
+%type <string> symbol_option_arg word_opt assign_val
 
 %destructor {
 	fprintf(stderr, "%s:%d: missing end statement for this entry\n",
@@ -143,6 +146,7 @@ common_stmt:
 	| config_stmt
 	| menuconfig_stmt
 	| source_stmt
+	| assignment_stmt
 ;
 
 option_error:
@@ -511,6 +515,15 @@ symbol:	  nonconst_symbol
 word_opt: /* empty */			{ $$ = NULL; }
 	| T_WORD
 
+/* assignment statement */
+
+assignment_stmt:  T_VARIABLE T_ASSIGN assign_val T_EOL	{ variable_add($1, $3); free($1); free($3); }
+
+assign_val:
+	/* empty */		{ $$ = xstrdup(""); };
+	| T_ASSIGN_VAL
+;
+
 %%
 
 void conf_parse(const char *name)
@@ -525,6 +538,10 @@ void conf_parse(const char *name)
 	if (getenv("ZCONF_DEBUG"))
 		yydebug = 1;
 	yyparse();
+
+	/* Variables are expanded in the parse phase. We can free them here. */
+	variable_all_del();
+
 	if (yynerrs)
 		exit(1);
 	if (!modules_sym)
-- 
2.7.4

[PATCH v4 12/31] kconfig: support simply expanded variable

[Masahiro Yamada]

The previous commit added variable and user-defined function.  They
work similarly in the sense that the evaluation is deferred until
they are used.

This commit adds another type of variable, simply expanded variable,
as we see in Make.

The := operator defines a simply expanded variable, expanding the
righthand side immediately.  This works like traditional programming
language variables.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4: None
Changes in v3:
  - newly added

Changes in v2: None

 scripts/kconfig/lkc_proto.h  | 7 ++++++-
 scripts/kconfig/preprocess.c | 6 ++++--
 scripts/kconfig/zconf.l      | 3 ++-
 scripts/kconfig/zconf.y      | 5 +++--
 4 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
index 2b16d6e..6303193 100644
--- a/scripts/kconfig/lkc_proto.h
+++ b/scripts/kconfig/lkc_proto.h
@@ -49,8 +49,13 @@ const char * sym_get_string_value(struct symbol *sym);
 const char * prop_get_type_name(enum prop_type type);
 
 /* preprocess.c */
+enum variable_flavor {
+	VAR_SIMPLE,
+	VAR_RECURSIVE,
+};
 void env_write_dep(FILE *f, const char *auto_conf_name);
-void variable_add(const char *name, const char *value);
+void variable_add(const char *name, const char *value,
+		  enum variable_flavor flavor);
 void variable_all_del(void);
 char *expand_string(const char *in);
 char *expand_dollar(const char **str);
diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
index 88580c4..5d7bd9d 100644
--- a/scripts/kconfig/preprocess.c
+++ b/scripts/kconfig/preprocess.c
@@ -246,7 +246,8 @@ static char *variable_expand(const char *name, int argc, char *argv[],
 	return res;
 }
 
-void variable_add(const char *name, const char *value)
+void variable_add(const char *name, const char *value,
+		  enum variable_flavor flavor)
 {
 	struct variable *v;
 
@@ -259,7 +260,8 @@ void variable_add(const char *name, const char *value)
 		list_add_tail(&v->node, &variable_list);
 	}
 
-	v->value = xstrdup(value);
+	v->value = (flavor == VAR_SIMPLE) ? expand_string(value) :
+								xstrdup(value);
 }
 
 static void variable_del(struct variable *v)
diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
index 19e5ebf..aa76942 100644
--- a/scripts/kconfig/zconf.l
+++ b/scripts/kconfig/zconf.l
@@ -114,7 +114,8 @@ n	[A-Za-z0-9_-]
 		yylval.string = text;
 		return T_VARIABLE;
 	}
-	"="	{ BEGIN(ASSIGN_VAL); return T_ASSIGN; }
+	"="	{ BEGIN(ASSIGN_VAL); yylval.flavor = VAR_RECURSIVE; return T_ASSIGN; }
+	":="	{ BEGIN(ASSIGN_VAL); yylval.flavor = VAR_SIMPLE; return T_ASSIGN; }
 	[[:blank:]]+
 	.	warn_ignored_character(*yytext);
 	\n	{
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index 6201119..16432d0 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -41,6 +41,7 @@ static struct menu *current_menu, *current_entry;
 	struct expr *expr;
 	struct menu *menu;
 	const struct kconf_id *id;
+	enum variable_flavor flavor;
 }
 
 %token <id>T_MAINMENU
@@ -78,7 +79,7 @@ static struct menu *current_menu, *current_entry;
 %token T_OPEN_PAREN
 %token T_EOL
 %token <string> T_VARIABLE
-%token T_ASSIGN
+%token <flavor> T_ASSIGN
 %token <string> T_ASSIGN_VAL
 
 %left T_OR
@@ -517,7 +518,7 @@ word_opt: /* empty */			{ $$ = NULL; }
 
 /* assignment statement */
 
-assignment_stmt:  T_VARIABLE T_ASSIGN assign_val T_EOL	{ variable_add($1, $3); free($1); free($3); }
+assignment_stmt:  T_VARIABLE T_ASSIGN assign_val T_EOL	{ variable_add($1, $3, $2); free($1); free($3); }
 
 assign_val:
 	/* empty */		{ $$ = xstrdup(""); };
-- 
2.7.4

[PATCH v4 13/31] kconfig: support append assignment operator

[Masahiro Yamada]

Support += operator.  This appends a space and the text on the
righthand side to a variable.

The timing of the evaluation of the righthand side depends on the
flavor of the variable.  If the lefthand side was originally defined
as a simple variable, the righthand side is expanded immediately.
Otherwise, the expansion is deferred.  Appending something to an
undefined variable results in a recursive variable.

To implement this, we need to remember the flavor of variables.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4: None
Changes in v3:
  - newly added

Changes in v2: None

 scripts/kconfig/lkc_proto.h  |  1 +
 scripts/kconfig/preprocess.c | 29 +++++++++++++++++++++++++++--
 scripts/kconfig/zconf.l      |  1 +
 3 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
index 6303193..a8b7a33 100644
--- a/scripts/kconfig/lkc_proto.h
+++ b/scripts/kconfig/lkc_proto.h
@@ -52,6 +52,7 @@ const char * prop_get_type_name(enum prop_type type);
 enum variable_flavor {
 	VAR_SIMPLE,
 	VAR_RECURSIVE,
+	VAR_APPEND,
 };
 void env_write_dep(FILE *f, const char *auto_conf_name);
 void variable_add(const char *name, const char *value,
diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
index 5d7bd9d..47b32dc 100644
--- a/scripts/kconfig/preprocess.c
+++ b/scripts/kconfig/preprocess.c
@@ -208,6 +208,7 @@ static LIST_HEAD(variable_list);
 struct variable {
 	char *name;
 	char *value;
+	enum variable_flavor flavor;
 	struct list_head node;
 };
 
@@ -250,18 +251,42 @@ void variable_add(const char *name, const char *value,
 		  enum variable_flavor flavor)
 {
 	struct variable *v;
+	char *new_value;
+	bool append = false;
 
 	v = variable_lookup(name);
 	if (v) {
-		free(v->value);
+		/* For defined variables, += inherits the existing flavor */
+		if (flavor == VAR_APPEND) {
+			flavor = v->flavor;
+			append = true;
+		} else {
+			free(v->value);
+		}
 	} else {
+		/* For undefined variables, += assumes the recursive flavor */
+		if (flavor == VAR_APPEND)
+			flavor = VAR_RECURSIVE;
+
 		v = xmalloc(sizeof(*v));
 		v->name = xstrdup(name);
 		list_add_tail(&v->node, &variable_list);
 	}
 
-	v->value = (flavor == VAR_SIMPLE) ? expand_string(value) :
+	v->flavor = flavor;
+
+	new_value = (flavor == VAR_SIMPLE) ? expand_string(value) :
 								xstrdup(value);
+
+	if (append) {
+		v->value = xrealloc(v->value,
+				    strlen(v->value) + strlen(new_value) + 2);
+		strcat(v->value, " ");
+		strcat(v->value, new_value);
+		free(new_value);
+	} else {
+		v->value = new_value;
+	}
 }
 
 static void variable_del(struct variable *v)
diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
index aa76942..c68ca56b 100644
--- a/scripts/kconfig/zconf.l
+++ b/scripts/kconfig/zconf.l
@@ -116,6 +116,7 @@ n	[A-Za-z0-9_-]
 	}
 	"="	{ BEGIN(ASSIGN_VAL); yylval.flavor = VAR_RECURSIVE; return T_ASSIGN; }
 	":="	{ BEGIN(ASSIGN_VAL); yylval.flavor = VAR_SIMPLE; return T_ASSIGN; }
+	"+="	{ BEGIN(ASSIGN_VAL); yylval.flavor = VAR_APPEND; return T_ASSIGN; }
 	[[:blank:]]+
 	.	warn_ignored_character(*yytext);
 	\n	{
-- 
2.7.4

[PATCH v4 14/31] kconfig: expand lefthand side of assignment statement

[Masahiro Yamada]

Make expands the lefthand side of assignment statements.  In fact,
Kbuild relies on it since kernel makefiles mostly look like follows:

  obj-$(CONFIG_FOO) += foo.o

Do likewise in Kconfig.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4: None
Changes in v3: None
Changes in v2: None

 scripts/kconfig/zconf.l | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
index c68ca56b..ae33e9b 100644
--- a/scripts/kconfig/zconf.l
+++ b/scripts/kconfig/zconf.l
@@ -114,6 +114,13 @@ n	[A-Za-z0-9_-]
 		yylval.string = text;
 		return T_VARIABLE;
 	}
+	({n}|$)+	{
+		/* this token includes at least one '$' */
+		yylval.string = expand_token(yytext, yyleng);
+		if (strlen(yylval.string))
+			return T_VARIABLE;
+		free(yylval.string);
+	}
 	"="	{ BEGIN(ASSIGN_VAL); yylval.flavor = VAR_RECURSIVE; return T_ASSIGN; }
 	":="	{ BEGIN(ASSIGN_VAL); yylval.flavor = VAR_SIMPLE; return T_ASSIGN; }
 	"+="	{ BEGIN(ASSIGN_VAL); yylval.flavor = VAR_APPEND; return T_ASSIGN; }
-- 
2.7.4

[PATCH v4 15/31] kconfig: add 'info', 'warning', and 'error' built-in functions

[Masahiro Yamada]

Add 'info', 'warning', and 'error' functions as in Make.

They print messages during parsing Kconfig files. 'error' will be
useful to terminate the parsing immediately in fatal cases.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4:
  - Add 'error' function

Changes in v3: None
Changes in v2: None

 scripts/kconfig/preprocess.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
index 47b32dc..591bcf7 100644
--- a/scripts/kconfig/preprocess.c
+++ b/scripts/kconfig/preprocess.c
@@ -104,6 +104,20 @@ struct function {
 	char *(*func)(int argc, char *argv[], int old_argc, char *old_argv[]);
 };
 
+static char *do_error(int argc, char *argv[], int old_argc, char *old_argv[])
+{
+	pperror("%s", argv[0]);
+
+	return NULL;
+}
+
+static char *do_info(int argc, char *argv[], int old_argc, char *old_argv[])
+{
+	printf("%s\n", argv[0]);
+
+	return xstrdup("");
+}
+
 static char *do_shell(int argc, char *argv[], int old_argc, char *old_argv[])
 {
 	FILE *p;
@@ -144,9 +158,19 @@ static char *do_shell(int argc, char *argv[], int old_argc, char *old_argv[])
 	return xstrdup(buf);
 }
 
+static char *do_warning(int argc, char *argv[], int old_argc, char *old_argv[])
+{
+	fprintf(stderr, "%s:%d: %s\n", current_file->name, yylineno, argv[0]);
+
+	return xstrdup("");
+}
+
 static const struct function function_table[] = {
 	/* Name		MIN	MAX	EXP?	Function */
+	{ "error",	1,	1,	true,	do_error },
+	{ "info",	1,	1,	true,	do_info },
 	{ "shell",	1,	1,	true,	do_shell },
+	{ "warning",	1,	1,	true,	do_warning },
 };
 
 #define FUNCTION_MAX_ARGS		16
-- 
2.7.4

[PATCH v4 16/31] kconfig: add 'if' built-in function

[Masahiro Yamada]

The 'if' function is invoked in the form:

  $(if,condition,then-part,else-part)

The behavior is slightly different from that of Make.

In Make, the condition is true if the expansion contains any characters
(even space).  That is why we sometimes need $(strip ...) in the
condition part.

I thought it was a nasty part in Make, so I changed it for Kconfig;
the condition is true if the expansion contains any characters except
whitespaces.

Unlike the other functions, the parameters passed to 'if' are lazily
expanded.  The then-part is expanded only when the condition true,
and the else-part when false.  If the parameters were evaluated
before passed to the 'if' function, $(if,$(cond),$(error,...))
would terminate the parsing regardless of $(cond).

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4:
 - Newly added

Changes in v3: None
Changes in v2: None

 scripts/kconfig/preprocess.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
index 591bcf7..88844a7 100644
--- a/scripts/kconfig/preprocess.c
+++ b/scripts/kconfig/preprocess.c
@@ -111,6 +111,32 @@ static char *do_error(int argc, char *argv[], int old_argc, char *old_argv[])
 	return NULL;
 }
 
+static char *do_if(int argc, char *argv[], int old_argc, char *old_argv[])
+{
+	char *cond, *p, *res;
+	const char *sel;
+
+	cond = expand_string_with_args(argv[0], old_argc, old_argv);
+	p = cond;
+
+	/* condition is true if any character except whitespaces is contained */
+	while (*p == ' ' || *p == '\t')
+		p++;
+
+	if (*p)
+		sel = argv[1];
+	else if (argc >= 3)
+		sel = argv[2];
+	else
+		sel = "";
+
+	res = expand_string_with_args(sel, old_argc, old_argv);
+
+	free(cond);
+
+	return res;
+}
+
 static char *do_info(int argc, char *argv[], int old_argc, char *old_argv[])
 {
 	printf("%s\n", argv[0]);
@@ -168,6 +194,7 @@ static char *do_warning(int argc, char *argv[], int old_argc, char *old_argv[])
 static const struct function function_table[] = {
 	/* Name		MIN	MAX	EXP?	Function */
 	{ "error",	1,	1,	true,	do_error },
+	{ "if",		2,	3,	false,	do_if },
 	{ "info",	1,	1,	true,	do_info },
 	{ "shell",	1,	1,	true,	do_shell },
 	{ "warning",	1,	1,	true,	do_warning },
-- 
2.7.4

[PATCH v4 17/31] kconfig: add 'filename' and 'lineno' built-in variables

[Masahiro Yamada]

The special variables, $(filename) and $(lineno), are expanded to a
file name and its line number being parsed, respectively.

Suggested-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4:
 - Newly added

Changes in v3: None
Changes in v2: None

 scripts/kconfig/preprocess.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
index 88844a7..c39e30e 100644
--- a/scripts/kconfig/preprocess.c
+++ b/scripts/kconfig/preprocess.c
@@ -111,6 +111,11 @@ static char *do_error(int argc, char *argv[], int old_argc, char *old_argv[])
 	return NULL;
 }
 
+static char *do_filename(int argc, char *argv[], int old_argc, char *old_argv[])
+{
+	return xstrdup(current_file->name);
+}
+
 static char *do_if(int argc, char *argv[], int old_argc, char *old_argv[])
 {
 	char *cond, *p, *res;
@@ -144,6 +149,15 @@ static char *do_info(int argc, char *argv[], int old_argc, char *old_argv[])
 	return xstrdup("");
 }
 
+static char *do_lineno(int argc, char *argv[], int old_argc, char *old_argv[])
+{
+	char buf[16];
+
+	sprintf(buf, "%d", yylineno);
+
+	return xstrdup(buf);
+}
+
 static char *do_shell(int argc, char *argv[], int old_argc, char *old_argv[])
 {
 	FILE *p;
@@ -194,8 +208,10 @@ static char *do_warning(int argc, char *argv[], int old_argc, char *old_argv[])
 static const struct function function_table[] = {
 	/* Name		MIN	MAX	EXP?	Function */
 	{ "error",	1,	1,	true,	do_error },
+	{ "filename",	0,	0,	false,	do_filename },
 	{ "if",		2,	3,	false,	do_if },
 	{ "info",	1,	1,	true,	do_info },
+	{ "lineno",	0,	0,	false,	do_lineno },
 	{ "shell",	1,	1,	true,	do_shell },
 	{ "warning",	1,	1,	true,	do_warning },
 };
-- 
2.7.4

[PATCH v4 18/31] kconfig: error out if a recursive variable references itself

[Masahiro Yamada]

When using a recursively expanded variable, it is a common mistake
to make circular reference.

For example, Make terminates the following code:

  X = $(X)
  Y := $(X)

Let's detect the circular expansion in Kconfig, too.

On the other hand, a function that recurses itself is a commonly-used
programming technique.  So, Make does not check recursion in the
reference with 'call'.  For example, the following code continues
running eternally:

  X = $(call X)
  Y := $(X)

Kconfig allows circular expansion if one or more arguments are given,
but terminates when the same function is recursively invoked 1000 times,
assuming it is a programming mistake.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4:
 - Newly added

Changes in v3: None
Changes in v2: None

 scripts/kconfig/preprocess.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
index c39e30e..246f879 100644
--- a/scripts/kconfig/preprocess.c
+++ b/scripts/kconfig/preprocess.c
@@ -276,6 +276,7 @@ struct variable {
 	char *name;
 	char *value;
 	enum variable_flavor flavor;
+	int exp_count;
 	struct list_head node;
 };
 
@@ -306,8 +307,19 @@ static char *variable_expand(const char *name, int argc, char *argv[],
 		expanded_argv[i] = expand_string_with_args(argv[i],
 							   old_argc, old_argv);
 
+	if (argc == 0 && v->exp_count)
+		pperror("Recursive variable '%s' references itself (eventually)",
+			name);
+
+	if (v->exp_count > 1000)
+		pperror("Too deep recursive expansion");
+
+	v->exp_count++;
+
 	res = expand_string_with_args(v->value, argc, expanded_argv);
 
+	v->exp_count--;
+
 	for (i = 0; i < argc; i++)
 		free(expanded_argv[i]);
 
@@ -337,6 +349,7 @@ void variable_add(const char *name, const char *value,
 
 		v = xmalloc(sizeof(*v));
 		v->name = xstrdup(name);
+		v->exp_count = 0;
 		list_add_tail(&v->node, &variable_list);
 	}
 
-- 
2.7.4

[PATCH v4 19/31] Documentation: kconfig: document a new Kconfig macro language

[Masahiro Yamada]

Add a document for the macro language introduced to Kconfig.

The motivation of this work is to move the compiler option tests to
Kconfig from Makefile.  A number of kernel features require the
compiler support.  Enabling such features blindly in Kconfig ends up
with a lot of nasty build-time testing in Makefiles.  If a chosen
feature turns out unsupported by the compiler, what the build system
can do is either to disable it (silently!) or to forcibly break the
build, despite Kconfig has let the user to enable it.  By moving the
compiler capability tests to Kconfig, features unsupported by the
compiler will be hidden automatically.

This change was strongly prompted by Linus Torvalds.  You can find
his suggestions [1] [2] in ML.  The original idea was to add a new
attribute with 'option shell=...', but I found more generalized text
expansion would make Kconfig more powerful and lovely.  The basic
ideas are from Make, but there are some differences.

[1]: https://lkml.org/lkml/2016/12/9/577
[2]: https://lkml.org/lkml/2018/2/7/527

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4:
 - Update according to the syntax change

Changes in v3:
 - Newly added

Changes in v2: None

 Documentation/kbuild/kconfig-macro-language.txt | 252 ++++++++++++++++++++++++
 MAINTAINERS                                     |   2 +-
 2 files changed, 253 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/kbuild/kconfig-macro-language.txt

diff --git a/Documentation/kbuild/kconfig-macro-language.txt b/Documentation/kbuild/kconfig-macro-language.txt
new file mode 100644
index 0000000..a8dc792
--- /dev/null
+++ b/Documentation/kbuild/kconfig-macro-language.txt
@@ -0,0 +1,252 @@
+Concept
+-------
+
+The basic idea was inspired by Make. When we look at Make, we notice sort of
+two languages in one. One language describes dependency graphs consisting of
+targets and prerequisites. The other is a macro language for performing textual
+substitution.
+
+There is clear distinction between the two language stages. For example, you
+can write a makefile like follows:
+
+    APP := foo
+    SRC := foo.c
+    CC := gcc
+
+    $(APP): $(SRC)
+            $(CC) -o $(APP) $(SRC)
+
+The macro language replaces the variable references with their expanded form,
+and handles as if the source file were input like follows:
+
+    foo: foo.c
+            gcc -o foo foo.c
+
+Then, Make analyzes the dependency graph and determines the targets to be
+updated.
+
+The idea is quite similar in Kconfig - it is possible to describe a Kconfig
+file like this:
+
+    CC := gcc
+
+    config CC_HAS_FOO
+            def_bool $(shell, $(srctree)/scripts/gcc-check-foo.sh $(CC))
+
+The macro language in Kconfig processes the source file into the following
+intermediate:
+
+    config CC_HAS_FOO
+            def_bool y
+
+Then, Kconfig moves onto the evaluation stage to resolve inter-symbol
+dependency as explained in kconfig-language.txt.
+
+
+Variables
+---------
+
+Like in Make, a variable in Kconfig works as a macro variable.  A macro
+variable is expanded "in place" to yield a text string that may then be
+expanded further. To get the value of a variable, enclose the variable name in
+$( ). The parentheses are required even for single-letter variable names; $X is
+a syntax error. The curly brace form as in ${CC} is not supported either.
+
+There are two types of variables: simply expanded variables and recursively
+expanded variables.
+
+A simply expanded variable is defined using the := assignment operator. Its
+righthand side is expanded immediately upon reading the line from the Kconfig
+file.
+
+A recursively expanded variable is defined using the = assignment operator.
+Its righthand side is simply stored as the value of the variable without
+expanding it in any way. Instead, the expansion is performed when the variable
+is used.
+
+There is another type of assignment operator; += is used to append text to a
+variable. The righthand side of += is expanded immediately if the lefthand
+side was originally defined as a simple variable. Otherwise, its evaluation is
+deferred.
+
+The variable reference can take parameters, in the following form:
+
+  $(name,arg1,arg2,arg3)
+
+You can consider the parameterized reference as a function. (more precisely,
+"user-defined function" in the contrast to "built-in function" listed below).
+
+Useful functions must be expanded when they are used since the same function is
+expanded differently if different parameters are passed. Hence, a user-defined
+function is defined using the = assignment operator. The parameters are
+referenced within the body definition with $(1), $(2), etc.
+
+In fact, recursively expanded variables and user-defined functions are the same
+internally. (In other words, "variable" is "function with zero argument".)
+When we say "variable" in a broad sense, it includes "user-defined function".
+
+
+Built-in functions
+------------------
+
+Like Make, Kconfig provides several built-in functions. Every function takes a
+particular number of arguments.
+
+In Make, every built-in function takes at least one argument. Kconfig allows
+zero argument for built-in functions, such as $(fileno), $(lineno). You could
+consider those as "built-in variable", but it is just a matter of how we call
+it after all. Let's say "built-in function" here to refer to natively supported
+functionality.
+
+Kconfig currently supports the following built-in functions.
+
+ - $(shell,command)
+
+  The "shell" function accepts a single argument that is expanded and passed
+  to a subshell for execution. The standard output of the command is then read
+  and returned as the value of the function. Every newline in the output is
+  replaced with a space. Any trailing newlines are deleted. The standard error
+  is not returned, nor is any program exit status.
+
+ - $(info,text)
+
+  The "info" function takes a single argument and prints it to stdout.
+  It evaluates to an empty string.
+
+ - $(warning,text)
+
+  The "warning" function is similar to "info" except that it sends its argument
+  to stderr and prefixes the output with the name of the current Kconfig file
+  and the current line number.
+
+ - $(error,text)
+
+  The "error" function is similar to "warning", but it terminates the parsing
+  immediately.
+
+ - $(if,condition,then-part[,else-part])
+
+  The "if" function takes two or three arguments ('else-part' is optional).
+  Depending on the value of the condition part, the argument to be expanded
+  is selected. The condition is true if its expansion contains any characters
+  except whitespaces. In this case, the then-part is expanded. Otherwise, the
+  else-part is expanded.
+
+  Note:
+  In Make, the condition is true if it contains any characters including
+  whitespaces, which is why $(strip ...) is sometimes necessary in the
+  condition part. Kconfig changed the behavior to make it handier.
+
+ - $(filename)
+
+  The 'filename' takes no argument, and $(filename) is expanded to a file name
+  being parsed.
+
+ - $(lineno)
+
+  The 'lineno' takes no argument, and $(lineno) is expanded to a line number
+  being parsed.
+
+
+Difference of function call syntax
+----------------------------------
+
+Kconfig adopts Make-like macro language, but the function call syntax is
+slightly different.
+
+A function call in Make looks like follows:
+
+  $(func-name arg1,arg2,arg3)
+
+The function name and the first argument are separated by at least one
+whitespace. Then, leading whitespaces are trimmed from the first argument,
+but whitespaces in the other arguments are kept. You need to use a kind of
+trick to start the first parameter with spaces. For example, if you want
+to make "info" function print "  hello", you can write like follows:
+
+  $(info $(space)$(space)hello)
+
+Kconfig uses only commas for delimiters, and keeps all whitespaces in the
+function call. Some people prefer putting a space after each comma delimiter:
+
+  $(func-name, arg1, arg2, arg3)
+
+In this case, "func-name" will receive " arg1", " arg2", " arg3". The presence
+of leading spaces may really matter depending on the function. The same applies
+to Make - for example, $(subst .c, .o, $(sources)) is a typical mistake.
+
+In Make, a user-defined function is referenced by using a built-in function,
+'call', like this:
+
+    $(call my-func,arg1,arg2,arg3)
+
+Kconfig invokes user-defined functions and built-in functions in the same way.
+The omission of 'call' makes the syntax shorter.
+
+In Make, some functions exceptionally treat commas verbatim instead of argument
+separators. For example, $(shell echo hello, world) evaluates to "hello, world".
+Likewise, $(info hello, world) prints "hello, world" to stdout. You could say
+this is _useful_ inconsistency.
+
+For simpler implementation and grammatical consistency, Kconfig always treats
+commas that appear in the $( ) form as delimiters. It means
+
+  $(shell, echo hello, world)
+
+is an error because it is passing two parameters where the 'shell' function
+accepts only one. To pass commas in arguments, you can use the following trick:
+
+  comma := ,
+  $(shell, echo hello$(comma) world)
+
+
+Caveats
+-------
+
+A variable (or function) cannot be expanded across tokens. So, you cannot use
+a variable as a shorthand for an expression that consists of multiple tokens.
+The following works:
+
+    RANGE_MIN := 1
+    RANGE_MAX := 3
+
+    config FOO
+            int "foo"
+            range $(RANGE_MIN) $(RANGE_MAX)
+
+But, the following does not work:
+
+    RANGES := 1 3
+
+    config FOO
+            int "foo"
+            range $(RANGES)
+
+A variable cannot be expanded to any keyword in Kconfig.  The following does
+not work:
+
+    MY_TYPE := tristate
+
+    config FOO
+            $(MY_TYPE) "foo"
+            default y
+
+Obviously from the design, $(shell command) is expanded in the textual
+substitution phase. You cannot pass symbols to the 'shell' function.
+The following does not work as expected.
+
+    config ENDIAN_FLAG
+            string
+            default "-mbig-endian" if CPU_BIG_ENDIAN
+            default "-mlittle-endian" if CPU_LITTLE_ENDIAN
+
+    config CC_HAS_ENDIAN_FLAG
+            def_bool $(shell $(srctree)/scripts/gcc-check-flag ENDIAN_FLAG)
+
+Instead, you can do like follows so that any function call is statically
+expanded.
+
+    config CC_HAS_ENDIAN_FLAG
+            bool
+            default $(shell $(srctree)/scripts/gcc-check-flag -mbig-endian) if CPU_BIG_ENDIAN
+            default $(shell $(srctree)/scripts/gcc-check-flag -mlittle-endian) if CPU_LITTLE_ENDIAN
diff --git a/MAINTAINERS b/MAINTAINERS
index 58b9861..b7d7ae61 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7632,7 +7632,7 @@ M:	Masahiro Yamada <yamada.masahiro@socionext.com>
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git kconfig
 L:	linux-kbuild@vger.kernel.org
 S:	Maintained
-F:	Documentation/kbuild/kconfig-language.txt
+F:	Documentation/kbuild/kconfig*
 F:	scripts/kconfig/
 
 KDUMP
-- 
2.7.4

[PATCH v4 20/31] kconfig: test: add Kconfig macro language tests

[Masahiro Yamada]

Here are the test cases I used for developing the text expansion
feature.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4:
  - Adjust for the change in function call syntax.
  - Remove unnecessarily tricky tests.

Changes in v3:
  - newly added

Changes in v2: None

 .../kconfig/tests/preprocess/builtin_func/Kconfig  | 29 +++++++++++++
 .../tests/preprocess/builtin_func/__init__.py      |  8 ++++
 .../tests/preprocess/builtin_func/expected_stderr  |  7 ++++
 .../tests/preprocess/builtin_func/expected_stdout  |  1 +
 .../tests/preprocess/circular_expansion/Kconfig    |  3 ++
 .../preprocess/circular_expansion/__init__.py      | 10 +++++
 .../preprocess/circular_expansion/expected_stderr  |  1 +
 scripts/kconfig/tests/preprocess/escape/Kconfig    | 21 ++++++++++
 .../kconfig/tests/preprocess/escape/__init__.py    |  7 ++++
 .../tests/preprocess/escape/expected_stderr        |  5 +++
 scripts/kconfig/tests/preprocess/variable/Kconfig  | 48 ++++++++++++++++++++++
 .../kconfig/tests/preprocess/variable/__init__.py  |  7 ++++
 .../tests/preprocess/variable/expected_stderr      |  9 ++++
 13 files changed, 156 insertions(+)
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/expected_stderr
 create mode 100644 scripts/kconfig/tests/preprocess/builtin_func/expected_stdout
 create mode 100644 scripts/kconfig/tests/preprocess/circular_expansion/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/circular_expansion/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/circular_expansion/expected_stderr
 create mode 100644 scripts/kconfig/tests/preprocess/escape/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/escape/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/escape/expected_stderr
 create mode 100644 scripts/kconfig/tests/preprocess/variable/Kconfig
 create mode 100644 scripts/kconfig/tests/preprocess/variable/__init__.py
 create mode 100644 scripts/kconfig/tests/preprocess/variable/expected_stderr

diff --git a/scripts/kconfig/tests/preprocess/builtin_func/Kconfig b/scripts/kconfig/tests/preprocess/builtin_func/Kconfig
new file mode 100644
index 0000000..5f51135
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/builtin_func/Kconfig
@@ -0,0 +1,29 @@
+# 'info' prints the argument to stdout.
+$(info,hello world 0)
+
+# 'warning' is similar, but it sends its argument to stderr,
+# and the message is prefixed with the current file name and line number.
+$(warning,hello world 1)
+
+# 'shell' executes a command, and returns its stdout.
+$(warning,$(shell,echo hello world 3))
+
+# Every newline in the output is replaced with a space,
+# but any trailing newlines are deleted.
+$(warning,$(shell,printf 'hello\nworld\n\n4\n\n\n'))
+
+# 'filename' is expanded to the currently parsed file name,
+# 'lineno' to the line number.
+$(warning,filename=$(filename))
+$(warning,lineno=$(lineno))
+
+# 'if' selects which argument to expand
+# depending on the condition part.
+cond := 1
+$(warning,$(if,$(cond),y,n))
+cond :=
+$(warning,$(if,$(cond),y,n))
+
+# 'if' delays the expansion of arguments.
+# In the following, the 'error' function should not be evaluated.
+$(if,,$(error,this should not be expanded))
diff --git a/scripts/kconfig/tests/preprocess/builtin_func/__init__.py b/scripts/kconfig/tests/preprocess/builtin_func/__init__.py
new file mode 100644
index 0000000..ec7c3e2
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/builtin_func/__init__.py
@@ -0,0 +1,8 @@
+"""
+Built-in function tests.
+"""
+
+def test(conf):
+    assert conf.oldaskconfig() == 0
+    assert conf.stdout_contains('expected_stdout')
+    assert conf.stderr_matches('expected_stderr')
diff --git a/scripts/kconfig/tests/preprocess/builtin_func/expected_stderr b/scripts/kconfig/tests/preprocess/builtin_func/expected_stderr
new file mode 100644
index 0000000..a4f27a6
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/builtin_func/expected_stderr
@@ -0,0 +1,7 @@
+Kconfig:6: hello world 1
+Kconfig:9: hello world 3
+Kconfig:13: hello world  4
+Kconfig:17: filename=Kconfig
+Kconfig:18: lineno=18
+Kconfig:23: y
+Kconfig:25: n
diff --git a/scripts/kconfig/tests/preprocess/builtin_func/expected_stdout b/scripts/kconfig/tests/preprocess/builtin_func/expected_stdout
new file mode 100644
index 0000000..82de3a7
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/builtin_func/expected_stdout
@@ -0,0 +1 @@
+hello world 0
diff --git a/scripts/kconfig/tests/preprocess/circular_expansion/Kconfig b/scripts/kconfig/tests/preprocess/circular_expansion/Kconfig
new file mode 100644
index 0000000..54debf1
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/circular_expansion/Kconfig
@@ -0,0 +1,3 @@
+X = $(Y)
+Y = $(X)
+$(info $(X))
diff --git a/scripts/kconfig/tests/preprocess/circular_expansion/__init__.py b/scripts/kconfig/tests/preprocess/circular_expansion/__init__.py
new file mode 100644
index 0000000..21413d9
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/circular_expansion/__init__.py
@@ -0,0 +1,10 @@
+"""
+Detect circular variable expansion.
+
+If a recursively expanded variable references itself (eventually),
+it should fail with an error message.
+"""
+
+def test(conf):
+    assert conf.oldaskconfig() != 0
+    assert conf.stderr_matches('expected_stderr')
diff --git a/scripts/kconfig/tests/preprocess/circular_expansion/expected_stderr b/scripts/kconfig/tests/preprocess/circular_expansion/expected_stderr
new file mode 100644
index 0000000..c638a8c
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/circular_expansion/expected_stderr
@@ -0,0 +1 @@
+Kconfig:3: Recursive variable 'X' references itself (eventually)
diff --git a/scripts/kconfig/tests/preprocess/escape/Kconfig b/scripts/kconfig/tests/preprocess/escape/Kconfig
new file mode 100644
index 0000000..b38c897
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/escape/Kconfig
@@ -0,0 +1,21 @@
+# You can not pass commas directly to a function since they are treated as
+# delimiters. You can use the following trick to do so.
+comma   := ,
+$(warning,hello$(comma) world)
+
+# Like Make, single quotes, double quotes, spaces are treated verbatim.
+# The following prints the text as-is.
+$(warning,' " '"   ' ''' "'")
+
+# You can use '$$' to escape '$' itself
+$(warning,$$)
+
+# The escaped '$' loses its special meaning. The following should print '$(X)'.
+# Do not expand '$(X)' even further.
+$(warning,$$(X))
+
+# In Make, a variable name can contain almost any characters.  The only
+# disallowed characters are : # and =
+# '$' can be used as a variable name in Kconfig, although it is nasty
+$$ = nasty
+$(warning,$($$))
diff --git a/scripts/kconfig/tests/preprocess/escape/__init__.py b/scripts/kconfig/tests/preprocess/escape/__init__.py
new file mode 100644
index 0000000..602861e
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/escape/__init__.py
@@ -0,0 +1,7 @@
+"""
+Escape sequence tests.
+"""
+
+def test(conf):
+    assert conf.oldaskconfig() == 0
+    assert conf.stderr_matches('expected_stderr')
diff --git a/scripts/kconfig/tests/preprocess/escape/expected_stderr b/scripts/kconfig/tests/preprocess/escape/expected_stderr
new file mode 100644
index 0000000..6a80134
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/escape/expected_stderr
@@ -0,0 +1,5 @@
+Kconfig:4: hello, world
+Kconfig:8: ' " '"   ' ''' "'"
+Kconfig:11: $
+Kconfig:15: $(X)
+Kconfig:21: nasty
diff --git a/scripts/kconfig/tests/preprocess/variable/Kconfig b/scripts/kconfig/tests/preprocess/variable/Kconfig
new file mode 100644
index 0000000..271834e
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/variable/Kconfig
@@ -0,0 +1,48 @@
+# Simply expanded variable.
+X := 1
+SIMPLE := $(X)
+X := 2
+$(warning,SIMPLE = $(SIMPLE))
+
+# Recursively expanded variable.
+X := 1
+RECURSIVE = $(X)
+X := 2
+$(warning,RECURSIVE = $(RECURSIVE))
+
+# Append something to a simply expanded variable.
+Y := 3
+SIMPLE += $(Y)
+Y := 4
+$(warning,SIMPLE = $(SIMPLE))
+
+# Append something to a recursively expanded variable.
+Y := 3
+RECURSIVE += $(Y)
+Y := 4
+$(warning,RECURSIVE = $(RECURSIVE))
+
+# Use += operator to an undefined variable.
+# This works as a recursively expanded variable.
+Y := 3
+UNDEFINED_VARIABLE += $(Y)
+Y := 4
+$(warning,UNDEFINED_VARIABLE = $(UNDEFINED_VARIABLE))
+
+# You can use variable references for the lefthand side of assignment statement.
+X := A
+Y := B
+$(X)$(Y) := 5
+$(warning,AB = $(AB))
+
+# User-defined function.
+greeting = $(1), my name is $(2).
+$(warning,$(greeting,Hello,John))
+
+# The number of arguments is not checked for user-defined functions.
+# If some arguments are optional, it is useful to pass fewer parameters.
+# $(2) will be blank in this case.
+$(warning,$(greeting,Hello))
+
+# Unreferenced parameters are just ignored.
+$(warning,$(greeting,Hello,John,ignored,ignored))
diff --git a/scripts/kconfig/tests/preprocess/variable/__init__.py b/scripts/kconfig/tests/preprocess/variable/__init__.py
new file mode 100644
index 0000000..5b2c1be
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/variable/__init__.py
@@ -0,0 +1,7 @@
+"""
+Variable and user-defined function tests.
+"""
+
+def test(conf):
+    assert conf.oldaskconfig() == 0
+    assert conf.stderr_matches('expected_stderr')
diff --git a/scripts/kconfig/tests/preprocess/variable/expected_stderr b/scripts/kconfig/tests/preprocess/variable/expected_stderr
new file mode 100644
index 0000000..bddfa3b
--- /dev/null
+++ b/scripts/kconfig/tests/preprocess/variable/expected_stderr
@@ -0,0 +1,9 @@
+Kconfig:5: SIMPLE = 1
+Kconfig:11: RECURSIVE = 2
+Kconfig:17: SIMPLE = 1 3
+Kconfig:23: RECURSIVE = 2 4
+Kconfig:30: UNDEFINED_VARIABLE = 4
+Kconfig:36: AB = 5
+Kconfig:40: Hello, my name is John.
+Kconfig:45: Hello, my name is .
+Kconfig:48: Hello, my name is John.
-- 
2.7.4

[PATCH v4 21/31] kconfig: show compiler version text in the top comment

[Masahiro Yamada]

The kernel configuration phase is now tightly coupled with the compiler
in use.  It will be nice to show the compiler information in Kconfig.

The compiler information will be displayed like this:

  $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- config
  scripts/kconfig/conf  --oldaskconfig Kconfig
  *
  * Linux/arm64 4.16.0-rc1 Kernel Configuration
  *
  *
  * Compiler: aarch64-linux-gnu-gcc (Linaro GCC 7.2-2017.11) 7.2.1 20171011
  *
  *
  * General setup
  *
  Compile also drivers which will not load (COMPILE_TEST) [N/y/?]

If you use GUI methods such as menuconfig, it will be displayed in the
top menu.

This is simply implemented by using the 'comment' statement.  So, it
will be saved into the .config file as well.

This commit has a very important meaning.  If the compiler is upgraded,
Kconfig must be re-run since different compilers have different sets
of supported options.

All referenced environments are written to include/config/auto.conf.cmd
so that any environment change triggers syncconfig, and prompt the user
to input new values if needed.

With this commit, something like follows will be added to
include/config/auto.conf.cmd

  ifneq "$(CC_VERSION_TEXT)" "aarch64-linux-gnu-gcc (Linaro GCC 7.2-2017.11) 7.2.1 20171011"
  include/config/auto.conf: FORCE
  endif

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
---

Changes in v4:
  - Add comment to arch/x86/um/Kconfig as well

Changes in v3: None
Changes in v2: None

 Kconfig             | 2 ++
 Makefile            | 2 ++
 arch/x86/um/Kconfig | 2 ++
 3 files changed, 6 insertions(+)

diff --git a/Kconfig b/Kconfig
index 4af1b42..5b55d87 100644
--- a/Kconfig
+++ b/Kconfig
@@ -5,4 +5,6 @@
 #
 mainmenu "Linux/$(ARCH) $(KERNELVERSION) Kernel Configuration"
 
+comment "Compiler: $(CC_VERSION_TEXT)"
+
 source "arch/$(SRCARCH)/Kconfig"
diff --git a/Makefile b/Makefile
index 01b2211..7529825 100644
--- a/Makefile
+++ b/Makefile
@@ -442,6 +442,8 @@ export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_LDFLAGS_MODULE
 export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL
 export KBUILD_ARFLAGS
 
+export CC_VERSION_TEXT := $(shell $(CC) --version | head -n 1)
+
 # When compiling out-of-tree modules, put MODVERDIR in the module
 # tree rather than in the kernel tree. The kernel tree might
 # even be read-only.
diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
index 6a15c4d..a992f8e 100644
--- a/arch/x86/um/Kconfig
+++ b/arch/x86/um/Kconfig
@@ -1,6 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
 mainmenu "User Mode Linux/$(SUBARCH) $(KERNELVERSION) Kernel Configuration"
 
+comment "Compiler: $(CC_VERSION_TEXT)"
+
 source "arch/um/Kconfig.common"
 
 menu "UML-specific options"
-- 
2.7.4

[PATCH v4 22/31] kconfig: add basic helper macros to scripts/Kconfig.include

[Masahiro Yamada]

Kconfig got text processing tools like we see in Make.  Add Kconfig
helper macros to scripts/Kconfig.include like we collect Makefile
macros in scripts/Kbuild.include.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Ulf Magnusson <ulfalizer@gmail.com>
---

Changes in v4:
  - source scripts/Kconfig.include
    from arch/x86/um/Kconfig since UML is special
  - Use { $(1); } for 'success' macro for better performance
  - Use -E instead of -c for 'cc-option' macro for better performance
  - Add 'if-success' macro

Changes in v3:
  - Move helpers to scripts/Kconfig.include

Changes in v2: None

 Kconfig                 |  2 ++
 MAINTAINERS             |  1 +
 arch/x86/um/Kconfig     |  2 ++
 scripts/Kconfig.include | 24 ++++++++++++++++++++++++
 4 files changed, 29 insertions(+)
 create mode 100644 scripts/Kconfig.include

diff --git a/Kconfig b/Kconfig
index 5b55d87..a90d9f9 100644
--- a/Kconfig
+++ b/Kconfig
@@ -7,4 +7,6 @@ mainmenu "Linux/$(ARCH) $(KERNELVERSION) Kernel Configuration"
 
 comment "Compiler: $(CC_VERSION_TEXT)"
 
+source "scripts/Kconfig.include"
+
 source "arch/$(SRCARCH)/Kconfig"
diff --git a/MAINTAINERS b/MAINTAINERS
index b7d7ae61..1667cee 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7634,6 +7634,7 @@ L:	linux-kbuild@vger.kernel.org
 S:	Maintained
 F:	Documentation/kbuild/kconfig*
 F:	scripts/kconfig/
+F:	scripts/Kconfig.include
 
 KDUMP
 M:	Dave Young <dyoung@redhat.com>
diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
index a992f8e..9d529f2 100644
--- a/arch/x86/um/Kconfig
+++ b/arch/x86/um/Kconfig
@@ -3,6 +3,8 @@ mainmenu "User Mode Linux/$(SUBARCH) $(KERNELVERSION) Kernel Configuration"
 
 comment "Compiler: $(CC_VERSION_TEXT)"
 
+source "scripts/Kconfig.include"
+
 source "arch/um/Kconfig.common"
 
 menu "UML-specific options"
diff --git a/scripts/Kconfig.include b/scripts/Kconfig.include
new file mode 100644
index 0000000..b1502f4
--- /dev/null
+++ b/scripts/Kconfig.include
@@ -0,0 +1,24 @@
+# Kconfig helper macros
+
+# Convenient variables
+comma   := ,
+quote   := "
+squote  := '
+empty   :=
+space   := $(empty) $(empty)
+
+# $(if-success,<command>,<then-part>,<else-part>)
+# <then-part> if <command> exits with 0, <else-part> otherwise.
+if-success = $(shell, { $(1); } >/dev/null 2>&1 && echo "$(2)" || echo "$(3)")
+
+# $(success,<command>)
+# y if <command> exits with 0, n otherwise
+success = $(if-success, $(1),y,n)
+
+# $(cc-option,<flag>)
+# y if the compiler supports <flag>, n otherwise
+cc-option = $(success, $(CC) -Werror $(1) -E -x c /dev/null -o /dev/null)
+
+# $(ld-option,<flag>)
+# y if the linker supports <flag>, n otherwise
+ld-option = $(success, $(LD) -v $(1))
-- 
2.7.4

[PATCH v4 23/31] stack-protector: test compiler capability in Kconfig and drop AUTO mode

[Masahiro Yamada]

Move the test for -fstack-protector(-strong) option to Kconfig.

If the compiler does not support the option, the corresponding menu
is automatically hidden.  If STRONG is not supported, it will fall
back to REGULAR.  If REGULAR is not supported, it will be disabled.
This means, AUTO is implicitly handled by the dependency solver of
Kconfig, hence removed.

I also turned the 'choice' into only two boolean symbols.  The use of
'choice' is not a good idea here, because all of all{yes,mod,no}config
would choose the first visible value, while we want allnoconfig to
disable as many features as possible.

X86 has additional shell scripts in case the compiler supports those
options, but generates broken code.  I added CC_HAS_SANE_STACKPROTECTOR
to test this.  I had to add -m32 to gcc-x86_32-has-stack-protector.sh
to make it work correctly.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4: None
Changes in v3: None
Changes in v2: None

 Makefile                                  | 93 ++-----------------------------
 arch/Kconfig                              | 32 ++++-------
 arch/x86/Kconfig                          | 11 +++-
 scripts/gcc-x86_32-has-stack-protector.sh |  7 +--
 scripts/gcc-x86_64-has-stack-protector.sh |  5 --
 5 files changed, 28 insertions(+), 120 deletions(-)

diff --git a/Makefile b/Makefile
index 7529825..12c222d 100644
--- a/Makefile
+++ b/Makefile
@@ -672,55 +672,11 @@ ifneq ($(CONFIG_FRAME_WARN),0)
 KBUILD_CFLAGS += $(call cc-option,-Wframe-larger-than=${CONFIG_FRAME_WARN})
 endif
 
-# This selects the stack protector compiler flag. Testing it is delayed
-# until after .config has been reprocessed, in the prepare-compiler-check
-# target.
-ifdef CONFIG_CC_STACKPROTECTOR_AUTO
-  stackp-flag := $(call cc-option,-fstack-protector-strong,$(call cc-option,-fstack-protector))
-  stackp-name := AUTO
-else
-ifdef CONFIG_CC_STACKPROTECTOR_REGULAR
-  stackp-flag := -fstack-protector
-  stackp-name := REGULAR
-else
-ifdef CONFIG_CC_STACKPROTECTOR_STRONG
-  stackp-flag := -fstack-protector-strong
-  stackp-name := STRONG
-else
-  # If either there is no stack protector for this architecture or
-  # CONFIG_CC_STACKPROTECTOR_NONE is selected, we're done, and $(stackp-name)
-  # is empty, skipping all remaining stack protector tests.
-  #
-  # Force off for distro compilers that enable stack protector by default.
-  KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
-endif
-endif
-endif
-# Find arch-specific stack protector compiler sanity-checking script.
-ifdef stackp-name
-ifneq ($(stackp-flag),)
-  stackp-path := $(srctree)/scripts/gcc-$(SRCARCH)_$(BITS)-has-stack-protector.sh
-  stackp-check := $(wildcard $(stackp-path))
-  # If the wildcard test matches a test script, run it to check functionality.
-  ifdef stackp-check
-    ifneq ($(shell $(CONFIG_SHELL) $(stackp-check) $(CC) $(KBUILD_CPPFLAGS) $(biarch)),y)
-      stackp-broken := y
-    endif
-  endif
-  ifndef stackp-broken
-    # If the stack protector is functional, enable code that depends on it.
-    KBUILD_CPPFLAGS += -DCONFIG_CC_STACKPROTECTOR
-    # Either we've already detected the flag (for AUTO) or we'll fail the
-    # build in the prepare-compiler-check rule (for specific flag).
-    KBUILD_CFLAGS += $(stackp-flag)
-  else
-    # We have to make sure stack protector is unconditionally disabled if
-    # the compiler is broken (in case we're going to continue the build in
-    # AUTO mode).
-    KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
-  endif
-endif
-endif
+stackp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector
+stackp-flags-$(CONFIG_CC_STACKPROTECTOR)          := -fstack-protector
+stackp-flags-$(CONFIG_CC_STACKPROTECTOR_STRONG)   := -fstack-protector-strong
+
+KBUILD_CFLAGS += $(stackp-flags-y)
 
 ifeq ($(cc-name),clang)
 KBUILD_CPPFLAGS += $(call cc-option,-Qunused-arguments,)
@@ -1099,7 +1055,7 @@ endif
 # prepare2 creates a makefile if using a separate output directory.
 # From this point forward, .config has been reprocessed, so any rules
 # that need to depend on updated CONFIG_* values can be checked here.
-prepare2: prepare3 prepare-compiler-check outputmakefile asm-generic
+prepare2: prepare3 outputmakefile asm-generic
 
 prepare1: prepare2 $(version_h) $(autoksyms_h) include/generated/utsrelease.h \
                    include/config/auto.conf
@@ -1125,43 +1081,6 @@ uapi-asm-generic:
 PHONY += prepare-objtool
 prepare-objtool: $(objtool_target)
 
-# Check for CONFIG flags that require compiler support. Abort the build
-# after .config has been processed, but before the kernel build starts.
-#
-# For security-sensitive CONFIG options, we don't want to fallback and/or
-# silently change which compiler flags will be used, since that leads to
-# producing kernels with different security feature characteristics
-# depending on the compiler used. (For example, "But I selected
-# CC_STACKPROTECTOR_STRONG! Why did it build with _REGULAR?!")
-PHONY += prepare-compiler-check
-prepare-compiler-check: FORCE
-# Make sure compiler supports requested stack protector flag.
-ifdef stackp-name
-  # Warn about CONFIG_CC_STACKPROTECTOR_AUTO having found no option.
-  ifeq ($(stackp-flag),)
-	@echo CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-		  Compiler does not support any known stack-protector >&2
-  else
-  # Fail if specifically requested stack protector is missing.
-  ifeq ($(call cc-option, $(stackp-flag)),)
-	@echo Cannot use CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-		  $(stackp-flag) not supported by compiler >&2 && exit 1
-  endif
-  endif
-endif
-# Make sure compiler does not have buggy stack-protector support. If a
-# specific stack-protector was requested, fail the build, otherwise warn.
-ifdef stackp-broken
-  ifeq ($(stackp-name),AUTO)
-	@echo CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-                  $(stackp-flag) available but compiler is broken: disabling >&2
-  else
-	@echo Cannot use CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-                  $(stackp-flag) available but compiler is broken >&2 && exit 1
-  endif
-endif
-	@:
-
 # Generate some files
 # ---------------------------------------------------------------------------
 
diff --git a/arch/Kconfig b/arch/Kconfig
index 75dd23a..ca32950 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -539,13 +539,16 @@ config HAVE_CC_STACKPROTECTOR
 	bool
 	help
 	  An arch should select this symbol if:
-	  - its compiler supports the -fstack-protector option
 	  - it has implemented a stack canary (e.g. __stack_chk_guard)
 
-choice
-	prompt "Stack Protector buffer overflow detection"
+config CC_HAS_STACKPROTECTOR_NONE
+	def_bool $(cc-option, -fno-stack-protector)
+
+config CC_STACKPROTECTOR
+	bool "Stack Protector buffer overflow detection"
 	depends on HAVE_CC_STACKPROTECTOR
-	default CC_STACKPROTECTOR_AUTO
+	depends on $(cc-option, -fstack-protector)
+	default y
 	help
 	  This option turns on the "stack-protector" GCC feature. This
 	  feature puts, at the beginning of functions, a canary value on
@@ -555,14 +558,6 @@ choice
 	  overwrite the canary, which gets detected and the attack is then
 	  neutralized via a kernel panic.
 
-config CC_STACKPROTECTOR_NONE
-	bool "None"
-	help
-	  Disable "stack-protector" GCC feature.
-
-config CC_STACKPROTECTOR_REGULAR
-	bool "Regular"
-	help
 	  Functions will have the stack-protector canary logic added if they
 	  have an 8-byte or larger character array on the stack.
 
@@ -574,7 +569,10 @@ config CC_STACKPROTECTOR_REGULAR
 	  by about 0.3%.
 
 config CC_STACKPROTECTOR_STRONG
-	bool "Strong"
+	bool "Strong Stack Protector"
+	depends on CC_STACKPROTECTOR
+	depends on $(cc-option, -fstack-protector-strong)
+	default y
 	help
 	  Functions will have the stack-protector canary logic added in any
 	  of the following conditions:
@@ -592,14 +590,6 @@ config CC_STACKPROTECTOR_STRONG
 	  about 20% of all kernel functions, which increases the kernel code
 	  size by about 2%.
 
-config CC_STACKPROTECTOR_AUTO
-	bool "Automatic"
-	help
-	  If the compiler supports it, the best available stack-protector
-	  option will be chosen.
-
-endchoice
-
 config LD_DEAD_CODE_DATA_ELIMINATION
 	bool
 	help
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 2236505..8e9eb75 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -126,7 +126,7 @@ config X86
 	select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64
 	select HAVE_ARCH_VMAP_STACK		if X86_64
 	select HAVE_ARCH_WITHIN_STACK_FRAMES
-	select HAVE_CC_STACKPROTECTOR
+	select HAVE_CC_STACKPROTECTOR		if CC_HAS_SANE_STACKPROTECTOR
 	select HAVE_CMPXCHG_DOUBLE
 	select HAVE_CMPXCHG_LOCAL
 	select HAVE_CONTEXT_TRACKING		if X86_64
@@ -345,6 +345,15 @@ config PGTABLE_LEVELS
 	default 2
 
 source "init/Kconfig"
+
+config CC_HAS_SANE_STACKPROTECTOR
+	bool
+	default $(success, $(srctree)/scripts/gcc-x86_64-has-stack-protector.sh $(CC)) if 64BIT
+	default $(success, $(srctree)/scripts/gcc-x86_32-has-stack-protector.sh $(CC))
+	help
+	   We have to make sure stack protector is unconditionally disabled if
+	   the compiler produces broken code.
+
 source "kernel/Kconfig.freezer"
 
 menu "Processor type and features"
diff --git a/scripts/gcc-x86_32-has-stack-protector.sh b/scripts/gcc-x86_32-has-stack-protector.sh
index 6b2aeef..f5c1194 100755
--- a/scripts/gcc-x86_32-has-stack-protector.sh
+++ b/scripts/gcc-x86_32-has-stack-protector.sh
@@ -1,9 +1,4 @@
 #!/bin/sh
 # SPDX-License-Identifier: GPL-2.0
 
-echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -O0 -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
-if [ "$?" -eq "0" ] ; then
-	echo y
-else
-	echo n
-fi
+echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -m32 -O0 -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
diff --git a/scripts/gcc-x86_64-has-stack-protector.sh b/scripts/gcc-x86_64-has-stack-protector.sh
index 4a48bdc..3755af0 100755
--- a/scripts/gcc-x86_64-has-stack-protector.sh
+++ b/scripts/gcc-x86_64-has-stack-protector.sh
@@ -2,8 +2,3 @@
 # SPDX-License-Identifier: GPL-2.0
 
 echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -O0 -mcmodel=kernel -fno-PIE -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
-if [ "$?" -eq "0" ] ; then
-	echo y
-else
-	echo n
-fi
-- 
2.7.4

[PATCH v4 24/31] kconfig: add CC_IS_GCC and GCC_VERSION

[Masahiro Yamada]

This will be useful to specify the required compiler version,
like this:

config FOO
        bool "Use Foo"
        depends on GCC_VERSION >= 40800
        help
          This feature requires GCC 4.8 or newer.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
---

Changes in v4: None
Changes in v3: None
Changes in v2: None

 init/Kconfig | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/init/Kconfig b/init/Kconfig
index 66f0463..c6ac856 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -8,6 +8,14 @@ config DEFCONFIG_LIST
 	default ARCH_DEFCONFIG
 	default "arch/$(ARCH)/defconfig"
 
+config CC_IS_GCC
+	def_bool $(success, $(CC) --version | grep -q gcc)
+
+config GCC_VERSION
+	int
+	default $(shell, $(srctree)/scripts/gcc-version.sh -p $(CC) | sed 's/^0*//') if CC_IS_GCC
+	default 0
+
 config CONSTRUCTORS
 	bool
 	depends on !UML
-- 
2.7.4

[PATCH v4 25/31] kconfig: add CC_IS_CLANG and CLANG_VERSION

[Masahiro Yamada]

This will be useful to describe the clang version dependency.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
---

Changes in v4: None
Changes in v3: None
Changes in v2: None

 init/Kconfig             |  7 +++++++
 scripts/clang-version.sh | 18 ++++--------------
 2 files changed, 11 insertions(+), 14 deletions(-)

diff --git a/init/Kconfig b/init/Kconfig
index c6ac856..3aeb37a 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -16,6 +16,13 @@ config GCC_VERSION
 	default $(shell, $(srctree)/scripts/gcc-version.sh -p $(CC) | sed 's/^0*//') if CC_IS_GCC
 	default 0
 
+config CC_IS_CLANG
+	def_bool $(success, $(CC) --version | grep -q clang)
+
+config CLANG_VERSION
+	int
+	default $(shell, $(srctree)/scripts/clang-version.sh $(CC))
+
 config CONSTRUCTORS
 	bool
 	depends on !UML
diff --git a/scripts/clang-version.sh b/scripts/clang-version.sh
index 9780efa..dbf0a31 100755
--- a/scripts/clang-version.sh
+++ b/scripts/clang-version.sh
@@ -10,24 +10,14 @@
 # clang-5.0.1 etc.
 #
 
-if [ "$1" = "-p" ] ; then
-	with_patchlevel=1;
-	shift;
-fi
-
 compiler="$*"
 
-if [ ${#compiler} -eq 0 ]; then
-	echo "Error: No compiler specified."
-	printf "Usage:\n\t$0 <clang-command>\n"
+if !( $compiler --version | grep -q clang) ; then
+	echo 0
 	exit 1
 fi
 
 MAJOR=$(echo __clang_major__ | $compiler -E -x c - | tail -n 1)
 MINOR=$(echo __clang_minor__ | $compiler -E -x c - | tail -n 1)
-if [ "x$with_patchlevel" != "x" ] ; then
-	PATCHLEVEL=$(echo __clang_patchlevel__ | $compiler -E -x c - | tail -n 1)
-	printf "%02d%02d%02d\\n" $MAJOR $MINOR $PATCHLEVEL
-else
-	printf "%02d%02d\\n" $MAJOR $MINOR
-fi
+PATCHLEVEL=$(echo __clang_patchlevel__ | $compiler -E -x c - | tail -n 1)
+printf "%d%02d%02d\\n" $MAJOR $MINOR $PATCHLEVEL
-- 
2.7.4

[PATCH v4 26/31] gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT

[Masahiro Yamada]

CONFIG_GCOV_FORMAT_AUTODETECT compiles either gcc_3_4.c or gcc_4_7.c
according to your GCC version.

We can achieve the equivalent behavior by setting reasonable dependency
with the knowledge of the compiler version.

If GCC older than 4.7 is used, GCOV_FORMAT_3_4 is the default, but users
are still allowed to select GCOV_FORMAT_4_7 in case the newer format is
back-ported.

On the other hand, If GCC 4.7 or newer is used, there is no reason to
use GCOV_FORMAT_3_4, so it should be hidden.

If you downgrade the compiler to GCC 4.7 or older, oldconfig/syncconfig
will display a prompt for the choice because GCOV_FORMAT_3_4 becomes
visible as a new symbol.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Peter Oberparleiter <oberpar@linux.vnet.ibm.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
---

Changes in v4: None
Changes in v3: None
Changes in v2: None

 kernel/gcov/Kconfig  | 17 +++++------------
 kernel/gcov/Makefile |  2 --
 2 files changed, 5 insertions(+), 14 deletions(-)

diff --git a/kernel/gcov/Kconfig b/kernel/gcov/Kconfig
index 1276aab..1e3823f 100644
--- a/kernel/gcov/Kconfig
+++ b/kernel/gcov/Kconfig
@@ -53,23 +53,16 @@ config GCOV_PROFILE_ALL
 choice
 	prompt "Specify GCOV format"
 	depends on GCOV_KERNEL
-	default GCOV_FORMAT_AUTODETECT
 	---help---
-	The gcov format is usually determined by the GCC version, but there are
+	The gcov format is usually determined by the GCC version, and the
+	default is chosen according to your GCC version. However, there are
 	exceptions where format changes are integrated in lower-version GCCs.
-	In such a case use this option to adjust the format used in the kernel
-	accordingly.
-
-	If unsure, choose "Autodetect".
-
-config GCOV_FORMAT_AUTODETECT
-	bool "Autodetect"
-	---help---
-	Select this option to use the format that corresponds to your GCC
-	version.
+	In such a case, change this option to adjust the format used in the
+	kernel accordingly.
 
 config GCOV_FORMAT_3_4
 	bool "GCC 3.4 format"
+	depends on CC_IS_GCC && GCC_VERSION < 40700
 	---help---
 	Select this option to use the format defined by GCC 3.4.
 
diff --git a/kernel/gcov/Makefile b/kernel/gcov/Makefile
index c6c50e5..ff06d64 100644
--- a/kernel/gcov/Makefile
+++ b/kernel/gcov/Makefile
@@ -4,5 +4,3 @@ ccflags-y := -DSRCTREE='"$(srctree)"' -DOBJTREE='"$(objtree)"'
 obj-y := base.o fs.o
 obj-$(CONFIG_GCOV_FORMAT_3_4) += gcc_3_4.o
 obj-$(CONFIG_GCOV_FORMAT_4_7) += gcc_4_7.o
-obj-$(CONFIG_GCOV_FORMAT_AUTODETECT) += $(call cc-ifversion, -lt, 0407, \
-							gcc_3_4.o, gcc_4_7.o)
-- 
2.7.4

[PATCH v4 27/31] kcov: test compiler capability in Kconfig and correct dependency

[Masahiro Yamada]

As Documentation/kbuild/kconfig-language.txt notes, 'select' should be
be used with care - it forces a lower limit of another symbol, ignoring
the dependency.  Currently, KCOV can select GCC_PLUGINS even if arch
does not select HAVE_GCC_PLUGINS.  This could cause the unmet direct
dependency.

Now that Kconfig can test compiler capability, let's handle this in a
more sophisticated way.

There are two ways to enable KCOV; use the compiler that natively
supports -fsanitize-coverage=trace-pc, or build the SANCOV plugin if
the compiler has ability to build GCC plugins.  Hence, the correct
dependency for KCOV is:

  depends on CC_HAS_SANCOV_TRACE_PC || GCC_PLUGINS

You do not need to build the SANCOV plugin if the compiler already
supports -fsanitize-coverage=trace-pc.  Hence, the select should be:

  select GCC_PLUGIN_SANCOV if !CC_HAS_SANCOV_TRACE_PC

With this, GCC_PLUGIN_SANCOV is selected only when necessary, so
scripts/Makefile.gcc-plugins can be cleaner.

I also cleaned up Kconfig and scripts/Makefile.kcov as well.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4: None
Changes in v3:
  - Replace the previous 'select -> imply' patch with
    a new approach

Changes in v2:
  - Drop depends on GCC_VERSION

 Makefile                     |  2 +-
 lib/Kconfig.debug            | 11 +++++++----
 scripts/Makefile.gcc-plugins |  6 +-----
 scripts/Makefile.kcov        | 10 ++++++----
 4 files changed, 15 insertions(+), 14 deletions(-)

diff --git a/Makefile b/Makefile
index 12c222d..b5c1301 100644
--- a/Makefile
+++ b/Makefile
@@ -623,7 +623,7 @@ all: vmlinux
 KBUILD_CFLAGS	+= $(call cc-option,-fno-PIE)
 KBUILD_AFLAGS	+= $(call cc-option,-fno-PIE)
 CFLAGS_GCOV	:= -fprofile-arcs -ftest-coverage -fno-tree-loop-im $(call cc-disable-warning,maybe-uninitialized,)
-export CFLAGS_GCOV CFLAGS_KCOV
+export CFLAGS_GCOV
 
 # The arch Makefile can set ARCH_{CPP,A,C}FLAGS to override the default
 # values of the respective KBUILD_* variables
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index c40c7b7..bba2298 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -736,12 +736,15 @@ config ARCH_HAS_KCOV
 	  only for x86_64. KCOV requires testing on other archs, and most likely
 	  disabling of instrumentation for some early boot code.
 
+config CC_HAS_SANCOV_TRACE_PC
+	def_bool $(cc-option, -fsanitize-coverage=trace-pc)
+
 config KCOV
 	bool "Code coverage for fuzzing"
 	depends on ARCH_HAS_KCOV
+	depends on CC_HAS_SANCOV_TRACE_PC || GCC_PLUGINS
 	select DEBUG_FS
-	select GCC_PLUGINS if !COMPILE_TEST
-	select GCC_PLUGIN_SANCOV if !COMPILE_TEST
+	select GCC_PLUGIN_SANCOV if !CC_HAS_SANCOV_TRACE_PC
 	help
 	  KCOV exposes kernel code coverage information in a form suitable
 	  for coverage-guided fuzzing (randomized testing).
@@ -755,7 +758,7 @@ config KCOV
 config KCOV_ENABLE_COMPARISONS
 	bool "Enable comparison operands collection by KCOV"
 	depends on KCOV
-	default n
+	depends on $(cc-option, -fsanitize-coverage=trace-cmp)
 	help
 	  KCOV also exposes operands of every comparison in the instrumented
 	  code along with operand sizes and PCs of the comparison instructions.
@@ -765,7 +768,7 @@ config KCOV_ENABLE_COMPARISONS
 config KCOV_INSTRUMENT_ALL
 	bool "Instrument all code by default"
 	depends on KCOV
-	default y if KCOV
+	default y
 	help
 	  If you are doing generic system call fuzzing (like e.g. syzkaller),
 	  then you will want to instrument the whole kernel and you should
diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index 7f5c862..0ce3802 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -14,16 +14,12 @@ ifdef CONFIG_GCC_PLUGINS
   endif
 
   ifdef CONFIG_GCC_PLUGIN_SANCOV
-    ifeq ($(strip $(CFLAGS_KCOV)),)
       # It is needed because of the gcc-plugin.sh and gcc version checks.
       gcc-plugin-$(CONFIG_GCC_PLUGIN_SANCOV)           += sancov_plugin.so
 
-      ifneq ($(PLUGINCC),)
-        CFLAGS_KCOV := $(SANCOV_PLUGIN)
-      else
+      ifeq ($(PLUGINCC),)
         $(warning warning: cannot use CONFIG_KCOV: -fsanitize-coverage=trace-pc is not supported by compiler)
       endif
-    endif
   endif
 
   gcc-plugin-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= structleak_plugin.so
diff --git a/scripts/Makefile.kcov b/scripts/Makefile.kcov
index 5cc7203..d71ba73 100644
--- a/scripts/Makefile.kcov
+++ b/scripts/Makefile.kcov
@@ -1,7 +1,9 @@
 ifdef CONFIG_KCOV
-CFLAGS_KCOV	:= $(call cc-option,-fsanitize-coverage=trace-pc,)
-ifeq ($(CONFIG_KCOV_ENABLE_COMPARISONS),y)
-CFLAGS_KCOV += $(call cc-option,-fsanitize-coverage=trace-cmp,)
-endif
+
+kcov-flags-$(CONFIG_CC_HAS_SANCOV_TRACE_PC)	+= -fsanitize-coverage=trace-pc
+kcov-flags-$(CONFIG_KCOV_ENABLE_COMPARISONS)	+= -fsanitize-coverage=trace-cmp
+kcov-flags-$(CONFIG_GCC_PLUGIN_SANKOV)		+= -fplugin=$(objtree)/scripts/gcc-plugins/sancov_plugin.so
+
+export CFLAGS_KCOV := $(kcov-flags-y)
 
 endif
-- 
2.7.4

[PATCH v4 28/31] gcc-plugins: move GCC version check for PowerPC to Kconfig

[Masahiro Yamada]

For PowerPC, GCC 5.2 is the requirement for GCC plugins.  Move the
version check to Kconfig so that the GCC plugin menus will be hidden
if an older compiler is in use.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
---

Changes in v4: None
Changes in v3:
  - Move comment to Kconfig as well

Changes in v2: None

 arch/powerpc/Kconfig         | 2 +-
 scripts/Makefile.gcc-plugins | 8 --------
 2 files changed, 1 insertion(+), 9 deletions(-)

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index c32a181..17e53e6 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -195,7 +195,7 @@ config PPC
 	select HAVE_FTRACE_MCOUNT_RECORD
 	select HAVE_FUNCTION_GRAPH_TRACER
 	select HAVE_FUNCTION_TRACER
-	select HAVE_GCC_PLUGINS
+	select HAVE_GCC_PLUGINS			if GCC_VERSION >= 50200   # plugin support on gcc <= 5.1 is buggy on PPC
 	select HAVE_GENERIC_GUP
 	select HAVE_HW_BREAKPOINT		if PERF_EVENTS && (PPC_BOOK3S || PPC_8xx)
 	select HAVE_IDE
diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index 0ce3802..1e92353 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -53,14 +53,6 @@ gcc-plugins-check: FORCE
 ifdef CONFIG_GCC_PLUGINS
   ifeq ($(PLUGINCC),)
     ifneq ($(GCC_PLUGINS_CFLAGS),)
-      # Various gccs between 4.5 and 5.1 have bugs on powerpc due to missing
-      # header files. gcc <= 4.6 doesn't work at all, gccs from 4.8 to 5.1 have
-      # issues with 64-bit targets.
-      ifeq ($(ARCH),powerpc)
-        ifeq ($(call cc-ifversion, -le, 0501, y), y)
-	  @echo "Cannot use CONFIG_GCC_PLUGINS: plugin support on gcc <= 5.1 is buggy on powerpc, please upgrade to gcc 5.2 or newer" >&2 && exit 1
-        endif
-      endif
       ifeq ($(call cc-ifversion, -ge, 0405, y), y)
 	$(Q)$(srctree)/scripts/gcc-plugin.sh --show-error "$(__PLUGINCC)" "$(HOSTCXX)" "$(CC)" || true
 	@echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
-- 
2.7.4

[PATCH v4 29/31] gcc-plugins: test plugin support in Kconfig and clean up Makefile

[Masahiro Yamada]

Run scripts/gcc-plugin.sh from Kconfig so that users can enable
GCC_PLUGINS only when the compiler supports building plugins.

Kconfig defines a new symbol, PLUGIN_HOSTCC.  This will contain
the compiler (g++ or gcc) used for building plugins, or empty
if the plugin can not be supported at all.

This allows us to remove all ugly testing in Makefile.gcc-plugins.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4:
  - Use if-success macro for clean-up

Changes in v3:
  - Respin to keep scripts/gcc-plugin.sh as-is.

Changes in v2: None

 arch/Kconfig                 | 10 ++++++
 scripts/Kconfig.include      |  3 ++
 scripts/Makefile.gcc-plugins | 79 ++++++++++++--------------------------------
 scripts/gcc-plugins/Makefile |  1 +
 4 files changed, 36 insertions(+), 57 deletions(-)

diff --git a/arch/Kconfig b/arch/Kconfig
index ca32950..70f585f 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -401,6 +401,15 @@ config SECCOMP_FILTER
 
 	  See Documentation/prctl/seccomp_filter.txt for details.
 
+preferred-plugin-hostcc := $(if-success, [ $(gcc-version) -ge 40800 ],$(HOSTCXX),$(HOSTCC))
+
+config PLUGIN_HOSTCC
+	string
+	default "$(shell, $(srctree)/scripts/gcc-plugin.sh $(preferred-plugin-hostcc) $(HOSTCXX) $(CC))"
+	help
+	  Host compiler used to build GCC plugins.  This can be $(HOSTCXX),
+	  $(HOSTCC), or a null string if GCC plugin is unsupported.
+
 config HAVE_GCC_PLUGINS
 	bool
 	help
@@ -410,6 +419,7 @@ config HAVE_GCC_PLUGINS
 menuconfig GCC_PLUGINS
 	bool "GCC plugins"
 	depends on HAVE_GCC_PLUGINS
+	depends on PLUGIN_HOSTCC != ""
 	depends on !COMPILE_TEST
 	help
 	  GCC plugins are loadable modules that provide extra features to the
diff --git a/scripts/Kconfig.include b/scripts/Kconfig.include
index b1502f4..bfecf61 100644
--- a/scripts/Kconfig.include
+++ b/scripts/Kconfig.include
@@ -22,3 +22,6 @@ cc-option = $(success, $(CC) -Werror $(1) -E -x c /dev/null -o /dev/null)
 # $(ld-option,<flag>)
 # y if the linker supports <flag>, n otherwise
 ld-option = $(success, $(LD) -v $(1))
+
+# gcc version including patch level
+gcc-version := $(shell, $(srctree)/scripts/gcc-version.sh -p $(CC) | sed 's/^0*//')
diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index 1e92353..da5d38d 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -1,72 +1,37 @@
 # SPDX-License-Identifier: GPL-2.0
-ifdef CONFIG_GCC_PLUGINS
-  __PLUGINCC := $(call cc-ifversion, -ge, 0408, $(HOSTCXX), $(HOSTCC))
-  PLUGINCC := $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-plugin.sh "$(__PLUGINCC)" "$(HOSTCXX)" "$(CC)")
-
-  SANCOV_PLUGIN := -fplugin=$(objtree)/scripts/gcc-plugins/sancov_plugin.so
-
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_CYC_COMPLEXITY)	+= cyc_complexity_plugin.so
+gcc-plugin-$(CONFIG_GCC_PLUGIN_CYC_COMPLEXITY)	+= cyc_complexity_plugin.so
 
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= latent_entropy_plugin.so
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= -DLATENT_ENTROPY_PLUGIN
-  ifdef CONFIG_GCC_PLUGIN_LATENT_ENTROPY
+gcc-plugin-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= latent_entropy_plugin.so
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= -DLATENT_ENTROPY_PLUGIN
+ifdef CONFIG_GCC_PLUGIN_LATENT_ENTROPY
     DISABLE_LATENT_ENTROPY_PLUGIN			+= -fplugin-arg-latent_entropy_plugin-disable
-  endif
-
-  ifdef CONFIG_GCC_PLUGIN_SANCOV
-      # It is needed because of the gcc-plugin.sh and gcc version checks.
-      gcc-plugin-$(CONFIG_GCC_PLUGIN_SANCOV)           += sancov_plugin.so
-
-      ifeq ($(PLUGINCC),)
-        $(warning warning: cannot use CONFIG_KCOV: -fsanitize-coverage=trace-pc is not supported by compiler)
-      endif
-  endif
-
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= structleak_plugin.so
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_VERBOSE)	+= -fplugin-arg-structleak_plugin-verbose
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)	+= -fplugin-arg-structleak_plugin-byref-all
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= -DSTRUCTLEAK_PLUGIN
+endif
 
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= randomize_layout_plugin.so
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= -DRANDSTRUCT_PLUGIN
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT_PERFORMANCE)	+= -fplugin-arg-randomize_layout_plugin-performance-mode
+gcc-plugin-$(CONFIG_GCC_PLUGIN_SANCOV)		+= sancov_plugin.so
+gcc-plugin-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= structleak_plugin.so
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_VERBOSE)	+= -fplugin-arg-structleak_plugin-verbose
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)	+= -fplugin-arg-structleak_plugin-byref-all
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= -DSTRUCTLEAK_PLUGIN
 
-  GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y))
+gcc-plugin-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= randomize_layout_plugin.so
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= -DRANDSTRUCT_PLUGIN
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT_PERFORMANCE)	+= -fplugin-arg-randomize_layout_plugin-performance-mode
 
-  export PLUGINCC GCC_PLUGINS_CFLAGS GCC_PLUGIN GCC_PLUGIN_SUBDIR
-  export SANCOV_PLUGIN DISABLE_LATENT_ENTROPY_PLUGIN
+GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y))
 
-  ifneq ($(PLUGINCC),)
-    # SANCOV_PLUGIN can be only in CFLAGS_KCOV because avoid duplication.
-    GCC_PLUGINS_CFLAGS := $(filter-out $(SANCOV_PLUGIN), $(GCC_PLUGINS_CFLAGS))
-  endif
+export GCC_PLUGINS_CFLAGS GCC_PLUGIN GCC_PLUGIN_SUBDIR
+export DISABLE_LATENT_ENTROPY_PLUGIN
 
-  KBUILD_CFLAGS += $(GCC_PLUGINS_CFLAGS)
-  GCC_PLUGIN := $(gcc-plugin-y)
-  GCC_PLUGIN_SUBDIR := $(gcc-plugin-subdir-y)
-endif
+# SANCOV_PLUGIN can be only in CFLAGS_KCOV because avoid duplication.
+GCC_PLUGINS_CFLAGS := $(filter-out %/sancov_plugin.so, $(GCC_PLUGINS_CFLAGS))
 
-# If plugins aren't supported, abort the build before hard-to-read compiler
-# errors start getting spewed by the main build.
-PHONY += gcc-plugins-check
-gcc-plugins-check: FORCE
-ifdef CONFIG_GCC_PLUGINS
-  ifeq ($(PLUGINCC),)
-    ifneq ($(GCC_PLUGINS_CFLAGS),)
-      ifeq ($(call cc-ifversion, -ge, 0405, y), y)
-	$(Q)$(srctree)/scripts/gcc-plugin.sh --show-error "$(__PLUGINCC)" "$(HOSTCXX)" "$(CC)" || true
-	@echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
-      else
-	@echo "Cannot use CONFIG_GCC_PLUGINS: your gcc version does not support plugins, you should upgrade it to at least gcc 4.5" >&2 && exit 1
-      endif
-    endif
-  endif
-endif
-	@:
+KBUILD_CFLAGS += $(GCC_PLUGINS_CFLAGS)
+GCC_PLUGIN := $(gcc-plugin-y)
+GCC_PLUGIN_SUBDIR := $(gcc-plugin-subdir-y)
 
 # Actually do the build, if requested.
 PHONY += gcc-plugins
-gcc-plugins: scripts_basic gcc-plugins-check
+gcc-plugins: scripts_basic
 ifdef CONFIG_GCC_PLUGINS
 	$(Q)$(MAKE) $(build)=scripts/gcc-plugins
 endif
diff --git a/scripts/gcc-plugins/Makefile b/scripts/gcc-plugins/Makefile
index e2ff425..c47198c 100644
--- a/scripts/gcc-plugins/Makefile
+++ b/scripts/gcc-plugins/Makefile
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
+PLUGINCC := $(CONFIG_PLUGIN_HOSTCC:"%"=%)
 GCC_PLUGINS_DIR := $(shell $(CC) -print-file-name=plugin)
 
 ifeq ($(PLUGINCC),$(HOSTCC))
-- 
2.7.4

[PATCH v4 30/31] gcc-plugins: allow to enable GCC_PLUGINS for COMPILE_TEST

[Masahiro Yamada]

Now that the compiler's plugin support is checked in Kconfig,
all{yes,mod}config will not be bothered.

Remove 'depends on !COMPILE_TEST' for GCC_PLUGINS.

'depends on !COMPILE_TEST' for the following three are still kept:
  GCC_PLUGIN_CYC_COMPLEXITY
  GCC_PLUGIN_STRUCTLEAK_VERBOSE
  GCC_PLUGIN_RANDSTRUCT_PERFORMANCE

Kees said to do so because the first two are too noisy, and the last
one would reduce the compile test coverage.  I commented the reasons
in arch/Kconfig.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v4: None
Changes in v3:
  - Add comment about reason

Changes in v2: None

 arch/Kconfig | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/arch/Kconfig b/arch/Kconfig
index 70f585f..6e8f0cf 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -420,7 +420,6 @@ menuconfig GCC_PLUGINS
 	bool "GCC plugins"
 	depends on HAVE_GCC_PLUGINS
 	depends on PLUGIN_HOSTCC != ""
-	depends on !COMPILE_TEST
 	help
 	  GCC plugins are loadable modules that provide extra features to the
 	  compiler. They are useful for runtime instrumentation and static analysis.
@@ -430,7 +429,7 @@ menuconfig GCC_PLUGINS
 config GCC_PLUGIN_CYC_COMPLEXITY
 	bool "Compute the cyclomatic complexity of a function" if EXPERT
 	depends on GCC_PLUGINS
-	depends on !COMPILE_TEST
+	depends on !COMPILE_TEST	# too noisy
 	help
 	  The complexity M of a function's control flow graph is defined as:
 	   M = E - N + 2P
@@ -497,7 +496,7 @@ config GCC_PLUGIN_STRUCTLEAK_BYREF_ALL
 config GCC_PLUGIN_STRUCTLEAK_VERBOSE
 	bool "Report forcefully initialized variables"
 	depends on GCC_PLUGIN_STRUCTLEAK
-	depends on !COMPILE_TEST
+	depends on !COMPILE_TEST	# too noisy
 	help
 	  This option will cause a warning to be printed each time the
 	  structleak plugin finds a variable it thinks needs to be
@@ -537,7 +536,7 @@ config GCC_PLUGIN_RANDSTRUCT
 config GCC_PLUGIN_RANDSTRUCT_PERFORMANCE
 	bool "Use cacheline-aware structure randomization"
 	depends on GCC_PLUGIN_RANDSTRUCT
-	depends on !COMPILE_TEST
+	depends on !COMPILE_TEST	# do not reduce test coverage
 	help
 	  If you say Y here, the RANDSTRUCT randomization will make a
 	  best effort at restricting randomization to cacheline-sized
-- 
2.7.4

[PATCH v4 31/31] arm64: move GCC version check for ARCH_SUPPORTS_INT128 to Kconfig

[Masahiro Yamada]

This becomes much neater in Kconfig.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
---

Changes in v4:
 - Rebase

Changes in v3:
 - Newly added

Changes in v2: None

 arch/arm64/Kconfig  | 1 +
 arch/arm64/Makefile | 6 ------
 2 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index eb2cf49..119b18e 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -43,6 +43,7 @@ config ARM64
 	select ARCH_USE_QUEUED_RWLOCKS
 	select ARCH_SUPPORTS_MEMORY_FAILURE
 	select ARCH_SUPPORTS_ATOMIC_RMW
+	select ARCH_SUPPORTS_INT128 if GCC_VERSION >= 50000 || CC_IS_CLANG
 	select ARCH_SUPPORTS_NUMA_BALANCING
 	select ARCH_WANT_COMPAT_IPC_PARSE_VERSION
 	select ARCH_WANT_FRAME_POINTERS
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 87f7d2f..cefd1e9 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -56,12 +56,6 @@ KBUILD_AFLAGS	+= $(lseinstr) $(brokengasinst)
 KBUILD_CFLAGS	+= $(call cc-option,-mabi=lp64)
 KBUILD_AFLAGS	+= $(call cc-option,-mabi=lp64)
 
-ifeq ($(cc-name),clang)
-KBUILD_CFLAGS	+= -DCONFIG_ARCH_SUPPORTS_INT128
-else
-KBUILD_CFLAGS	+= $(call cc-ifversion, -ge, 0500, -DCONFIG_ARCH_SUPPORTS_INT128)
-endif
-
 ifeq ($(CONFIG_CPU_BIG_ENDIAN), y)
 KBUILD_CPPFLAGS	+= -mbig-endian
 CHECKFLAGS	+= -D__AARCH64EB__
-- 
2.7.4

Re: [PATCH v4 23/31] stack-protector: test compiler capability in Kconfig and drop AUTO mode

[Kees Cook]

On Wed, May 16, 2018 at 11:17 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Move the test for -fstack-protector(-strong) option to Kconfig.
>
> If the compiler does not support the option, the corresponding menu
> is automatically hidden.  If STRONG is not supported, it will fall
> back to REGULAR.  If REGULAR is not supported, it will be disabled.
> This means, AUTO is implicitly handled by the dependency solver of
> Kconfig, hence removed.
>
> I also turned the 'choice' into only two boolean symbols.  The use of
> 'choice' is not a good idea here, because all of all{yes,mod,no}config
> would choose the first visible value, while we want allnoconfig to
> disable as many features as possible.
>
> X86 has additional shell scripts in case the compiler supports those
> options, but generates broken code.  I added CC_HAS_SANE_STACKPROTECTOR
> to test this.  I had to add -m32 to gcc-x86_32-has-stack-protector.sh
> to make it work correctly.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Thanks!

Acked-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v4 30/31] gcc-plugins: allow to enable GCC_PLUGINS for COMPILE_TEST

[Kees Cook]

On Wed, May 16, 2018 at 11:17 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Now that the compiler's plugin support is checked in Kconfig,
> all{yes,mod}config will not be bothered.
>
> Remove 'depends on !COMPILE_TEST' for GCC_PLUGINS.
>
> 'depends on !COMPILE_TEST' for the following three are still kept:
>   GCC_PLUGIN_CYC_COMPLEXITY
>   GCC_PLUGIN_STRUCTLEAK_VERBOSE
>   GCC_PLUGIN_RANDSTRUCT_PERFORMANCE
>
> Kees said to do so because the first two are too noisy, and the last
> one would reduce the compile test coverage.  I commented the reasons
> in arch/Kconfig.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

I'm really looking forward to these being enabled for COMPILE_TEST. :)

Acked-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v4 06/31] kconfig: remove sym_expand_string_value()

[Kees Cook]

On Wed, May 16, 2018 at 11:16 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> There is no more caller of sym_expand_string_value().
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

Easy to review! :)

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v4 28/31] gcc-plugins: move GCC version check for PowerPC to Kconfig

[Kees Cook]

On Wed, May 16, 2018 at 11:17 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> For PowerPC, GCC 5.2 is the requirement for GCC plugins.  Move the
> version check to Kconfig so that the GCC plugin menus will be hidden
> if an older compiler is in use.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v4 29/31] gcc-plugins: test plugin support in Kconfig and clean up Makefile

[Kees Cook]

On Wed, May 16, 2018 at 11:17 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Run scripts/gcc-plugin.sh from Kconfig so that users can enable
> GCC_PLUGINS only when the compiler supports building plugins.
>
> Kconfig defines a new symbol, PLUGIN_HOSTCC.  This will contain
> the compiler (g++ or gcc) used for building plugins, or empty
> if the plugin can not be supported at all.
>
> This allows us to remove all ugly testing in Makefile.gcc-plugins.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Acked-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v4 27/31] kcov: test compiler capability in Kconfig and correct dependency

[Kees Cook]

On Wed, May 16, 2018 at 11:17 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> As Documentation/kbuild/kconfig-language.txt notes, 'select' should be
> be used with care - it forces a lower limit of another symbol, ignoring
> the dependency.  Currently, KCOV can select GCC_PLUGINS even if arch
> does not select HAVE_GCC_PLUGINS.  This could cause the unmet direct
> dependency.
>
> Now that Kconfig can test compiler capability, let's handle this in a
> more sophisticated way.
>
> There are two ways to enable KCOV; use the compiler that natively
> supports -fsanitize-coverage=trace-pc, or build the SANCOV plugin if
> the compiler has ability to build GCC plugins.  Hence, the correct
> dependency for KCOV is:
>
>   depends on CC_HAS_SANCOV_TRACE_PC || GCC_PLUGINS
>
> You do not need to build the SANCOV plugin if the compiler already
> supports -fsanitize-coverage=trace-pc.  Hence, the select should be:
>
>   select GCC_PLUGIN_SANCOV if !CC_HAS_SANCOV_TRACE_PC
>
> With this, GCC_PLUGIN_SANCOV is selected only when necessary, so
> scripts/Makefile.gcc-plugins can be cleaner.
>
> I also cleaned up Kconfig and scripts/Makefile.kcov as well.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v4 19/31] Documentation: kconfig: document a new Kconfig macro language

[Kees Cook]

On Wed, May 16, 2018 at 11:16 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Add a document for the macro language introduced to Kconfig.
>
> The motivation of this work is to move the compiler option tests to
> Kconfig from Makefile.  A number of kernel features require the
> compiler support.  Enabling such features blindly in Kconfig ends up
> with a lot of nasty build-time testing in Makefiles.  If a chosen
> feature turns out unsupported by the compiler, what the build system
> can do is either to disable it (silently!) or to forcibly break the
> build, despite Kconfig has let the user to enable it.  By moving the
> compiler capability tests to Kconfig, features unsupported by the
> compiler will be hidden automatically.
>
> This change was strongly prompted by Linus Torvalds.  You can find
> his suggestions [1] [2] in ML.  The original idea was to add a new
> attribute with 'option shell=...', but I found more generalized text
> expansion would make Kconfig more powerful and lovely.  The basic
> ideas are from Make, but there are some differences.
>
> [1]: https://lkml.org/lkml/2016/12/9/577
> [2]: https://lkml.org/lkml/2018/2/7/527
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

(Added Randy, Jon, and linux-doc to CC for more review)

This should likely be written in .rst and linked to from the developer index...

https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#writing-documentation

As for the content, though:

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

> ---
>
> Changes in v4:
>  - Update according to the syntax change
>
> Changes in v3:
>  - Newly added
>
> Changes in v2: None
>
>  Documentation/kbuild/kconfig-macro-language.txt | 252 ++++++++++++++++++++++++
>  MAINTAINERS                                     |   2 +-
>  2 files changed, 253 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/kbuild/kconfig-macro-language.txt
>
> diff --git a/Documentation/kbuild/kconfig-macro-language.txt b/Documentation/kbuild/kconfig-macro-language.txt
> new file mode 100644
> index 0000000..a8dc792
> --- /dev/null
> +++ b/Documentation/kbuild/kconfig-macro-language.txt
> @@ -0,0 +1,252 @@
> +Concept
> +-------
> +
> +The basic idea was inspired by Make. When we look at Make, we notice sort of
> +two languages in one. One language describes dependency graphs consisting of
> +targets and prerequisites. The other is a macro language for performing textual
> +substitution.
> +
> +There is clear distinction between the two language stages. For example, you
> +can write a makefile like follows:
> +
> +    APP := foo
> +    SRC := foo.c
> +    CC := gcc
> +
> +    $(APP): $(SRC)
> +            $(CC) -o $(APP) $(SRC)
> +
> +The macro language replaces the variable references with their expanded form,
> +and handles as if the source file were input like follows:
> +
> +    foo: foo.c
> +            gcc -o foo foo.c
> +
> +Then, Make analyzes the dependency graph and determines the targets to be
> +updated.
> +
> +The idea is quite similar in Kconfig - it is possible to describe a Kconfig
> +file like this:
> +
> +    CC := gcc
> +
> +    config CC_HAS_FOO
> +            def_bool $(shell, $(srctree)/scripts/gcc-check-foo.sh $(CC))
> +
> +The macro language in Kconfig processes the source file into the following
> +intermediate:
> +
> +    config CC_HAS_FOO
> +            def_bool y
> +
> +Then, Kconfig moves onto the evaluation stage to resolve inter-symbol
> +dependency as explained in kconfig-language.txt.
> +
> +
> +Variables
> +---------
> +
> +Like in Make, a variable in Kconfig works as a macro variable.  A macro
> +variable is expanded "in place" to yield a text string that may then be
> +expanded further. To get the value of a variable, enclose the variable name in
> +$( ). The parentheses are required even for single-letter variable names; $X is
> +a syntax error. The curly brace form as in ${CC} is not supported either.
> +
> +There are two types of variables: simply expanded variables and recursively
> +expanded variables.
> +
> +A simply expanded variable is defined using the := assignment operator. Its
> +righthand side is expanded immediately upon reading the line from the Kconfig
> +file.
> +
> +A recursively expanded variable is defined using the = assignment operator.
> +Its righthand side is simply stored as the value of the variable without
> +expanding it in any way. Instead, the expansion is performed when the variable
> +is used.
> +
> +There is another type of assignment operator; += is used to append text to a
> +variable. The righthand side of += is expanded immediately if the lefthand
> +side was originally defined as a simple variable. Otherwise, its evaluation is
> +deferred.
> +
> +The variable reference can take parameters, in the following form:
> +
> +  $(name,arg1,arg2,arg3)
> +
> +You can consider the parameterized reference as a function. (more precisely,
> +"user-defined function" in the contrast to "built-in function" listed below).
> +
> +Useful functions must be expanded when they are used since the same function is
> +expanded differently if different parameters are passed. Hence, a user-defined
> +function is defined using the = assignment operator. The parameters are
> +referenced within the body definition with $(1), $(2), etc.
> +
> +In fact, recursively expanded variables and user-defined functions are the same
> +internally. (In other words, "variable" is "function with zero argument".)
> +When we say "variable" in a broad sense, it includes "user-defined function".
> +
> +
> +Built-in functions
> +------------------
> +
> +Like Make, Kconfig provides several built-in functions. Every function takes a
> +particular number of arguments.
> +
> +In Make, every built-in function takes at least one argument. Kconfig allows
> +zero argument for built-in functions, such as $(fileno), $(lineno). You could
> +consider those as "built-in variable", but it is just a matter of how we call
> +it after all. Let's say "built-in function" here to refer to natively supported
> +functionality.
> +
> +Kconfig currently supports the following built-in functions.
> +
> + - $(shell,command)
> +
> +  The "shell" function accepts a single argument that is expanded and passed
> +  to a subshell for execution. The standard output of the command is then read
> +  and returned as the value of the function. Every newline in the output is
> +  replaced with a space. Any trailing newlines are deleted. The standard error
> +  is not returned, nor is any program exit status.
> +
> + - $(info,text)
> +
> +  The "info" function takes a single argument and prints it to stdout.
> +  It evaluates to an empty string.
> +
> + - $(warning,text)
> +
> +  The "warning" function is similar to "info" except that it sends its argument
> +  to stderr and prefixes the output with the name of the current Kconfig file
> +  and the current line number.
> +
> + - $(error,text)
> +
> +  The "error" function is similar to "warning", but it terminates the parsing
> +  immediately.
> +
> + - $(if,condition,then-part[,else-part])
> +
> +  The "if" function takes two or three arguments ('else-part' is optional).
> +  Depending on the value of the condition part, the argument to be expanded
> +  is selected. The condition is true if its expansion contains any characters
> +  except whitespaces. In this case, the then-part is expanded. Otherwise, the
> +  else-part is expanded.
> +
> +  Note:
> +  In Make, the condition is true if it contains any characters including
> +  whitespaces, which is why $(strip ...) is sometimes necessary in the
> +  condition part. Kconfig changed the behavior to make it handier.
> +
> + - $(filename)
> +
> +  The 'filename' takes no argument, and $(filename) is expanded to a file name
> +  being parsed.
> +
> + - $(lineno)
> +
> +  The 'lineno' takes no argument, and $(lineno) is expanded to a line number
> +  being parsed.
> +
> +
> +Difference of function call syntax
> +----------------------------------
> +
> +Kconfig adopts Make-like macro language, but the function call syntax is
> +slightly different.
> +
> +A function call in Make looks like follows:
> +
> +  $(func-name arg1,arg2,arg3)
> +
> +The function name and the first argument are separated by at least one
> +whitespace. Then, leading whitespaces are trimmed from the first argument,
> +but whitespaces in the other arguments are kept. You need to use a kind of
> +trick to start the first parameter with spaces. For example, if you want
> +to make "info" function print "  hello", you can write like follows:
> +
> +  $(info $(space)$(space)hello)
> +
> +Kconfig uses only commas for delimiters, and keeps all whitespaces in the
> +function call. Some people prefer putting a space after each comma delimiter:
> +
> +  $(func-name, arg1, arg2, arg3)
> +
> +In this case, "func-name" will receive " arg1", " arg2", " arg3". The presence
> +of leading spaces may really matter depending on the function. The same applies
> +to Make - for example, $(subst .c, .o, $(sources)) is a typical mistake.
> +
> +In Make, a user-defined function is referenced by using a built-in function,
> +'call', like this:
> +
> +    $(call my-func,arg1,arg2,arg3)
> +
> +Kconfig invokes user-defined functions and built-in functions in the same way.
> +The omission of 'call' makes the syntax shorter.
> +
> +In Make, some functions exceptionally treat commas verbatim instead of argument
> +separators. For example, $(shell echo hello, world) evaluates to "hello, world".
> +Likewise, $(info hello, world) prints "hello, world" to stdout. You could say
> +this is _useful_ inconsistency.
> +
> +For simpler implementation and grammatical consistency, Kconfig always treats
> +commas that appear in the $( ) form as delimiters. It means
> +
> +  $(shell, echo hello, world)
> +
> +is an error because it is passing two parameters where the 'shell' function
> +accepts only one. To pass commas in arguments, you can use the following trick:
> +
> +  comma := ,
> +  $(shell, echo hello$(comma) world)
> +
> +
> +Caveats
> +-------
> +
> +A variable (or function) cannot be expanded across tokens. So, you cannot use
> +a variable as a shorthand for an expression that consists of multiple tokens.
> +The following works:
> +
> +    RANGE_MIN := 1
> +    RANGE_MAX := 3
> +
> +    config FOO
> +            int "foo"
> +            range $(RANGE_MIN) $(RANGE_MAX)
> +
> +But, the following does not work:
> +
> +    RANGES := 1 3
> +
> +    config FOO
> +            int "foo"
> +            range $(RANGES)
> +
> +A variable cannot be expanded to any keyword in Kconfig.  The following does
> +not work:
> +
> +    MY_TYPE := tristate
> +
> +    config FOO
> +            $(MY_TYPE) "foo"
> +            default y
> +
> +Obviously from the design, $(shell command) is expanded in the textual
> +substitution phase. You cannot pass symbols to the 'shell' function.
> +The following does not work as expected.
> +
> +    config ENDIAN_FLAG
> +            string
> +            default "-mbig-endian" if CPU_BIG_ENDIAN
> +            default "-mlittle-endian" if CPU_LITTLE_ENDIAN
> +
> +    config CC_HAS_ENDIAN_FLAG
> +            def_bool $(shell $(srctree)/scripts/gcc-check-flag ENDIAN_FLAG)
> +
> +Instead, you can do like follows so that any function call is statically
> +expanded.
> +
> +    config CC_HAS_ENDIAN_FLAG
> +            bool
> +            default $(shell $(srctree)/scripts/gcc-check-flag -mbig-endian) if CPU_BIG_ENDIAN
> +            default $(shell $(srctree)/scripts/gcc-check-flag -mlittle-endian) if CPU_LITTLE_ENDIAN
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 58b9861..b7d7ae61 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -7632,7 +7632,7 @@ M:        Masahiro Yamada <yamada.masahiro@socionext.com>
>  T:     git git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git kconfig
>  L:     linux-kbuild@vger.kernel.org
>  S:     Maintained
> -F:     Documentation/kbuild/kconfig-language.txt
> +F:     Documentation/kbuild/kconfig*
>  F:     scripts/kconfig/
>
>  KDUMP
> --
> 2.7.4
>



-- 
Kees Cook
Pixel Security

Re: [PATCH v4 19/31] Documentation: kconfig: document a new Kconfig macro language

[Kees Cook]

On Wed, May 16, 2018 at 11:16 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Add a document for the macro language introduced to Kconfig.
>
> The motivation of this work is to move the compiler option tests to
> Kconfig from Makefile.  A number of kernel features require the
> compiler support.  Enabling such features blindly in Kconfig ends up
> with a lot of nasty build-time testing in Makefiles.  If a chosen
> feature turns out unsupported by the compiler, what the build system
> can do is either to disable it (silently!) or to forcibly break the
> build, despite Kconfig has let the user to enable it.  By moving the
> compiler capability tests to Kconfig, features unsupported by the
> compiler will be hidden automatically.
>
> This change was strongly prompted by Linus Torvalds.  You can find
> his suggestions [1] [2] in ML.  The original idea was to add a new
> attribute with 'option shell=...', but I found more generalized text
> expansion would make Kconfig more powerful and lovely.  The basic
> ideas are from Make, but there are some differences.
>
> [1]: https://lkml.org/lkml/2016/12/9/577
> [2]: https://lkml.org/lkml/2018/2/7/527
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

(Added Randy, Jon, and linux-doc to CC for more review)

This should likely be written in .rst and linked to from the developer index...

https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#writing-documentation

As for the content, though:

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

> ---
>
> Changes in v4:
>  - Update according to the syntax change
>
> Changes in v3:
>  - Newly added
>
> Changes in v2: None
>
>  Documentation/kbuild/kconfig-macro-language.txt | 252 ++++++++++++++++++++++++
>  MAINTAINERS                                     |   2 +-
>  2 files changed, 253 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/kbuild/kconfig-macro-language.txt
>
> diff --git a/Documentation/kbuild/kconfig-macro-language.txt b/Documentation/kbuild/kconfig-macro-language.txt
> new file mode 100644
> index 0000000..a8dc792
> --- /dev/null
> +++ b/Documentation/kbuild/kconfig-macro-language.txt
> @@ -0,0 +1,252 @@
> +Concept
> +-------
> +
> +The basic idea was inspired by Make. When we look at Make, we notice sort of
> +two languages in one. One language describes dependency graphs consisting of
> +targets and prerequisites. The other is a macro language for performing textual
> +substitution.
> +
> +There is clear distinction between the two language stages. For example, you
> +can write a makefile like follows:
> +
> +    APP := foo
> +    SRC := foo.c
> +    CC := gcc
> +
> +    $(APP): $(SRC)
> +            $(CC) -o $(APP) $(SRC)
> +
> +The macro language replaces the variable references with their expanded form,
> +and handles as if the source file were input like follows:
> +
> +    foo: foo.c
> +            gcc -o foo foo.c
> +
> +Then, Make analyzes the dependency graph and determines the targets to be
> +updated.
> +
> +The idea is quite similar in Kconfig - it is possible to describe a Kconfig
> +file like this:
> +
> +    CC := gcc
> +
> +    config CC_HAS_FOO
> +            def_bool $(shell, $(srctree)/scripts/gcc-check-foo.sh $(CC))
> +
> +The macro language in Kconfig processes the source file into the following
> +intermediate:
> +
> +    config CC_HAS_FOO
> +            def_bool y
> +
> +Then, Kconfig moves onto the evaluation stage to resolve inter-symbol
> +dependency as explained in kconfig-language.txt.
> +
> +
> +Variables
> +---------
> +
> +Like in Make, a variable in Kconfig works as a macro variable.  A macro
> +variable is expanded "in place" to yield a text string that may then be
> +expanded further. To get the value of a variable, enclose the variable name in
> +$( ). The parentheses are required even for single-letter variable names; $X is
> +a syntax error. The curly brace form as in ${CC} is not supported either.
> +
> +There are two types of variables: simply expanded variables and recursively
> +expanded variables.
> +
> +A simply expanded variable is defined using the := assignment operator. Its
> +righthand side is expanded immediately upon reading the line from the Kconfig
> +file.
> +
> +A recursively expanded variable is defined using the = assignment operator.
> +Its righthand side is simply stored as the value of the variable without
> +expanding it in any way. Instead, the expansion is performed when the variable
> +is used.
> +
> +There is another type of assignment operator; += is used to append text to a
> +variable. The righthand side of += is expanded immediately if the lefthand
> +side was originally defined as a simple variable. Otherwise, its evaluation is
> +deferred.
> +
> +The variable reference can take parameters, in the following form:
> +
> +  $(name,arg1,arg2,arg3)
> +
> +You can consider the parameterized reference as a function. (more precisely,
> +"user-defined function" in the contrast to "built-in function" listed below).
> +
> +Useful functions must be expanded when they are used since the same function is
> +expanded differently if different parameters are passed. Hence, a user-defined
> +function is defined using the = assignment operator. The parameters are
> +referenced within the body definition with $(1), $(2), etc.
> +
> +In fact, recursively expanded variables and user-defined functions are the same
> +internally. (In other words, "variable" is "function with zero argument".)
> +When we say "variable" in a broad sense, it includes "user-defined function".
> +
> +
> +Built-in functions
> +------------------
> +
> +Like Make, Kconfig provides several built-in functions. Every function takes a
> +particular number of arguments.
> +
> +In Make, every built-in function takes at least one argument. Kconfig allows
> +zero argument for built-in functions, such as $(fileno), $(lineno). You could
> +consider those as "built-in variable", but it is just a matter of how we call
> +it after all. Let's say "built-in function" here to refer to natively supported
> +functionality.
> +
> +Kconfig currently supports the following built-in functions.
> +
> + - $(shell,command)
> +
> +  The "shell" function accepts a single argument that is expanded and passed
> +  to a subshell for execution. The standard output of the command is then read
> +  and returned as the value of the function. Every newline in the output is
> +  replaced with a space. Any trailing newlines are deleted. The standard error
> +  is not returned, nor is any program exit status.
> +
> + - $(info,text)
> +
> +  The "info" function takes a single argument and prints it to stdout.
> +  It evaluates to an empty string.
> +
> + - $(warning,text)
> +
> +  The "warning" function is similar to "info" except that it sends its argument
> +  to stderr and prefixes the output with the name of the current Kconfig file
> +  and the current line number.
> +
> + - $(error,text)
> +
> +  The "error" function is similar to "warning", but it terminates the parsing
> +  immediately.
> +
> + - $(if,condition,then-part[,else-part])
> +
> +  The "if" function takes two or three arguments ('else-part' is optional).
> +  Depending on the value of the condition part, the argument to be expanded
> +  is selected. The condition is true if its expansion contains any characters
> +  except whitespaces. In this case, the then-part is expanded. Otherwise, the
> +  else-part is expanded.
> +
> +  Note:
> +  In Make, the condition is true if it contains any characters including
> +  whitespaces, which is why $(strip ...) is sometimes necessary in the
> +  condition part. Kconfig changed the behavior to make it handier.
> +
> + - $(filename)
> +
> +  The 'filename' takes no argument, and $(filename) is expanded to a file name
> +  being parsed.
> +
> + - $(lineno)
> +
> +  The 'lineno' takes no argument, and $(lineno) is expanded to a line number
> +  being parsed.
> +
> +
> +Difference of function call syntax
> +----------------------------------
> +
> +Kconfig adopts Make-like macro language, but the function call syntax is
> +slightly different.
> +
> +A function call in Make looks like follows:
> +
> +  $(func-name arg1,arg2,arg3)
> +
> +The function name and the first argument are separated by at least one
> +whitespace. Then, leading whitespaces are trimmed from the first argument,
> +but whitespaces in the other arguments are kept. You need to use a kind of
> +trick to start the first parameter with spaces. For example, if you want
> +to make "info" function print "  hello", you can write like follows:
> +
> +  $(info $(space)$(space)hello)
> +
> +Kconfig uses only commas for delimiters, and keeps all whitespaces in the
> +function call. Some people prefer putting a space after each comma delimiter:
> +
> +  $(func-name, arg1, arg2, arg3)
> +
> +In this case, "func-name" will receive " arg1", " arg2", " arg3". The presence
> +of leading spaces may really matter depending on the function. The same applies
> +to Make - for example, $(subst .c, .o, $(sources)) is a typical mistake.
> +
> +In Make, a user-defined function is referenced by using a built-in function,
> +'call', like this:
> +
> +    $(call my-func,arg1,arg2,arg3)
> +
> +Kconfig invokes user-defined functions and built-in functions in the same way.
> +The omission of 'call' makes the syntax shorter.
> +
> +In Make, some functions exceptionally treat commas verbatim instead of argument
> +separators. For example, $(shell echo hello, world) evaluates to "hello, world".
> +Likewise, $(info hello, world) prints "hello, world" to stdout. You could say
> +this is _useful_ inconsistency.
> +
> +For simpler implementation and grammatical consistency, Kconfig always treats
> +commas that appear in the $( ) form as delimiters. It means
> +
> +  $(shell, echo hello, world)
> +
> +is an error because it is passing two parameters where the 'shell' function
> +accepts only one. To pass commas in arguments, you can use the following trick:
> +
> +  comma := ,
> +  $(shell, echo hello$(comma) world)
> +
> +
> +Caveats
> +-------
> +
> +A variable (or function) cannot be expanded across tokens. So, you cannot use
> +a variable as a shorthand for an expression that consists of multiple tokens.
> +The following works:
> +
> +    RANGE_MIN := 1
> +    RANGE_MAX := 3
> +
> +    config FOO
> +            int "foo"
> +            range $(RANGE_MIN) $(RANGE_MAX)
> +
> +But, the following does not work:
> +
> +    RANGES := 1 3
> +
> +    config FOO
> +            int "foo"
> +            range $(RANGES)
> +
> +A variable cannot be expanded to any keyword in Kconfig.  The following does
> +not work:
> +
> +    MY_TYPE := tristate
> +
> +    config FOO
> +            $(MY_TYPE) "foo"
> +            default y
> +
> +Obviously from the design, $(shell command) is expanded in the textual
> +substitution phase. You cannot pass symbols to the 'shell' function.
> +The following does not work as expected.
> +
> +    config ENDIAN_FLAG
> +            string
> +            default "-mbig-endian" if CPU_BIG_ENDIAN
> +            default "-mlittle-endian" if CPU_LITTLE_ENDIAN
> +
> +    config CC_HAS_ENDIAN_FLAG
> +            def_bool $(shell $(srctree)/scripts/gcc-check-flag ENDIAN_FLAG)
> +
> +Instead, you can do like follows so that any function call is statically
> +expanded.
> +
> +    config CC_HAS_ENDIAN_FLAG
> +            bool
> +            default $(shell $(srctree)/scripts/gcc-check-flag -mbig-endian) if CPU_BIG_ENDIAN
> +            default $(shell $(srctree)/scripts/gcc-check-flag -mlittle-endian) if CPU_LITTLE_ENDIAN
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 58b9861..b7d7ae61 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -7632,7 +7632,7 @@ M:        Masahiro Yamada <yamada.masahiro@socionext.com>
>  T:     git git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git kconfig
>  L:     linux-kbuild@vger.kernel.org
>  S:     Maintained
> -F:     Documentation/kbuild/kconfig-language.txt
> +F:     Documentation/kbuild/kconfig*
>  F:     scripts/kconfig/
>
>  KDUMP
> --
> 2.7.4
>



-- 
Kees Cook
Pixel Security
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 15/31] kconfig: add 'info', 'warning', and 'error' built-in functions

[Kees Cook]

On Wed, May 16, 2018 at 11:16 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Add 'info', 'warning', and 'error' functions as in Make.
>
> They print messages during parsing Kconfig files. 'error' will be
> useful to terminate the parsing immediately in fatal cases.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

> ---
>
> Changes in v4:
>   - Add 'error' function
>
> Changes in v3: None
> Changes in v2: None
>
>  scripts/kconfig/preprocess.c | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
>
> diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
> index 47b32dc..591bcf7 100644
> --- a/scripts/kconfig/preprocess.c
> +++ b/scripts/kconfig/preprocess.c
> @@ -104,6 +104,20 @@ struct function {
>         char *(*func)(int argc, char *argv[], int old_argc, char *old_argv[]);
>  };
>
> +static char *do_error(int argc, char *argv[], int old_argc, char *old_argv[])
> +{
> +       pperror("%s", argv[0]);
> +
> +       return NULL;
> +}
> +
> +static char *do_info(int argc, char *argv[], int old_argc, char *old_argv[])
> +{
> +       printf("%s\n", argv[0]);
> +
> +       return xstrdup("");
> +}
> +
>  static char *do_shell(int argc, char *argv[], int old_argc, char *old_argv[])
>  {
>         FILE *p;
> @@ -144,9 +158,19 @@ static char *do_shell(int argc, char *argv[], int old_argc, char *old_argv[])
>         return xstrdup(buf);
>  }
>
> +static char *do_warning(int argc, char *argv[], int old_argc, char *old_argv[])
> +{
> +       fprintf(stderr, "%s:%d: %s\n", current_file->name, yylineno, argv[0]);
> +
> +       return xstrdup("");
> +}
> +
>  static const struct function function_table[] = {
>         /* Name         MIN     MAX     EXP?    Function */
> +       { "error",      1,      1,      true,   do_error },
> +       { "info",       1,      1,      true,   do_info },
>         { "shell",      1,      1,      true,   do_shell },
> +       { "warning",    1,      1,      true,   do_warning },
>  };
>
>  #define FUNCTION_MAX_ARGS              16
> --
> 2.7.4
>



-- 
Kees Cook
Pixel Security

Re: [PATCH v4 17/31] kconfig: add 'filename' and 'lineno' built-in variables

[Kees Cook]

On Wed, May 16, 2018 at 11:16 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> The special variables, $(filename) and $(lineno), are expanded to a
> file name and its line number being parsed, respectively.
>
> Suggested-by: Randy Dunlap <rdunlap@infradead.org>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

> ---
>
> Changes in v4:
>  - Newly added
>
> Changes in v3: None
> Changes in v2: None
>
>  scripts/kconfig/preprocess.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
>
> diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
> index 88844a7..c39e30e 100644
> --- a/scripts/kconfig/preprocess.c
> +++ b/scripts/kconfig/preprocess.c
> @@ -111,6 +111,11 @@ static char *do_error(int argc, char *argv[], int old_argc, char *old_argv[])
>         return NULL;
>  }
>
> +static char *do_filename(int argc, char *argv[], int old_argc, char *old_argv[])
> +{
> +       return xstrdup(current_file->name);
> +}
> +
>  static char *do_if(int argc, char *argv[], int old_argc, char *old_argv[])
>  {
>         char *cond, *p, *res;
> @@ -144,6 +149,15 @@ static char *do_info(int argc, char *argv[], int old_argc, char *old_argv[])
>         return xstrdup("");
>  }
>
> +static char *do_lineno(int argc, char *argv[], int old_argc, char *old_argv[])
> +{
> +       char buf[16];
> +
> +       sprintf(buf, "%d", yylineno);
> +
> +       return xstrdup(buf);
> +}
> +
>  static char *do_shell(int argc, char *argv[], int old_argc, char *old_argv[])
>  {
>         FILE *p;
> @@ -194,8 +208,10 @@ static char *do_warning(int argc, char *argv[], int old_argc, char *old_argv[])
>  static const struct function function_table[] = {
>         /* Name         MIN     MAX     EXP?    Function */
>         { "error",      1,      1,      true,   do_error },
> +       { "filename",   0,      0,      false,  do_filename },
>         { "if",         2,      3,      false,  do_if },
>         { "info",       1,      1,      true,   do_info },
> +       { "lineno",     0,      0,      false,  do_lineno },
>         { "shell",      1,      1,      true,   do_shell },
>         { "warning",    1,      1,      true,   do_warning },
>  };
> --
> 2.7.4
>



-- 
Kees Cook
Pixel Security

Re: [PATCH v4 20/31] kconfig: test: add Kconfig macro language tests

[Kees Cook]

On Wed, May 16, 2018 at 11:16 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Here are the test cases I used for developing the text expansion
> feature.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Should these be tied to the global Makefile or selftests in any way?

-Kees

-- 
Kees Cook
Pixel Security

Re: [PATCH v4 20/31] kconfig: test: add Kconfig macro language tests

[Masahiro Yamada]

2018-05-17 15:41 GMT+09:00 Kees Cook <keescook@chromium.org>:
> On Wed, May 16, 2018 at 11:16 PM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> Here are the test cases I used for developing the text expansion
>> feature.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>
> Should these be tied to the global Makefile or selftests in any way?
>
> -Kees
>

It is already.

"make testconfig" will run Kconfig unit tests.

You need python3 and pytest for that.


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v4 19/31] Documentation: kconfig: document a new Kconfig macro language

[Masahiro Yamada]

2018-05-17 15:38 GMT+09:00 Kees Cook <keescook@chromium.org>:
> On Wed, May 16, 2018 at 11:16 PM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> Add a document for the macro language introduced to Kconfig.
>>
>> The motivation of this work is to move the compiler option tests to
>> Kconfig from Makefile.  A number of kernel features require the
>> compiler support.  Enabling such features blindly in Kconfig ends up
>> with a lot of nasty build-time testing in Makefiles.  If a chosen
>> feature turns out unsupported by the compiler, what the build system
>> can do is either to disable it (silently!) or to forcibly break the
>> build, despite Kconfig has let the user to enable it.  By moving the
>> compiler capability tests to Kconfig, features unsupported by the
>> compiler will be hidden automatically.
>>
>> This change was strongly prompted by Linus Torvalds.  You can find
>> his suggestions [1] [2] in ML.  The original idea was to add a new
>> attribute with 'option shell=...', but I found more generalized text
>> expansion would make Kconfig more powerful and lovely.  The basic
>> ideas are from Make, but there are some differences.
>>
>> [1]: https://lkml.org/lkml/2016/12/9/577
>> [2]: https://lkml.org/lkml/2018/2/7/527
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>
> (Added Randy, Jon, and linux-doc to CC for more review)
>
> This should likely be written in .rst and linked to from the developer index...
>
> https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#writing-documentation
>
> As for the content, though:
>
> Reviewed-by: Kees Cook <keescook@chromium.org>
>
> -Kees

At least, nothing in Documentation/kbuild/ has not been
converted to ReST yet.





>> ---
>>
>> Changes in v4:
>>  - Update according to the syntax change
>>
>> Changes in v3:
>>  - Newly added
>>
>> Changes in v2: None
>>
>>  Documentation/kbuild/kconfig-macro-language.txt | 252 ++++++++++++++++++++++++
>>  MAINTAINERS                                     |   2 +-
>>  2 files changed, 253 insertions(+), 1 deletion(-)
>>  create mode 100644 Documentation/kbuild/kconfig-macro-language.txt
>>
>> diff --git a/Documentation/kbuild/kconfig-macro-language.txt b/Documentation/kbuild/kconfig-macro-language.txt
>> new file mode 100644
>> index 0000000..a8dc792
>> --- /dev/null
>> +++ b/Documentation/kbuild/kconfig-macro-language.txt
>> @@ -0,0 +1,252 @@
>> +Concept
>> +-------
>> +
>> +The basic idea was inspired by Make. When we look at Make, we notice sort of
>> +two languages in one. One language describes dependency graphs consisting of
>> +targets and prerequisites. The other is a macro language for performing textual
>> +substitution.
>> +
>> +There is clear distinction between the two language stages. For example, you
>> +can write a makefile like follows:
>> +
>> +    APP := foo
>> +    SRC := foo.c
>> +    CC := gcc
>> +
>> +    $(APP): $(SRC)
>> +            $(CC) -o $(APP) $(SRC)
>> +
>> +The macro language replaces the variable references with their expanded form,
>> +and handles as if the source file were input like follows:
>> +
>> +    foo: foo.c
>> +            gcc -o foo foo.c
>> +
>> +Then, Make analyzes the dependency graph and determines the targets to be
>> +updated.
>> +
>> +The idea is quite similar in Kconfig - it is possible to describe a Kconfig
>> +file like this:
>> +
>> +    CC := gcc
>> +
>> +    config CC_HAS_FOO
>> +            def_bool $(shell, $(srctree)/scripts/gcc-check-foo.sh $(CC))
>> +
>> +The macro language in Kconfig processes the source file into the following
>> +intermediate:
>> +
>> +    config CC_HAS_FOO
>> +            def_bool y
>> +
>> +Then, Kconfig moves onto the evaluation stage to resolve inter-symbol
>> +dependency as explained in kconfig-language.txt.
>> +
>> +
>> +Variables
>> +---------
>> +
>> +Like in Make, a variable in Kconfig works as a macro variable.  A macro
>> +variable is expanded "in place" to yield a text string that may then be
>> +expanded further. To get the value of a variable, enclose the variable name in
>> +$( ). The parentheses are required even for single-letter variable names; $X is
>> +a syntax error. The curly brace form as in ${CC} is not supported either.
>> +
>> +There are two types of variables: simply expanded variables and recursively
>> +expanded variables.
>> +
>> +A simply expanded variable is defined using the := assignment operator. Its
>> +righthand side is expanded immediately upon reading the line from the Kconfig
>> +file.
>> +
>> +A recursively expanded variable is defined using the = assignment operator.
>> +Its righthand side is simply stored as the value of the variable without
>> +expanding it in any way. Instead, the expansion is performed when the variable
>> +is used.
>> +
>> +There is another type of assignment operator; += is used to append text to a
>> +variable. The righthand side of += is expanded immediately if the lefthand
>> +side was originally defined as a simple variable. Otherwise, its evaluation is
>> +deferred.
>> +
>> +The variable reference can take parameters, in the following form:
>> +
>> +  $(name,arg1,arg2,arg3)
>> +
>> +You can consider the parameterized reference as a function. (more precisely,
>> +"user-defined function" in the contrast to "built-in function" listed below).
>> +
>> +Useful functions must be expanded when they are used since the same function is
>> +expanded differently if different parameters are passed. Hence, a user-defined
>> +function is defined using the = assignment operator. The parameters are
>> +referenced within the body definition with $(1), $(2), etc.
>> +
>> +In fact, recursively expanded variables and user-defined functions are the same
>> +internally. (In other words, "variable" is "function with zero argument".)
>> +When we say "variable" in a broad sense, it includes "user-defined function".
>> +
>> +
>> +Built-in functions
>> +------------------
>> +
>> +Like Make, Kconfig provides several built-in functions. Every function takes a
>> +particular number of arguments.
>> +
>> +In Make, every built-in function takes at least one argument. Kconfig allows
>> +zero argument for built-in functions, such as $(fileno), $(lineno). You could
>> +consider those as "built-in variable", but it is just a matter of how we call
>> +it after all. Let's say "built-in function" here to refer to natively supported
>> +functionality.
>> +
>> +Kconfig currently supports the following built-in functions.
>> +
>> + - $(shell,command)
>> +
>> +  The "shell" function accepts a single argument that is expanded and passed
>> +  to a subshell for execution. The standard output of the command is then read
>> +  and returned as the value of the function. Every newline in the output is
>> +  replaced with a space. Any trailing newlines are deleted. The standard error
>> +  is not returned, nor is any program exit status.
>> +
>> + - $(info,text)
>> +
>> +  The "info" function takes a single argument and prints it to stdout.
>> +  It evaluates to an empty string.
>> +
>> + - $(warning,text)
>> +
>> +  The "warning" function is similar to "info" except that it sends its argument
>> +  to stderr and prefixes the output with the name of the current Kconfig file
>> +  and the current line number.
>> +
>> + - $(error,text)
>> +
>> +  The "error" function is similar to "warning", but it terminates the parsing
>> +  immediately.
>> +
>> + - $(if,condition,then-part[,else-part])
>> +
>> +  The "if" function takes two or three arguments ('else-part' is optional).
>> +  Depending on the value of the condition part, the argument to be expanded
>> +  is selected. The condition is true if its expansion contains any characters
>> +  except whitespaces. In this case, the then-part is expanded. Otherwise, the
>> +  else-part is expanded.
>> +
>> +  Note:
>> +  In Make, the condition is true if it contains any characters including
>> +  whitespaces, which is why $(strip ...) is sometimes necessary in the
>> +  condition part. Kconfig changed the behavior to make it handier.
>> +
>> + - $(filename)
>> +
>> +  The 'filename' takes no argument, and $(filename) is expanded to a file name
>> +  being parsed.
>> +
>> + - $(lineno)
>> +
>> +  The 'lineno' takes no argument, and $(lineno) is expanded to a line number
>> +  being parsed.
>> +
>> +
>> +Difference of function call syntax
>> +----------------------------------
>> +
>> +Kconfig adopts Make-like macro language, but the function call syntax is
>> +slightly different.
>> +
>> +A function call in Make looks like follows:
>> +
>> +  $(func-name arg1,arg2,arg3)
>> +
>> +The function name and the first argument are separated by at least one
>> +whitespace. Then, leading whitespaces are trimmed from the first argument,
>> +but whitespaces in the other arguments are kept. You need to use a kind of
>> +trick to start the first parameter with spaces. For example, if you want
>> +to make "info" function print "  hello", you can write like follows:
>> +
>> +  $(info $(space)$(space)hello)
>> +
>> +Kconfig uses only commas for delimiters, and keeps all whitespaces in the
>> +function call. Some people prefer putting a space after each comma delimiter:
>> +
>> +  $(func-name, arg1, arg2, arg3)
>> +
>> +In this case, "func-name" will receive " arg1", " arg2", " arg3". The presence
>> +of leading spaces may really matter depending on the function. The same applies
>> +to Make - for example, $(subst .c, .o, $(sources)) is a typical mistake.
>> +
>> +In Make, a user-defined function is referenced by using a built-in function,
>> +'call', like this:
>> +
>> +    $(call my-func,arg1,arg2,arg3)
>> +
>> +Kconfig invokes user-defined functions and built-in functions in the same way.
>> +The omission of 'call' makes the syntax shorter.
>> +
>> +In Make, some functions exceptionally treat commas verbatim instead of argument
>> +separators. For example, $(shell echo hello, world) evaluates to "hello, world".
>> +Likewise, $(info hello, world) prints "hello, world" to stdout. You could say
>> +this is _useful_ inconsistency.
>> +
>> +For simpler implementation and grammatical consistency, Kconfig always treats
>> +commas that appear in the $( ) form as delimiters. It means
>> +
>> +  $(shell, echo hello, world)
>> +
>> +is an error because it is passing two parameters where the 'shell' function
>> +accepts only one. To pass commas in arguments, you can use the following trick:
>> +
>> +  comma := ,
>> +  $(shell, echo hello$(comma) world)
>> +
>> +
>> +Caveats
>> +-------
>> +
>> +A variable (or function) cannot be expanded across tokens. So, you cannot use
>> +a variable as a shorthand for an expression that consists of multiple tokens.
>> +The following works:
>> +
>> +    RANGE_MIN := 1
>> +    RANGE_MAX := 3
>> +
>> +    config FOO
>> +            int "foo"
>> +            range $(RANGE_MIN) $(RANGE_MAX)
>> +
>> +But, the following does not work:
>> +
>> +    RANGES := 1 3
>> +
>> +    config FOO
>> +            int "foo"
>> +            range $(RANGES)
>> +
>> +A variable cannot be expanded to any keyword in Kconfig.  The following does
>> +not work:
>> +
>> +    MY_TYPE := tristate
>> +
>> +    config FOO
>> +            $(MY_TYPE) "foo"
>> +            default y
>> +
>> +Obviously from the design, $(shell command) is expanded in the textual
>> +substitution phase. You cannot pass symbols to the 'shell' function.
>> +The following does not work as expected.
>> +
>> +    config ENDIAN_FLAG
>> +            string
>> +            default "-mbig-endian" if CPU_BIG_ENDIAN
>> +            default "-mlittle-endian" if CPU_LITTLE_ENDIAN
>> +
>> +    config CC_HAS_ENDIAN_FLAG
>> +            def_bool $(shell $(srctree)/scripts/gcc-check-flag ENDIAN_FLAG)
>> +
>> +Instead, you can do like follows so that any function call is statically
>> +expanded.
>> +
>> +    config CC_HAS_ENDIAN_FLAG
>> +            bool
>> +            default $(shell $(srctree)/scripts/gcc-check-flag -mbig-endian) if CPU_BIG_ENDIAN
>> +            default $(shell $(srctree)/scripts/gcc-check-flag -mlittle-endian) if CPU_LITTLE_ENDIAN
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 58b9861..b7d7ae61 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -7632,7 +7632,7 @@ M:        Masahiro Yamada <yamada.masahiro@socionext.com>
>>  T:     git git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git kconfig
>>  L:     linux-kbuild@vger.kernel.org
>>  S:     Maintained
>> -F:     Documentation/kbuild/kconfig-language.txt
>> +F:     Documentation/kbuild/kconfig*
>>  F:     scripts/kconfig/
>>
>>  KDUMP
>> --
>> 2.7.4
>>
>
>
>
> --
> Kees Cook
> Pixel Security



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v4 19/31] Documentation: kconfig: document a new Kconfig macro language

[Masahiro Yamada]

2018-05-17 15:38 GMT+09:00 Kees Cook <keescook@chromium.org>:
> On Wed, May 16, 2018 at 11:16 PM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> Add a document for the macro language introduced to Kconfig.
>>
>> The motivation of this work is to move the compiler option tests to
>> Kconfig from Makefile.  A number of kernel features require the
>> compiler support.  Enabling such features blindly in Kconfig ends up
>> with a lot of nasty build-time testing in Makefiles.  If a chosen
>> feature turns out unsupported by the compiler, what the build system
>> can do is either to disable it (silently!) or to forcibly break the
>> build, despite Kconfig has let the user to enable it.  By moving the
>> compiler capability tests to Kconfig, features unsupported by the
>> compiler will be hidden automatically.
>>
>> This change was strongly prompted by Linus Torvalds.  You can find
>> his suggestions [1] [2] in ML.  The original idea was to add a new
>> attribute with 'option shell=...', but I found more generalized text
>> expansion would make Kconfig more powerful and lovely.  The basic
>> ideas are from Make, but there are some differences.
>>
>> [1]: https://lkml.org/lkml/2016/12/9/577
>> [2]: https://lkml.org/lkml/2018/2/7/527
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>
> (Added Randy, Jon, and linux-doc to CC for more review)
>
> This should likely be written in .rst and linked to from the developer index...
>
> https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#writing-documentation
>
> As for the content, though:
>
> Reviewed-by: Kees Cook <keescook@chromium.org>
>
> -Kees

At least, nothing in Documentation/kbuild/ has not been
converted to ReST yet.





>> ---
>>
>> Changes in v4:
>>  - Update according to the syntax change
>>
>> Changes in v3:
>>  - Newly added
>>
>> Changes in v2: None
>>
>>  Documentation/kbuild/kconfig-macro-language.txt | 252 ++++++++++++++++++++++++
>>  MAINTAINERS                                     |   2 +-
>>  2 files changed, 253 insertions(+), 1 deletion(-)
>>  create mode 100644 Documentation/kbuild/kconfig-macro-language.txt
>>
>> diff --git a/Documentation/kbuild/kconfig-macro-language.txt b/Documentation/kbuild/kconfig-macro-language.txt
>> new file mode 100644
>> index 0000000..a8dc792
>> --- /dev/null
>> +++ b/Documentation/kbuild/kconfig-macro-language.txt
>> @@ -0,0 +1,252 @@
>> +Concept
>> +-------
>> +
>> +The basic idea was inspired by Make. When we look at Make, we notice sort of
>> +two languages in one. One language describes dependency graphs consisting of
>> +targets and prerequisites. The other is a macro language for performing textual
>> +substitution.
>> +
>> +There is clear distinction between the two language stages. For example, you
>> +can write a makefile like follows:
>> +
>> +    APP := foo
>> +    SRC := foo.c
>> +    CC := gcc
>> +
>> +    $(APP): $(SRC)
>> +            $(CC) -o $(APP) $(SRC)
>> +
>> +The macro language replaces the variable references with their expanded form,
>> +and handles as if the source file were input like follows:
>> +
>> +    foo: foo.c
>> +            gcc -o foo foo.c
>> +
>> +Then, Make analyzes the dependency graph and determines the targets to be
>> +updated.
>> +
>> +The idea is quite similar in Kconfig - it is possible to describe a Kconfig
>> +file like this:
>> +
>> +    CC := gcc
>> +
>> +    config CC_HAS_FOO
>> +            def_bool $(shell, $(srctree)/scripts/gcc-check-foo.sh $(CC))
>> +
>> +The macro language in Kconfig processes the source file into the following
>> +intermediate:
>> +
>> +    config CC_HAS_FOO
>> +            def_bool y
>> +
>> +Then, Kconfig moves onto the evaluation stage to resolve inter-symbol
>> +dependency as explained in kconfig-language.txt.
>> +
>> +
>> +Variables
>> +---------
>> +
>> +Like in Make, a variable in Kconfig works as a macro variable.  A macro
>> +variable is expanded "in place" to yield a text string that may then be
>> +expanded further. To get the value of a variable, enclose the variable name in
>> +$( ). The parentheses are required even for single-letter variable names; $X is
>> +a syntax error. The curly brace form as in ${CC} is not supported either.
>> +
>> +There are two types of variables: simply expanded variables and recursively
>> +expanded variables.
>> +
>> +A simply expanded variable is defined using the := assignment operator. Its
>> +righthand side is expanded immediately upon reading the line from the Kconfig
>> +file.
>> +
>> +A recursively expanded variable is defined using the = assignment operator.
>> +Its righthand side is simply stored as the value of the variable without
>> +expanding it in any way. Instead, the expansion is performed when the variable
>> +is used.
>> +
>> +There is another type of assignment operator; += is used to append text to a
>> +variable. The righthand side of += is expanded immediately if the lefthand
>> +side was originally defined as a simple variable. Otherwise, its evaluation is
>> +deferred.
>> +
>> +The variable reference can take parameters, in the following form:
>> +
>> +  $(name,arg1,arg2,arg3)
>> +
>> +You can consider the parameterized reference as a function. (more precisely,
>> +"user-defined function" in the contrast to "built-in function" listed below).
>> +
>> +Useful functions must be expanded when they are used since the same function is
>> +expanded differently if different parameters are passed. Hence, a user-defined
>> +function is defined using the = assignment operator. The parameters are
>> +referenced within the body definition with $(1), $(2), etc.
>> +
>> +In fact, recursively expanded variables and user-defined functions are the same
>> +internally. (In other words, "variable" is "function with zero argument".)
>> +When we say "variable" in a broad sense, it includes "user-defined function".
>> +
>> +
>> +Built-in functions
>> +------------------
>> +
>> +Like Make, Kconfig provides several built-in functions. Every function takes a
>> +particular number of arguments.
>> +
>> +In Make, every built-in function takes at least one argument. Kconfig allows
>> +zero argument for built-in functions, such as $(fileno), $(lineno). You could
>> +consider those as "built-in variable", but it is just a matter of how we call
>> +it after all. Let's say "built-in function" here to refer to natively supported
>> +functionality.
>> +
>> +Kconfig currently supports the following built-in functions.
>> +
>> + - $(shell,command)
>> +
>> +  The "shell" function accepts a single argument that is expanded and passed
>> +  to a subshell for execution. The standard output of the command is then read
>> +  and returned as the value of the function. Every newline in the output is
>> +  replaced with a space. Any trailing newlines are deleted. The standard error
>> +  is not returned, nor is any program exit status.
>> +
>> + - $(info,text)
>> +
>> +  The "info" function takes a single argument and prints it to stdout.
>> +  It evaluates to an empty string.
>> +
>> + - $(warning,text)
>> +
>> +  The "warning" function is similar to "info" except that it sends its argument
>> +  to stderr and prefixes the output with the name of the current Kconfig file
>> +  and the current line number.
>> +
>> + - $(error,text)
>> +
>> +  The "error" function is similar to "warning", but it terminates the parsing
>> +  immediately.
>> +
>> + - $(if,condition,then-part[,else-part])
>> +
>> +  The "if" function takes two or three arguments ('else-part' is optional).
>> +  Depending on the value of the condition part, the argument to be expanded
>> +  is selected. The condition is true if its expansion contains any characters
>> +  except whitespaces. In this case, the then-part is expanded. Otherwise, the
>> +  else-part is expanded.
>> +
>> +  Note:
>> +  In Make, the condition is true if it contains any characters including
>> +  whitespaces, which is why $(strip ...) is sometimes necessary in the
>> +  condition part. Kconfig changed the behavior to make it handier.
>> +
>> + - $(filename)
>> +
>> +  The 'filename' takes no argument, and $(filename) is expanded to a file name
>> +  being parsed.
>> +
>> + - $(lineno)
>> +
>> +  The 'lineno' takes no argument, and $(lineno) is expanded to a line number
>> +  being parsed.
>> +
>> +
>> +Difference of function call syntax
>> +----------------------------------
>> +
>> +Kconfig adopts Make-like macro language, but the function call syntax is
>> +slightly different.
>> +
>> +A function call in Make looks like follows:
>> +
>> +  $(func-name arg1,arg2,arg3)
>> +
>> +The function name and the first argument are separated by at least one
>> +whitespace. Then, leading whitespaces are trimmed from the first argument,
>> +but whitespaces in the other arguments are kept. You need to use a kind of
>> +trick to start the first parameter with spaces. For example, if you want
>> +to make "info" function print "  hello", you can write like follows:
>> +
>> +  $(info $(space)$(space)hello)
>> +
>> +Kconfig uses only commas for delimiters, and keeps all whitespaces in the
>> +function call. Some people prefer putting a space after each comma delimiter:
>> +
>> +  $(func-name, arg1, arg2, arg3)
>> +
>> +In this case, "func-name" will receive " arg1", " arg2", " arg3". The presence
>> +of leading spaces may really matter depending on the function. The same applies
>> +to Make - for example, $(subst .c, .o, $(sources)) is a typical mistake.
>> +
>> +In Make, a user-defined function is referenced by using a built-in function,
>> +'call', like this:
>> +
>> +    $(call my-func,arg1,arg2,arg3)
>> +
>> +Kconfig invokes user-defined functions and built-in functions in the same way.
>> +The omission of 'call' makes the syntax shorter.
>> +
>> +In Make, some functions exceptionally treat commas verbatim instead of argument
>> +separators. For example, $(shell echo hello, world) evaluates to "hello, world".
>> +Likewise, $(info hello, world) prints "hello, world" to stdout. You could say
>> +this is _useful_ inconsistency.
>> +
>> +For simpler implementation and grammatical consistency, Kconfig always treats
>> +commas that appear in the $( ) form as delimiters. It means
>> +
>> +  $(shell, echo hello, world)
>> +
>> +is an error because it is passing two parameters where the 'shell' function
>> +accepts only one. To pass commas in arguments, you can use the following trick:
>> +
>> +  comma := ,
>> +  $(shell, echo hello$(comma) world)
>> +
>> +
>> +Caveats
>> +-------
>> +
>> +A variable (or function) cannot be expanded across tokens. So, you cannot use
>> +a variable as a shorthand for an expression that consists of multiple tokens.
>> +The following works:
>> +
>> +    RANGE_MIN := 1
>> +    RANGE_MAX := 3
>> +
>> +    config FOO
>> +            int "foo"
>> +            range $(RANGE_MIN) $(RANGE_MAX)
>> +
>> +But, the following does not work:
>> +
>> +    RANGES := 1 3
>> +
>> +    config FOO
>> +            int "foo"
>> +            range $(RANGES)
>> +
>> +A variable cannot be expanded to any keyword in Kconfig.  The following does
>> +not work:
>> +
>> +    MY_TYPE := tristate
>> +
>> +    config FOO
>> +            $(MY_TYPE) "foo"
>> +            default y
>> +
>> +Obviously from the design, $(shell command) is expanded in the textual
>> +substitution phase. You cannot pass symbols to the 'shell' function.
>> +The following does not work as expected.
>> +
>> +    config ENDIAN_FLAG
>> +            string
>> +            default "-mbig-endian" if CPU_BIG_ENDIAN
>> +            default "-mlittle-endian" if CPU_LITTLE_ENDIAN
>> +
>> +    config CC_HAS_ENDIAN_FLAG
>> +            def_bool $(shell $(srctree)/scripts/gcc-check-flag ENDIAN_FLAG)
>> +
>> +Instead, you can do like follows so that any function call is statically
>> +expanded.
>> +
>> +    config CC_HAS_ENDIAN_FLAG
>> +            bool
>> +            default $(shell $(srctree)/scripts/gcc-check-flag -mbig-endian) if CPU_BIG_ENDIAN
>> +            default $(shell $(srctree)/scripts/gcc-check-flag -mlittle-endian) if CPU_LITTLE_ENDIAN
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 58b9861..b7d7ae61 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -7632,7 +7632,7 @@ M:        Masahiro Yamada <yamada.masahiro@socionext.com>
>>  T:     git git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git kconfig
>>  L:     linux-kbuild@vger.kernel.org
>>  S:     Maintained
>> -F:     Documentation/kbuild/kconfig-language.txt
>> +F:     Documentation/kbuild/kconfig*
>>  F:     scripts/kconfig/
>>
>>  KDUMP
>> --
>> 2.7.4
>>
>
>
>
> --
> Kees Cook
> Pixel Security



-- 
Best Regards
Masahiro Yamada
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 00/31] kconfig: move compiler capability tests to Kconfig

[Nicholas Piggin]

On Thu, 17 May 2018 15:16:39 +0900
Masahiro Yamada <yamada.masahiro@socionext.com> wrote:

> [Introduction]
> 
> The motivation of this work is to move the compiler option tests to
> Kconfig from Makefile.  A number of kernel features require the
> compiler support.  Enabling such features blindly in Kconfig ends up
> with a lot of nasty build-time testing in Makefiles.  If a chosen
> feature turns out unsupported by the compiler, what the build system
> can do is either to disable it (silently!) or to forcibly break the
> build, despite Kconfig has let the user to enable it.  By moving the
> compiler capability tests to Kconfig, Kconfig entries will be visible
> only when they are available.
> 
> [Major Changes in V4]

Do you have a git tree for v4? I can test it with the powerpc patches.

The new scripting capability in kconfig has allowed us to already
improve the config process on powerpc:

https://marc.info/?l=linuxppc-embedded&m=152648110727868&w=2

I'm sure there's more clever things we can do with it but I haven't
had time to think about it yet. One thing that comes to mind is that
It might be nice to show the option as disabled, then the user could
upgrade their compiler to get the options they want.

Anyway v3 worked fine for me, the documentation is really nice, I
could implement the above patch without any problem despite being a
kbuild dummy. Thanks for the series, ack from me.

Thanks,
Nick

Re: [PATCH v4 00/31] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

2018-05-17 16:51 GMT+09:00 Nicholas Piggin <npiggin@gmail.com>:
> On Thu, 17 May 2018 15:16:39 +0900
> Masahiro Yamada <yamada.masahiro@socionext.com> wrote:
>
>> [Introduction]
>>
>> The motivation of this work is to move the compiler option tests to
>> Kconfig from Makefile.  A number of kernel features require the
>> compiler support.  Enabling such features blindly in Kconfig ends up
>> with a lot of nasty build-time testing in Makefiles.  If a chosen
>> feature turns out unsupported by the compiler, what the build system
>> can do is either to disable it (silently!) or to forcibly break the
>> build, despite Kconfig has let the user to enable it.  By moving the
>> compiler capability tests to Kconfig, Kconfig entries will be visible
>> only when they are available.
>>
>> [Major Changes in V4]
>
> Do you have a git tree for v4? I can test it with the powerpc patches.
>
> The new scripting capability in kconfig has allowed us to already
> improve the config process on powerpc:
>
> https://marc.info/?l=linuxppc-embedded&m=152648110727868&w=2
>
> I'm sure there's more clever things we can do with it but I haven't
> had time to think about it yet. One thing that comes to mind is that
> It might be nice to show the option as disabled, then the user could
> upgrade their compiler to get the options they want.
>
> Anyway v3 worked fine for me, the documentation is really nice, I
> could implement the above patch without any problem despite being a
> kbuild dummy. Thanks for the series, ack from me.


For easier review and test,
I pushed v4 to the following branch:


git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
kconfig-shell-v4


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v4 05/31] kconfig: remove string expansion for mainmenu after yyparse()

[Sam Ravnborg]

On Thu, May 17, 2018 at 03:16:44PM +0900, Masahiro Yamada wrote:
> Now that environments are expanded in the lexer, conf_parse() does
> not need to expand them explicitly.
> 
> The hack introduced by commit 0724a7c32a54 ("kconfig: Don't leak
> main menus during parsing") can go away.
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> Reviewed-by: Kees Cook <keescook@chromium.org>
> Reviewed-by: Ulf Magnusson <ulfalizer@gmail.com>
> ---
> 
> Changes in v4: None
> Changes in v3: None
> Changes in v2:
>   - Simplify the patch.  Just remove the text expansion.
> 
>  scripts/kconfig/zconf.y | 24 +++++-------------------
>  1 file changed, 5 insertions(+), 19 deletions(-)
> 
> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
> index 3a4a0fa..22e318c 100644
> --- a/scripts/kconfig/zconf.y
> +++ b/scripts/kconfig/zconf.y
> @@ -109,7 +109,7 @@ static struct menu *current_menu, *current_entry;
>  %%
>  input: nl start | start;
>  
> -start: mainmenu_stmt stmt_list | no_mainmenu_stmt stmt_list;
> +start: mainmenu_stmt stmt_list | stmt_list;
>  
>  /* mainmenu entry */
>  
> @@ -118,19 +118,6 @@ mainmenu_stmt: T_MAINMENU prompt nl
>  	menu_add_prompt(P_MENU, $2, NULL);
>  };
>  
> -/* Default main menu, if there's no mainmenu entry */
> -
> -no_mainmenu_stmt: /* empty */
> -{
> -	/*
> -	 * Hack: Keep the main menu title on the heap so we can safely free it
> -	 * later regardless of whether it comes from the 'prompt' in
> -	 * mainmenu_stmt or here
> -	 */
> -	menu_add_prompt(P_MENU, xstrdup("Linux Kernel Configuration"), NULL);
> -};
> -
> -
>  stmt_list:
>  	  /* empty */
>  	| stmt_list common_stmt
> @@ -528,7 +515,6 @@ word_opt: /* empty */			{ $$ = NULL; }
>  
>  void conf_parse(const char *name)
>  {
> -	const char *tmp;
>  	struct symbol *sym;
>  	int i;
>  
> @@ -544,10 +530,10 @@ void conf_parse(const char *name)
>  	if (!modules_sym)
>  		modules_sym = sym_find( "n" );
>  
> -	tmp = rootmenu.prompt->text;
> -	rootmenu.prompt->text = _(rootmenu.prompt->text);
> -	rootmenu.prompt->text = sym_expand_string_value(rootmenu.prompt->text);
> -	free((char*)tmp);
> +	if (!menu_has_prompt(&rootmenu)) {
> +		current_entry = &rootmenu;
> +		menu_add_prompt(P_MENU, "Linux Kernel Configuration", NULL);

Could this be something less specific?
To help the other users of kconfig out is the wild.

For example:

	menu_add_prompt(P_MENU, _("Main menu"), NULL);

Note, the added _() to make the text translateable.

	Sam

Re: [PATCH v4 07/31] kconfig: add built-in function support

[Sam Ravnborg]

On Thu, May 17, 2018 at 03:16:46PM +0900, Masahiro Yamada wrote:
> This commit adds a new concept 'function' to do more text processing
> in Kconfig.
> 
> A function call looks like this:
> 
>   $(function,arg1,arg2,arg3,...)
> 
> This commit adds the basic infrastructure to expand functions.
> Change the text expansion helpers to take arguments.
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
> 
> Changes in v4:
>   - Error out if arguments more than FUNCTION_MAX_ARGS are passed
>   - Use a comma as a delimiter between the function name and the
>     first argument
>   - Check the number of arguments accepted by each function
>   - Support delayed expansion of arguments.
>     This will be needed to implement 'if' function
> 
> Changes in v3:
>   - Split base infrastructure and 'shell' function
>     into separate patches.
> 
> Changes in v2:
>   - Use 'shell' for getting stdout from the comment.
>     It was 'shell-stdout' in the previous version.
>   - Simplify the implementation since the expansion has been moved to
>     lexer.
> 
>  scripts/kconfig/preprocess.c | 168 ++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 159 insertions(+), 9 deletions(-)
> 
> diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
> index 1bf506c..5be28ec 100644
> --- a/scripts/kconfig/preprocess.c
> +++ b/scripts/kconfig/preprocess.c
> @@ -3,12 +3,17 @@
>  // Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
>  
>  #include <stdarg.h>
> +#include <stdbool.h>
>  #include <stdio.h>
>  #include <stdlib.h>
>  #include <string.h>
>  
>  #include "list.h"
>  
> +#define ARRAY_SIZE(arr)		(sizeof(arr) / sizeof((arr)[0]))
> +
> +static char *expand_string_with_args(const char *in, int argc, char *argv[]);
> +
>  static void __attribute__((noreturn)) pperror(const char *format, ...)
>  {
>  	va_list ap;
> @@ -88,9 +93,85 @@ void env_write_dep(FILE *f, const char *autoconfig_name)
>  	}
>  }
>  
> -static char *eval_clause(const char *in)
> +/*
> + * Built-in functions
> + */
> +struct function {
> +	const char *name;
> +	unsigned int min_args;
> +	unsigned int max_args;
> +	bool expand_args;
> +	char *(*func)(int argc, char *argv[], int old_argc, char *old_argv[]);
> +};
If a typedef was provided for the function then ...

> +
> +static const struct function function_table[] = {
> +	/* Name		MIN	MAX	EXP?	Function */
> +};
> +
> +#define FUNCTION_MAX_ARGS		16
> +
> +static char *function_expand_arg_and_call(char *(*func)(int argc, char *argv[],
> +							int old_argc,
> +							char *old_argv[]),
> +					  int argc, char *argv[],
> +					  int old_argc, char *old_argv[])
this could be much easier to read.

> +{
> +	char *expanded_argv[FUNCTION_MAX_ARGS], *res;
> +	int i;
> +
> +	for (i = 0; i < argc; i++)
> +		expanded_argv[i] = expand_string_with_args(argv[i],
> +							   old_argc, old_argv);

No check for too many arguments here - maybe it is done in some other place.

> +
> +	res = func(argc, expanded_argv, 0, NULL);
> +
> +	for (i = 0; i < argc; i++)
> +		free(expanded_argv[i]);
> +
> +	return res;
> +}
> +
> +static char *function_call(const char *name, int argc, char *argv[],
> +			   int old_argc, char *old_argv[])
> +{
> +	const struct function *f;
> +	int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(function_table); i++) {
> +		f = &function_table[i];
> +		if (strcmp(f->name, name))
> +			continue;
> +
> +		if (argc < f->min_args)
> +			pperror("too few function arguments passed to '%s'",
> +				name);
> +
> +		if (argc > f->max_args)
> +			pperror("too many function arguments passed to '%s'",
> +				name);
Number of arguments checked here, but max_args is not assiged in this patch.


> +
> +		if (f->expand_args)
> +			return function_expand_arg_and_call(f->func, argc, argv,
> +							    old_argc, old_argv);
> +		return f->func(argc, argv, old_argc, old_argv);
> +	}
> +
> +	return NULL;
> +}
> +
> +/*
> + * Evaluate a clause with arguments.  argc/argv are arguments from the upper
> + * function call.
> + *
> + * Returned string must be freed when done
> + */
> +static char *eval_clause(const char *in, int argc, char *argv[])
>  {
> -	char *res, *name;
> +	char *tmp, *prev, *p, *res, *name;
> +	int new_argc = 0;
> +	char *new_argv[FUNCTION_MAX_ARGS];
> +	int nest = 0;
> +	int i;
>  
>  	/*
>  	 * Returns an empty string because '$()' should be evaluated
> @@ -99,10 +180,69 @@ static char *eval_clause(const char *in)
>  	if (!*in)
>  		return xstrdup("");
>  
> -	name = expand_string(in);
> +	tmp = xstrdup(in);
> +
> +	prev = p = tmp;
> +
> +	/*
> +	 * Split into tokens
> +	 * The function name and arguments are separated by a comma.
> +	 * For example, if the function call is like this:
> +	 *   $(foo,abc,$(x),$(y))
> +	 *
> +	 * The input string for this helper should be:
> +	 *   foo,abc,$(x),$(y)
> +	 *
> +	 * and split into:
> +	 *   new_argv[0] = 'foo'
> +	 *   new_argv[1] = 'abc'
> +	 *   new_argv[2] = '$(x)'
> +	 *   new_argv[3] = '$(y)'
> +	 */
> +	while (*p) {
> +		if (nest == 0 && *p == ',') {
> +			*p = 0;
> +			if (new_argc >= FUNCTION_MAX_ARGS)
> +				pperror("too many function arguments");
> +			new_argv[new_argc++] = prev;
> +			prev = p + 1;
> +		} else if (*p == '(') {
> +			nest++;
> +		} else if (*p == ')') {
> +			nest--;
> +		}
> +
> +		p++;
> +	}
> +	new_argv[new_argc++] = prev;

Will the following be equal:

	$(foo,abc,$(x),$(y))
	$(foo, abc, $(x), $(y))

make is rather annoying as space is significant, but there seems no good reason
for kconfig to inheritate this.
So unless there are good arguments consider alloing the spaces.
If the current implmentation already supports optional spaces then I just missed
it whie reviewing.

	Sam

Re: [PATCH v4 03/31] kconfig: reference environment variables directly and remove 'option env='

[Ulf Magnusson]

On Thu, May 17, 2018 at 8:16 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> To get access to environment variables, Kconfig needs to define a
> symbol using "option env=" syntax.  It is tedious to add a symbol entry
> for each environment variable given that we need to define much more
> such as 'CC', 'AS', 'srctree' etc. to evaluate the compiler capability
> in Kconfig.
>
> Adding '$' for symbol references is grammatically inconsistent.
> Looking at the code, the symbols prefixed with 'S' are expanded by:
>  - conf_expand_value()
>    This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
>  - sym_expand_string_value()
>    This is used to expand strings in 'source' and 'mainmenu'
>
> All of them are fixed values independent of user configuration.  So,
> they can be changed into the direct expansion instead of symbols.
>
> This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
> 'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.
>
> sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
> should be replaced with an environment variable.
>
> ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
> without '$' prefix.
>
> The new syntax is addicted by Make.  The variable reference needs
> parentheses, like $(FOO), but you can omit them for single-letter
> variables, like $F.  Yet, in Makefiles, people tend to use the
> parenthetical form for consistency / clarification.
>
> At this moment, only the environment variable is supported, but I will
> extend the concept of 'variable' later on.
>
> The variables are expanded in the lexer so we can simplify the token
> handling on the parser side.
>
> For example, the following code works.
>
> [Example code]
>
>   config MY_TOOLCHAIN_LIST
>           string
>           default "My tools: CC=$(CC), AS=$(AS), CPP=$(CPP)"
>
> [Result]
>
>   $ make -s alldefconfig && tail -n 1 .config
>   CONFIG_MY_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> Reviewed-by: Kees Cook <keescook@chromium.org>
> ---
>
> Changes in v4:
>   - Enclose ARCH in conf_defname
>   - Drop single-letter support
>
> Changes in v3:
>   - Reimplement
>   - Variable reference need parentheses except single-letter variable
>
> Changes in v2:
>   - Move the string expansion to the lexer phase.
>   - Split environment helpers to env.c
>
>  Documentation/kbuild/kconfig-language.txt |   8 -
>  Kconfig                                   |   8 +-
>  Makefile                                  |   3 +-
>  arch/sh/Kconfig                           |   4 +-
>  arch/sparc/Kconfig                        |   4 +-
>  arch/um/Kconfig.common                    |   4 -
>  arch/x86/Kconfig                          |   4 +-
>  arch/x86/um/Kconfig                       |   6 +-
>  init/Kconfig                              |  16 +-
>  scripts/kconfig/confdata.c                |  33 +---
>  scripts/kconfig/kconf_id.c                |   1 -
>  scripts/kconfig/lkc.h                     |   4 -
>  scripts/kconfig/lkc_proto.h               |   6 +
>  scripts/kconfig/menu.c                    |   3 -
>  scripts/kconfig/preprocess.c              | 247 ++++++++++++++++++++++++++++++
>  scripts/kconfig/symbol.c                  |  56 -------
>  scripts/kconfig/util.c                    |  18 +--
>  scripts/kconfig/zconf.l                   |  67 +++++++-
>  scripts/kconfig/zconf.y                   |   2 +-
>  19 files changed, 340 insertions(+), 154 deletions(-)
>  create mode 100644 scripts/kconfig/preprocess.c
>
> diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
> index f5b9493..0e966e8 100644
> --- a/Documentation/kbuild/kconfig-language.txt
> +++ b/Documentation/kbuild/kconfig-language.txt
> @@ -198,14 +198,6 @@ applicable everywhere (see syntax).
>      enables the third modular state for all config symbols.
>      At most one symbol may have the "modules" option set.
>
> -  - "env"=<value>
> -    This imports the environment variable into Kconfig. It behaves like
> -    a default, except that the value comes from the environment, this
> -    also means that the behaviour when mixing it with normal defaults is
> -    undefined at this point. The symbol is currently not exported back
> -    to the build environment (if this is desired, it can be done via
> -    another symbol).
> -
>    - "allnoconfig_y"
>      This declares the symbol as one that should have the value y when
>      using "allnoconfig". Used for symbols that hide other symbols.
> diff --git a/Kconfig b/Kconfig
> index 8c4c1cb..4af1b42 100644
> --- a/Kconfig
> +++ b/Kconfig
> @@ -3,10 +3,6 @@
>  # For a description of the syntax of this configuration file,
>  # see Documentation/kbuild/kconfig-language.txt.
>  #
> -mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
> +mainmenu "Linux/$(ARCH) $(KERNELVERSION) Kernel Configuration"
>
> -config SRCARCH
> -       string
> -       option env="SRCARCH"
> -
> -source "arch/$SRCARCH/Kconfig"
> +source "arch/$(SRCARCH)/Kconfig"
> diff --git a/Makefile b/Makefile
> index 01b2211..c6278e6 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -284,7 +284,8 @@ include scripts/Kbuild.include
>  # Read KERNELRELEASE from include/config/kernel.release (if it exists)
>  KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
>  KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
> -export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
> +UNAME_RELEASE := $(shell uname --release)
> +export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
>
>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>  # first, and if a usermode build is happening, the "ARCH=um" on the command
> diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
> index 1851eae..c8400e3 100644
> --- a/arch/sh/Kconfig
> +++ b/arch/sh/Kconfig
> @@ -58,7 +58,7 @@ config SUPERH
>           <http://www.linux-sh.org/>.
>
>  config SUPERH32
> -       def_bool ARCH = "sh"
> +       def_bool "$(ARCH)" = "sh"
>         select HAVE_KPROBES
>         select HAVE_KRETPROBES
>         select HAVE_IOREMAP_PROT if MMU && !X2TLB
> @@ -77,7 +77,7 @@ config SUPERH32
>         select HAVE_CC_STACKPROTECTOR
>
>  config SUPERH64
> -       def_bool ARCH = "sh64"
> +       def_bool "$(ARCH)" = "sh64"
>         select HAVE_EXIT_THREAD
>         select KALLSYMS
>
> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
> index 8767e45..df7410c 100644
> --- a/arch/sparc/Kconfig
> +++ b/arch/sparc/Kconfig
> @@ -1,6 +1,6 @@
>  config 64BIT
> -       bool "64-bit kernel" if ARCH = "sparc"
> -       default ARCH = "sparc64"
> +       bool "64-bit kernel" if "$(ARCH)" = "sparc"
> +       default "$(ARCH)" = "sparc64"
>         help
>           SPARC is a family of RISC microprocessors designed and marketed by
>           Sun Microsystems, incorporated.  They are very widely found in Sun
> diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
> index c68add8..07f84c8 100644
> --- a/arch/um/Kconfig.common
> +++ b/arch/um/Kconfig.common
> @@ -54,10 +54,6 @@ config HZ
>         int
>         default 100
>
> -config SUBARCH
> -       string
> -       option env="SUBARCH"
> -
>  config NR_CPUS
>         int
>         range 1 1
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index c07f492..2236505 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1,8 +1,8 @@
>  # SPDX-License-Identifier: GPL-2.0
>  # Select 32 or 64 bit
>  config 64BIT
> -       bool "64-bit kernel" if ARCH = "x86"
> -       default ARCH != "i386"
> +       bool "64-bit kernel" if "$(ARCH)" = "x86"
> +       default "$(ARCH)" != "i386"
>         ---help---
>           Say yes to build a 64-bit kernel - formerly known as x86_64
>           Say no to build a 32-bit kernel - formerly known as i386
> diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
> index 13ed827..6a15c4d 100644
> --- a/arch/x86/um/Kconfig
> +++ b/arch/x86/um/Kconfig
> @@ -1,5 +1,5 @@
>  # SPDX-License-Identifier: GPL-2.0
> -mainmenu "User Mode Linux/$SUBARCH $KERNELVERSION Kernel Configuration"
> +mainmenu "User Mode Linux/$(SUBARCH) $(KERNELVERSION) Kernel Configuration"
>
>  source "arch/um/Kconfig.common"
>
> @@ -16,8 +16,8 @@ config UML_X86
>         select GENERIC_FIND_FIRST_BIT
>
>  config 64BIT
> -       bool "64-bit kernel" if SUBARCH = "x86"
> -       default SUBARCH != "i386"
> +       bool "64-bit kernel" if "$(SUBARCH)" = "x86"
> +       default "$(SUBARCH)" != "i386"
>
>  config X86_32
>         def_bool !64BIT
> diff --git a/init/Kconfig b/init/Kconfig
> index 4537962..752816b 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1,20 +1,12 @@
> -config ARCH
> -       string
> -       option env="ARCH"
> -
> -config KERNELVERSION
> -       string
> -       option env="KERNELVERSION"
> -
>  config DEFCONFIG_LIST
>         string
>         depends on !UML
>         option defconfig_list
> -       default "/lib/modules/$UNAME_RELEASE/.config"
> +       default "/lib/modules/$(UNAME_RELEASE)/.config"
>         default "/etc/kernel-config"
> -       default "/boot/config-$UNAME_RELEASE"
> -       default "$ARCH_DEFCONFIG"
> -       default "arch/$ARCH/defconfig"
> +       default "/boot/config-$(UNAME_RELEASE)"
> +       default ARCH_DEFCONFIG
> +       default "arch/$(ARCH)/defconfig"
>
>  config CONSTRUCTORS
>         bool
> diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
> index df26c7b..f72587c 100644
> --- a/scripts/kconfig/confdata.c
> +++ b/scripts/kconfig/confdata.c
> @@ -30,7 +30,7 @@ static void conf_message(const char *fmt, ...)
>  static const char *conf_filename;
>  static int conf_lineno, conf_warnings;
>
> -const char conf_defname[] = "arch/$ARCH/defconfig";
> +const char conf_defname[] = "arch/$(ARCH)/defconfig";
>
>  static void conf_warning(const char *fmt, ...)
>  {
> @@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
>         return name ? name : "include/config/auto.conf";
>  }
>
> -static char *conf_expand_value(const char *in)
> -{
> -       struct symbol *sym;
> -       const char *src;
> -       static char res_value[SYMBOL_MAXLENGTH];
> -       char *dst, name[SYMBOL_MAXLENGTH];
> -
> -       res_value[0] = 0;
> -       dst = name;
> -       while ((src = strchr(in, '$'))) {
> -               strncat(res_value, in, src - in);
> -               src++;
> -               dst = name;
> -               while (isalnum(*src) || *src == '_')
> -                       *dst++ = *src++;
> -               *dst = 0;
> -               sym = sym_lookup(name, 0);
> -               sym_calc_value(sym);
> -               strcat(res_value, sym_get_string_value(sym));
> -               in = src;
> -       }
> -       strcat(res_value, in);
> -
> -       return res_value;
> -}
> -
>  char *conf_get_default_confname(void)
>  {
>         struct stat buf;
>         static char fullname[PATH_MAX+1];
>         char *env, *name;
>
> -       name = conf_expand_value(conf_defname);
> +       name = expand_string(conf_defname);
>         env = getenv(SRCTREE);
>         if (env) {
>                 sprintf(fullname, "%s/%s", env, name);
> @@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
>                         if (expr_calc_value(prop->visible.expr) == no ||
>                             prop->expr->type != E_SYMBOL)
>                                 continue;
> -                       name = conf_expand_value(prop->expr->left.sym->name);
> +                       sym_calc_value(prop->expr->left.sym);
> +                       name = sym_get_string_value(prop->expr->left.sym);
>                         in = zconf_fopen(name);
>                         if (in) {
>                                 conf_message(_("using defaults found in %s"),
> diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
> index 3ea9c5f..b3e0ea0 100644
> --- a/scripts/kconfig/kconf_id.c
> +++ b/scripts/kconfig/kconf_id.c
> @@ -32,7 +32,6 @@ static struct kconf_id kconf_id_array[] = {
>         { "on",                 T_ON,                   TF_PARAM },
>         { "modules",            T_OPT_MODULES,          TF_OPTION },
>         { "defconfig_list",     T_OPT_DEFCONFIG_LIST,   TF_OPTION },
> -       { "env",                T_OPT_ENV,              TF_OPTION },
>         { "allnoconfig_y",      T_OPT_ALLNOCONFIG_Y,    TF_OPTION },
>  };
>
> diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
> index f4394af..553098a 100644
> --- a/scripts/kconfig/lkc.h
> +++ b/scripts/kconfig/lkc.h
> @@ -58,7 +58,6 @@ enum conf_def_mode {
>
>  #define T_OPT_MODULES          1
>  #define T_OPT_DEFCONFIG_LIST   2
> -#define T_OPT_ENV              3
>  #define T_OPT_ALLNOCONFIG_Y    4
>
>  struct kconf_id {
> @@ -134,9 +133,6 @@ void str_printf(struct gstr *gs, const char *fmt, ...);
>  const char *str_get(struct gstr *gs);
>
>  /* symbol.c */
> -extern struct expr *sym_env_list;
> -
> -void sym_init(void);
>  void sym_clear_all_valid(void);
>  struct symbol *sym_choice_default(struct symbol *sym);
>  const char *sym_get_string_default(struct symbol *sym);
> diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
> index 9dc8abf..9f465fe 100644
> --- a/scripts/kconfig/lkc_proto.h
> +++ b/scripts/kconfig/lkc_proto.h
> @@ -49,5 +49,11 @@ const char * sym_get_string_value(struct symbol *sym);
>
>  const char * prop_get_type_name(enum prop_type type);
>
> +/* preprocess.c */
> +void env_write_dep(FILE *f, const char *auto_conf_name);
> +char *expand_string(const char *in);
> +char *expand_dollar(const char **str);
> +char *expand_one_token(const char **str);
> +
>  /* expr.c */
>  void expr_print(struct expr *e, void (*fn)(void *, struct symbol *, const char *), void *data, int prevtoken);
> diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
> index 5c5c137..8148305 100644
> --- a/scripts/kconfig/menu.c
> +++ b/scripts/kconfig/menu.c
> @@ -214,9 +214,6 @@ void menu_add_option(int token, char *arg)
>                         zconf_error("trying to redefine defconfig symbol");
>                 sym_defconfig_list->flags |= SYMBOL_AUTO;
>                 break;
> -       case T_OPT_ENV:
> -               prop_add_env(arg);
> -               break;
>         case T_OPT_ALLNOCONFIG_Y:
>                 current_entry->sym->flags |= SYMBOL_ALLNOCONFIG_Y;
>                 break;
> diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
> new file mode 100644
> index 0000000..1bf506c
> --- /dev/null
> +++ b/scripts/kconfig/preprocess.c
> @@ -0,0 +1,247 @@
> +// SPDX-License-Identifier: GPL-2.0
> +//
> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
> +
> +#include <stdarg.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +
> +#include "list.h"
> +
> +static void __attribute__((noreturn)) pperror(const char *format, ...)
> +{
> +       va_list ap;
> +
> +       fprintf(stderr, "%s:%d: ", current_file->name, yylineno);
> +       va_start(ap, format);
> +       vfprintf(stderr, format, ap);
> +       va_end(ap);
> +       fprintf(stderr, "\n");
> +
> +       exit(1);
> +}
> +
> +/*
> + * Environment variables
> + */
> +static LIST_HEAD(env_list);
> +
> +struct env {
> +       char *name;
> +       char *value;
> +       struct list_head node;
> +};
> +
> +static void env_add(const char *name, const char *value)
> +{
> +       struct env *e;
> +
> +       e = xmalloc(sizeof(*e));
> +       e->name = xstrdup(name);
> +       e->value = xstrdup(value);
> +
> +       list_add_tail(&e->node, &env_list);
> +}
> +
> +static void env_del(struct env *e)
> +{
> +       list_del(&e->node);
> +       free(e->name);
> +       free(e->value);
> +       free(e);
> +}
> +
> +/* The returned pointer must be freed when done */
> +static char *env_expand(const char *name)
> +{
> +       struct env *e;
> +       const char *value;
> +
> +       list_for_each_entry(e, &env_list, node) {
> +               if (!strcmp(name, e->name))
> +                       return xstrdup(e->value);
> +       }
> +
> +       value = getenv(name);
> +       if (!value)
> +               value = "";
> +
> +       /*
> +        * We need to remember all referenced environments.

s/environments/environment variables/

> +        * They will be written out to include/config/auto.conf.cmd
> +        */
> +       env_add(name, value);
> +
> +       return xstrdup(value);
> +}
> +
> +void env_write_dep(FILE *f, const char *autoconfig_name)
> +{
> +       struct env *e, *tmp;
> +
> +       list_for_each_entry_safe(e, tmp, &env_list, node) {
> +               fprintf(f, "ifneq \"$(%s)\" \"%s\"\n", e->name, e->value);
> +               fprintf(f, "%s: FORCE\n", autoconfig_name);
> +               fprintf(f, "endif\n");
> +               env_del(e);
> +       }
> +}
> +
> +static char *eval_clause(const char *in)
> +{
> +       char *res, *name;
> +
> +       /*
> +        * Returns an empty string because '$()' should be evaluated
> +        * to a null string.
> +        */

Do you know of cases where this is more useful than erroring out?

Not saying it doesn't make sense. Just curious.

> +       if (!*in)
> +               return xstrdup("");
> +
> +       name = expand_string(in);
> +
> +       res = env_expand(name);
> +       free(name);
> +
> +       return res;
> +}
> +
> +/*
> + * Expand a string that follows '$'
> + *
> + * For example, if the input string is
> + *     ($(FOO)$($(BAR)))$(BAZ)
> + * this helper evaluates
> + *     $($(FOO)$($(BAR)))
> + * and returns the resulted string, then updates 'str' to point to the next

s/resulted/resulting/

Maybe something like this would make the behavior a bit clearer:

  ...and returns a new string containing the expansion, also advancing
  'str' to point to the next character after... (note that this function does
  a recursive expansion) ...

> + * character after the corresponding closing parenthesis, in this case, *str
> + * will be
> + *     $(BAR)
> + */
> +char *expand_dollar(const char **str)
> +{
> +       const char *p = *str;
> +       const char *q;
> +       char *tmp, *out;
> +       int nest = 0;
> +
> +       /* '$$' represents an escaped '$' */
> +       if (*p == '$') {
> +               *str = p + 1;
> +               return xstrdup("$");
> +       }
> +
> +       /*
> +        * Kconfig does not support single-letter variable as in $A
> +        * or curly braces as in ${CC}.
> +        */
> +       if (*p != '(')
> +               pperror("syntax error: '$' not followed by '('", p);

Could say "not followed by '(' by or '$'".

> +
> +       p++;
> +       q = p;
> +       while (*q) {
> +               if (*q == '(') {
> +                       nest++;
> +               } else if (*q == ')') {
> +                       if (nest-- == 0)
> +                               break;
> +               }
> +               q++;
> +       }
> +
> +       if (!*q)
> +               pperror("unterminated reference to '%s': missing ')'", p);
> +
> +       tmp = xmalloc(q - p + 1);
> +       memcpy(tmp, p, q - p);
> +       tmp[q - p] = 0;
> +       *str = q + 1;
> +       out = eval_clause(tmp);
> +       free(tmp);
> +
> +       return out;

This might be a bit clearer, since the 'str' update and the expansion
are independent:

  /* Advance 'str' to after the expanded initial portion of the string */
  *str = q + 1;

  /* Save the "FOO" part of "(FOO)" into 'tmp' and expand it recursively */
  tmp = xmalloc(q - p + 1);
  memcpy(tmp, p, q - p);
  tmp[q - p] = '\0';
  out = eval_clause(tmp);
  free(tmp);

  return out;

...or switched around, but thought putting the 'str' bit first might
emphasize that it isn't modified.

> +}
> +
> +/*
> + * Expand variables in the given string.  Undefined variables
> + * expand to an empty string.

I wonder what the tradeoffs are vs. erroring out here (or leaving
undefined variables as-is).

> + * The returned string must be freed when done.
> + */
> +char *expand_string(const char *in)
> +{
> +       const char *prev_in, *p;
> +       char *new, *out;
> +       size_t outlen;
> +
> +       out = xmalloc(1);
> +       *out = 0;
> +
> +       while ((p = strchr(in, '$'))) {
> +               prev_in = in;
> +               in = p + 1;
> +               new = expand_dollar(&in);
> +               outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
> +               out = xrealloc(out, outlen);
> +               strncat(out, prev_in, p - prev_in);
> +               strcat(out, new);
> +               free(new);

Some code duplication with expand_one_token() here. Do you think
something could be factored out/reorganized?

I wonder if it might be cleaner to have expand_dollar() take a pointer
to the '$'. Might even out in terms of the +1's you'd have to add, as
all callers manually bump the pointer before the call now. I haven't
tried implementing it though, so hard to say.

Some short inline comments similar to above would help too I think.

> +       }
> +
> +       outlen = strlen(out) + strlen(in) + 1;
> +       out = xrealloc(out, outlen);
> +       strcat(out, in);
> +
> +       return out;
> +}
> +
> +/*
> + * Expand variables in a token.  The parsing stops when a token separater
> + * (in most cases, it is a whitespace) is encountered.  'str' is updated to
> + * point to the next character.
> + *
> + * The returned string must be freed when done.
> + */
> +char *expand_one_token(const char **str)
> +{
> +       const char *in, *prev_in, *p;
> +       char *new, *out;
> +       size_t outlen;
> +
> +       out = xmalloc(1);
> +       *out = 0;
> +
> +       p = in = *str;
> +
> +       while (1) {
> +               if (*p == '$') {
> +                       prev_in = in;
> +                       in = p + 1;
> +                       new = expand_dollar(&in);
> +                       outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
> +                       out = xrealloc(out, outlen);
> +                       strncat(out, prev_in, p - prev_in);
> +                       strcat(out, new);
> +                       free(new);
> +                       p = in;
> +                       continue;
> +               }
> +
> +               /* Valid characters in a symbol (why '.' and '/' ?) */
> +               if (isalnum(*p) || *p == '_' || *p == '-' || *p == '.' || *p == '/') {
> +                       p++;
> +                       continue;
> +               }
> +
> +               break;
> +       }
> +
> +       outlen = strlen(out) + (p - in) + 1;
> +       out = xrealloc(out, outlen);
> +       strncat(out, in, p - in);
> +
> +       *str = p;
> +
> +       return out;
> +}
> diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
> index f0b2e3b..2460648 100644
> --- a/scripts/kconfig/symbol.c
> +++ b/scripts/kconfig/symbol.c
> @@ -33,33 +33,6 @@ struct symbol *sym_defconfig_list;
>  struct symbol *modules_sym;
>  tristate modules_val;
>
> -struct expr *sym_env_list;
> -
> -static void sym_add_default(struct symbol *sym, const char *def)
> -{
> -       struct property *prop = prop_alloc(P_DEFAULT, sym);
> -
> -       prop->expr = expr_alloc_symbol(sym_lookup(def, SYMBOL_CONST));
> -}
> -
> -void sym_init(void)
> -{
> -       struct symbol *sym;
> -       struct utsname uts;
> -       static bool inited = false;
> -
> -       if (inited)
> -               return;
> -       inited = true;
> -
> -       uname(&uts);
> -
> -       sym = sym_lookup("UNAME_RELEASE", 0);
> -       sym->type = S_STRING;
> -       sym->flags |= SYMBOL_AUTO;
> -       sym_add_default(sym, uts.release);
> -}
> -
>  enum symbol_type sym_get_type(struct symbol *sym)
>  {
>         enum symbol_type type = sym->type;
> @@ -1401,32 +1374,3 @@ const char *prop_get_type_name(enum prop_type type)
>         }
>         return "unknown";
>  }
> -
> -static void prop_add_env(const char *env)
> -{
> -       struct symbol *sym, *sym2;
> -       struct property *prop;
> -       char *p;
> -
> -       sym = current_entry->sym;
> -       sym->flags |= SYMBOL_AUTO;
> -       for_all_properties(sym, prop, P_ENV) {
> -               sym2 = prop_get_symbol(prop);
> -               if (strcmp(sym2->name, env))
> -                       menu_warn(current_entry, "redefining environment symbol from %s",
> -                                 sym2->name);
> -               return;
> -       }
> -
> -       prop = prop_alloc(P_ENV, sym);
> -       prop->expr = expr_alloc_symbol(sym_lookup(env, SYMBOL_CONST));
> -
> -       sym_env_list = expr_alloc_one(E_LIST, sym_env_list);
> -       sym_env_list->right.sym = sym;
> -
> -       p = getenv(env);
> -       if (p)
> -               sym_add_default(sym, p);
> -       else
> -               menu_warn(current_entry, "environment variable %s undefined", env);
> -}
> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
> index c6f6e21..807147e 100644
> --- a/scripts/kconfig/util.c
> +++ b/scripts/kconfig/util.c
> @@ -34,8 +34,6 @@ struct file *file_lookup(const char *name)
>  /* write a dependency file as used by kbuild to track dependencies */
>  int file_write_dep(const char *name)
>  {
> -       struct symbol *sym, *env_sym;
> -       struct expr *e;
>         struct file *file;
>         FILE *out;
>
> @@ -54,21 +52,7 @@ int file_write_dep(const char *name)
>         fprintf(out, "\n%s: \\\n"
>                      "\t$(deps_config)\n\n", conf_get_autoconfig_name());
>
> -       expr_list_for_each_sym(sym_env_list, e, sym) {
> -               struct property *prop;
> -               const char *value;
> -
> -               prop = sym_get_env_prop(sym);
> -               env_sym = prop_get_symbol(prop);
> -               if (!env_sym)
> -                       continue;
> -               value = getenv(env_sym->name);
> -               if (!value)
> -                       value = "";
> -               fprintf(out, "ifneq \"$(%s)\" \"%s\"\n", env_sym->name, value);
> -               fprintf(out, "%s: FORCE\n", conf_get_autoconfig_name());
> -               fprintf(out, "endif\n");
> -       }
> +       env_write_dep(out, conf_get_autoconfig_name());
>
>         fprintf(out, "\n$(deps_config): ;\n");
>         fclose(out);
> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
> index 045093d..9dc5fe3 100644
> --- a/scripts/kconfig/zconf.l
> +++ b/scripts/kconfig/zconf.l
> @@ -1,6 +1,5 @@
>  %option nostdinit noyywrap never-interactive full ecs
>  %option 8bit nodefault yylineno
> -%option noinput
>  %x COMMAND HELP STRING PARAM
>  %{
>  /*
> @@ -35,6 +34,8 @@ struct buffer *current_buf;
>
>  static int last_ts, first_ts;
>
> +static char *expand_token(const char *in, size_t n);
> +static void append_expanded_string(const char *in);
>  static void zconf_endhelp(void);
>  static void zconf_endfile(void);
>
> @@ -147,6 +148,13 @@ n  [A-Za-z0-9_-]
>                 yylval.string = text;
>                 return T_WORD;
>         }
> +       ({n}|[/.$])+    {
> +               /* this token includes at least one '$' */
> +               yylval.string = expand_token(yytext, yyleng);
> +               if (strlen(yylval.string))
> +                       return T_WORD;
> +               free(yylval.string);
> +       }
>         #.*     /* comment */
>         \\\n    ;
>         [[:blank:]]+
> @@ -157,12 +165,13 @@ n [A-Za-z0-9_-]
>  }
>
>  <STRING>{
> -       [^'"\\\n]+/\n   {
> +       "$".*   append_expanded_string(yytext);
> +       [^$'"\\\n]+/\n  {
>                 append_string(yytext, yyleng);
>                 yylval.string = text;
>                 return T_WORD_QUOTE;
>         }
> -       [^'"\\\n]+      {
> +       [^$'"\\\n]+     {
>                 append_string(yytext, yyleng);
>         }
>         \\.?/\n {
> @@ -249,6 +258,58 @@ n  [A-Za-z0-9_-]
>  }
>
>  %%
> +static char *expand_token(const char *in, size_t n)
> +{
> +       char *out;
> +       int c;
> +       char c2;
> +       const char *rest, *end;
> +
> +       new_string();
> +       append_string(in, n);
> +
> +       /* get the whole the line because we do not know the end of token. */
> +       while ((c = input()) != EOF) {
> +               if (c == '\n') {
> +                       unput(c);
> +                       break;
> +               }
> +               c2 = c;
> +               append_string(&c2, 1);
> +       }
> +
> +       rest = text;
> +       out = expand_one_token(&rest);
> +
> +       /* push back unused characters to the input stream */
> +       end = rest + strlen(rest);
> +       while (end > rest)
> +               unput(*--end);
> +
> +       free(text);
> +
> +       return out;
> +}
> +
> +static void append_expanded_string(const char *str)
> +{
> +       const char *end;
> +       char *res;
> +
> +       str++;
> +
> +       res = expand_dollar(&str);
> +
> +       /* push back unused characters to the input stream */
> +       end = str + strlen(str);
> +       while (end > str)
> +               unput(*--end);
> +
> +       append_string(res, strlen(res));
> +
> +       free(res);
> +}
> +
>  void zconf_starthelp(void)
>  {
>         new_string();
> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
> index ad6305b..3a4a0fa 100644
> --- a/scripts/kconfig/zconf.y
> +++ b/scripts/kconfig/zconf.y
> @@ -534,7 +534,6 @@ void conf_parse(const char *name)
>
>         zconf_initscan(name);
>
> -       sym_init();
>         _menu_init();
>
>         if (getenv("ZCONF_DEBUG"))
> @@ -780,3 +779,4 @@ void zconfdump(FILE *out)
>  #include "expr.c"
>  #include "symbol.c"
>  #include "menu.c"
> +#include "preprocess.c"
> --
> 2.7.4
>

Cheers,
Ulf

Re: [PATCH v4 03/31] kconfig: reference environment variables directly and remove 'option env='

[Masahiro Yamada]

Hi.



2018-05-21 0:46 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:

> s/environments/environment variables/

Will fix.


>
>> +        * They will be written out to include/config/auto.conf.cmd
>> +        */
>> +       env_add(name, value);
>> +
>> +       return xstrdup(value);
>> +}
>> +
>> +void env_write_dep(FILE *f, const char *autoconfig_name)
>> +{
>> +       struct env *e, *tmp;
>> +
>> +       list_for_each_entry_safe(e, tmp, &env_list, node) {
>> +               fprintf(f, "ifneq \"$(%s)\" \"%s\"\n", e->name, e->value);
>> +               fprintf(f, "%s: FORCE\n", autoconfig_name);
>> +               fprintf(f, "endif\n");
>> +               env_del(e);
>> +       }
>> +}
>> +
>> +static char *eval_clause(const char *in)
>> +{
>> +       char *res, *name;
>> +
>> +       /*
>> +        * Returns an empty string because '$()' should be evaluated
>> +        * to a null string.
>> +        */
>
> Do you know of cases where this is more useful than erroring out?
>
> Not saying it doesn't make sense. Just curious.


I just followed how Make works.

Anyway, eval_clause() will return null string for null input.
I will remove that hunk.




>> +       if (!*in)
>> +               return xstrdup("");
>> +
>> +       name = expand_string(in);
>> +
>> +       res = env_expand(name);
>> +       free(name);
>> +
>> +       return res;
>> +}
>> +
>> +/*
>> + * Expand a string that follows '$'
>> + *
>> + * For example, if the input string is
>> + *     ($(FOO)$($(BAR)))$(BAZ)
>> + * this helper evaluates
>> + *     $($(FOO)$($(BAR)))
>> + * and returns the resulted string, then updates 'str' to point to the next
>
> s/resulted/resulting/
>
> Maybe something like this would make the behavior a bit clearer:
>
>   ...and returns a new string containing the expansion, also advancing
>   'str' to point to the next character after... (note that this function does
>   a recursive expansion) ...


Is this OK?

/*
 * Expand a string that follows '$'
 *
 * For example, if the input string is
 *     ($(FOO)$($(BAR)))$(BAZ)
 * this helper evaluates
 *     $($(FOO)$($(BAR)))
 * and returns a new string containing the expansion (note that the string is
 * recursively expanded), also advancing 'str' to point to the next character
 * after the corresponding closing parenthesis, in this case, *str will be
 *     $(BAR)
 */



>> + * character after the corresponding closing parenthesis, in this case, *str
>> + * will be
>> + *     $(BAR)
>> + */
>> +char *expand_dollar(const char **str)
>> +{
>> +       const char *p = *str;
>> +       const char *q;
>> +       char *tmp, *out;
>> +       int nest = 0;
>> +
>> +       /* '$$' represents an escaped '$' */
>> +       if (*p == '$') {
>> +               *str = p + 1;
>> +               return xstrdup("$");
>> +       }
>> +
>> +       /*
>> +        * Kconfig does not support single-letter variable as in $A
>> +        * or curly braces as in ${CC}.
>> +        */
>> +       if (*p != '(')
>> +               pperror("syntax error: '$' not followed by '('", p);
>
> Could say "not followed by '(' by or '$'".

Will do.


>> +
>> +       p++;
>> +       q = p;
>> +       while (*q) {
>> +               if (*q == '(') {
>> +                       nest++;
>> +               } else if (*q == ')') {
>> +                       if (nest-- == 0)
>> +                               break;
>> +               }
>> +               q++;
>> +       }
>> +
>> +       if (!*q)
>> +               pperror("unterminated reference to '%s': missing ')'", p);
>> +
>> +       tmp = xmalloc(q - p + 1);
>> +       memcpy(tmp, p, q - p);
>> +       tmp[q - p] = 0;
>> +       *str = q + 1;
>> +       out = eval_clause(tmp);
>> +       free(tmp);
>> +
>> +       return out;
>
> This might be a bit clearer, since the 'str' update and the expansion
> are independent:

Indeed, will do.

>
>   /* Advance 'str' to after the expanded initial portion of the string */
>   *str = q + 1;
>
>   /* Save the "FOO" part of "(FOO)" into 'tmp' and expand it recursively */
>   tmp = xmalloc(q - p + 1);
>   memcpy(tmp, p, q - p);
>   tmp[q - p] = '\0';
>   out = eval_clause(tmp);
>   free(tmp);
>
>   return out;
>
> ...or switched around, but thought putting the 'str' bit first might
> emphasize that it isn't modified.


I prefer advancing the pointer at last.




>> +}
>> +
>> +/*
>> + * Expand variables in the given string.  Undefined variables
>> + * expand to an empty string.
>
> I wonder what the tradeoffs are vs. erroring out here (or leaving
> undefined variables as-is).


I want to stick to the Make-like behavior here and exploit it in some cases.
We can set some variables in some arch/*/Kconfig,
but unset in the others.




In some arch/*/Kconfig:

min-gcc-version := 40900




In init/Kconfig:

# GCC 4.5 is required to build Linux Kernel.
# Some architectures may override it (from arch/*/Kconfig) for their
requirement.
min-gcc-version := $(if,$(min-gcc-version),$(min-gcc-version),40500)

... check the gcc version, then show error
if it is older than $(min-gcc-version).








>> + * The returned string must be freed when done.
>> + */
>> +char *expand_string(const char *in)
>> +{
>> +       const char *prev_in, *p;
>> +       char *new, *out;
>> +       size_t outlen;
>> +
>> +       out = xmalloc(1);
>> +       *out = 0;
>> +
>> +       while ((p = strchr(in, '$'))) {
>> +               prev_in = in;
>> +               in = p + 1;
>> +               new = expand_dollar(&in);
>> +               outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
>> +               out = xrealloc(out, outlen);
>> +               strncat(out, prev_in, p - prev_in);
>> +               strcat(out, new);
>> +               free(new);
>
> Some code duplication with expand_one_token() here. Do you think
> something could be factored out/reorganized?


Yes.  For example, the following would work.
If you prefer that, I can update it in v5.




static char *__expand_string(const char **str, bool (*is_end)(const char *))
{
        const char *in, *prev_in, *p;
        char *new, *out;
        size_t outlen;

        out = xmalloc(1);
        *out = 0;

        p = in = *str;

        while (1) {
                if (*p == '$') {
                        prev_in = in;
                        in = p + 1;
                        new = expand_dollar(&in);
                        outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
                        out = xrealloc(out, outlen);
                        strncat(out, prev_in, p - prev_in);
                        strcat(out, new);
                        free(new);
                        p = in;
                        continue;
                }

                if (is_end(p))
                        break;

                p++;
        }

        outlen = strlen(out) + (p - in) + 1;
        out = xrealloc(out, outlen);
        strncat(out, in, p - in);

        *str = p;

        return out;
}

static bool is_end_of_str(const char *s)
{
        return !*s;
}

char *expand_string(const char *in)
{
        return __expand_string(&in, is_end_of_str);
}

static bool is_end_of_token(const char *s)
{
        return !(isalnum(*s) || *s == '_' || *s == '-' || *s == '.' ||
*s == '/');
}

char *expand_one_token(const char **str)
{
        return __expand_string(str, is_end_of_token);
}





> I wonder if it might be cleaner to have expand_dollar() take a pointer
> to the '$'. Might even out in terms of the +1's you'd have to add, as
> all callers manually bump the pointer before the call now. I haven't
> tried implementing it though, so hard to say.


If I do this, I'd want to add a sanity check to expand_dollar().
I'd rather like to avoid unneeded code.


static char *expand_dollar(...)
{
        assert(*p == '$');
        p++;

        /* '$$' represents an escaped '$' */
        if (*p == '$') {
               *str = p + 1;
               return xstrdup("$");
        }

         ...


}


>
> Some short inline comments similar to above would help too I think.
>



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v4 07/31] kconfig: add built-in function support

[Masahiro Yamada]

Sam,

2018-05-20 23:50 GMT+09:00 Sam Ravnborg <sam@ravnborg.org>:
> On Thu, May 17, 2018 at 03:16:46PM +0900, Masahiro Yamada wrote:
>> This commit adds a new concept 'function' to do more text processing
>> in Kconfig.
>>
>> A function call looks like this:
>>
>>   $(function,arg1,arg2,arg3,...)
>>
>> This commit adds the basic infrastructure to expand functions.
>> Change the text expansion helpers to take arguments.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> Changes in v4:
>>   - Error out if arguments more than FUNCTION_MAX_ARGS are passed
>>   - Use a comma as a delimiter between the function name and the
>>     first argument
>>   - Check the number of arguments accepted by each function
>>   - Support delayed expansion of arguments.
>>     This will be needed to implement 'if' function
>>
>> Changes in v3:
>>   - Split base infrastructure and 'shell' function
>>     into separate patches.
>>
>> Changes in v2:
>>   - Use 'shell' for getting stdout from the comment.
>>     It was 'shell-stdout' in the previous version.
>>   - Simplify the implementation since the expansion has been moved to
>>     lexer.
>>
>>  scripts/kconfig/preprocess.c | 168 ++++++++++++++++++++++++++++++++++++++++---
>>  1 file changed, 159 insertions(+), 9 deletions(-)
>>
>> diff --git a/scripts/kconfig/preprocess.c b/scripts/kconfig/preprocess.c
>> index 1bf506c..5be28ec 100644
>> --- a/scripts/kconfig/preprocess.c
>> +++ b/scripts/kconfig/preprocess.c
>> @@ -3,12 +3,17 @@
>>  // Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
>>
>>  #include <stdarg.h>
>> +#include <stdbool.h>
>>  #include <stdio.h>
>>  #include <stdlib.h>
>>  #include <string.h>
>>
>>  #include "list.h"
>>
>> +#define ARRAY_SIZE(arr)              (sizeof(arr) / sizeof((arr)[0]))
>> +
>> +static char *expand_string_with_args(const char *in, int argc, char *argv[]);
>> +
>>  static void __attribute__((noreturn)) pperror(const char *format, ...)
>>  {
>>       va_list ap;
>> @@ -88,9 +93,85 @@ void env_write_dep(FILE *f, const char *autoconfig_name)
>>       }
>>  }
>>
>> -static char *eval_clause(const char *in)
>> +/*
>> + * Built-in functions
>> + */
>> +struct function {
>> +     const char *name;
>> +     unsigned int min_args;
>> +     unsigned int max_args;
>> +     bool expand_args;
>> +     char *(*func)(int argc, char *argv[], int old_argc, char *old_argv[]);
>> +};
> If a typedef was provided for the function then ...


Yes, I can do this,
but I may rather consider to simplify the code.


>> +
>> +static const struct function function_table[] = {
>> +     /* Name         MIN     MAX     EXP?    Function */
>> +};
>> +
>> +#define FUNCTION_MAX_ARGS            16
>> +
>> +static char *function_expand_arg_and_call(char *(*func)(int argc, char *argv[],
>> +                                                     int old_argc,
>> +                                                     char *old_argv[]),
>> +                                       int argc, char *argv[],
>> +                                       int old_argc, char *old_argv[])
> this could be much easier to read.
>
>> +{
>> +     char *expanded_argv[FUNCTION_MAX_ARGS], *res;
>> +     int i;
>> +
>> +     for (i = 0; i < argc; i++)
>> +             expanded_argv[i] = expand_string_with_args(argv[i],
>> +                                                        old_argc, old_argv);
>
> No check for too many arguments here - maybe it is done in some other place.

Right.
This has already been checked by eval_clause().


>> +
>> +     res = func(argc, expanded_argv, 0, NULL);
>> +
>> +     for (i = 0; i < argc; i++)
>> +             free(expanded_argv[i]);
>> +
>> +     return res;
>> +}
>> +
>> +static char *function_call(const char *name, int argc, char *argv[],
>> +                        int old_argc, char *old_argv[])
>> +{
>> +     const struct function *f;
>> +     int i;
>> +
>> +     for (i = 0; i < ARRAY_SIZE(function_table); i++) {
>> +             f = &function_table[i];
>> +             if (strcmp(f->name, name))
>> +                     continue;
>> +
>> +             if (argc < f->min_args)
>> +                     pperror("too few function arguments passed to '%s'",
>> +                             name);
>> +
>> +             if (argc > f->max_args)
>> +                     pperror("too many function arguments passed to '%s'",
>> +                             name);
> Number of arguments checked here, but max_args is not assiged in this patch.

This is added to function_table[] by later patches.


>
>> +
>> +             if (f->expand_args)
>> +                     return function_expand_arg_and_call(f->func, argc, argv,
>> +                                                         old_argc, old_argv);
>> +             return f->func(argc, argv, old_argc, old_argv);
>> +     }
>> +
>> +     return NULL;
>> +}
>> +
>> +/*
>> + * Evaluate a clause with arguments.  argc/argv are arguments from the upper
>> + * function call.
>> + *
>> + * Returned string must be freed when done
>> + */
>> +static char *eval_clause(const char *in, int argc, char *argv[])
>>  {
>> -     char *res, *name;
>> +     char *tmp, *prev, *p, *res, *name;
>> +     int new_argc = 0;
>> +     char *new_argv[FUNCTION_MAX_ARGS];
>> +     int nest = 0;
>> +     int i;
>>
>>       /*
>>        * Returns an empty string because '$()' should be evaluated
>> @@ -99,10 +180,69 @@ static char *eval_clause(const char *in)
>>       if (!*in)
>>               return xstrdup("");
>>
>> -     name = expand_string(in);
>> +     tmp = xstrdup(in);
>> +
>> +     prev = p = tmp;
>> +
>> +     /*
>> +      * Split into tokens
>> +      * The function name and arguments are separated by a comma.
>> +      * For example, if the function call is like this:
>> +      *   $(foo,abc,$(x),$(y))
>> +      *
>> +      * The input string for this helper should be:
>> +      *   foo,abc,$(x),$(y)
>> +      *
>> +      * and split into:
>> +      *   new_argv[0] = 'foo'
>> +      *   new_argv[1] = 'abc'
>> +      *   new_argv[2] = '$(x)'
>> +      *   new_argv[3] = '$(y)'
>> +      */
>> +     while (*p) {
>> +             if (nest == 0 && *p == ',') {
>> +                     *p = 0;
>> +                     if (new_argc >= FUNCTION_MAX_ARGS)
>> +                             pperror("too many function arguments");
>> +                     new_argv[new_argc++] = prev;
>> +                     prev = p + 1;
>> +             } else if (*p == '(') {
>> +                     nest++;
>> +             } else if (*p == ')') {
>> +                     nest--;
>> +             }
>> +
>> +             p++;
>> +     }
>> +     new_argv[new_argc++] = prev;
>
> Will the following be equal:
>
>         $(foo,abc,$(x),$(y))
>         $(foo, abc, $(x), $(y))
>
> make is rather annoying as space is significant, but there seems no good reason
> for kconfig to inheritate this.
> So unless there are good arguments consider alloing the spaces.
> If the current implmentation already supports optional spaces then I just missed
> it whie reviewing.


I have been thinking of trimming the leading whitespaces.
(https://patchwork.kernel.org/patch/10405549/)

This is trade-off vs "how to pass spaces as arguments?"

GNU Make trims any leading whitespaces from the first argument.
At least, it was tedious to print messages with indentation.


$(info Tool information:)
$(info   CC: $(CC))
$(info   LD: $(LD))

will print

Tool information:
CC: gcc
LD: ld


To keep the indentation, I need to do like follows:

$(info Tool information:)
$(info $(space)$(space)CC: $(CC))
$(info $(space)$(space)LD: $(LD))


If this is acceptable limitation,
yes, I can do this.




-- 
Best Regards
Masahiro Yamada

Re: [PATCH v4 05/31] kconfig: remove string expansion for mainmenu after yyparse()

[Masahiro Yamada]

2018-05-20 23:39 GMT+09:00 Sam Ravnborg <sam@ravnborg.org>:

>> @@ -544,10 +530,10 @@ void conf_parse(const char *name)
>>       if (!modules_sym)
>>               modules_sym = sym_find( "n" );
>>
>> -     tmp = rootmenu.prompt->text;
>> -     rootmenu.prompt->text = _(rootmenu.prompt->text);
>> -     rootmenu.prompt->text = sym_expand_string_value(rootmenu.prompt->text);
>> -     free((char*)tmp);
>> +     if (!menu_has_prompt(&rootmenu)) {
>> +             current_entry = &rootmenu;
>> +             menu_add_prompt(P_MENU, "Linux Kernel Configuration", NULL);
>
> Could this be something less specific?
> To help the other users of kconfig out is the wild.
>
> For example:
>
>         menu_add_prompt(P_MENU, _("Main menu"), NULL);


OK, but this should be done as a separate work.

I am keeping the current string "Linux Kernel Configuration"



> Note, the added _() to make the text translateable.
>

Hmm, I just wonder how much helpful gettext things are...

Rather, it looks maintenance burden to me.
Once we add messages, it is difficult to change.


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v4 07/31] kconfig: add built-in function support

[Sam Ravnborg]

Hi Masahiro

> >> +     char *(*func)(int argc, char *argv[], int old_argc, char *old_argv[]);
> >> +};
> > If a typedef was provided for the function then ...
> 
> 
> Yes, I can do this,
> but I may rather consider to simplify the code.

Simplify is better.

> > Will the following be equal:
> >
> >         $(foo,abc,$(x),$(y))
> >         $(foo, abc, $(x), $(y))
> >
> > make is rather annoying as space is significant, but there seems no good reason
> > for kconfig to inheritate this.
> > So unless there are good arguments consider alloing the spaces.
> > If the current implmentation already supports optional spaces then I just missed
> > it whie reviewing.
> 
> 
> I have been thinking of trimming the leading whitespaces.
> (https://patchwork.kernel.org/patch/10405549/)
> 
> This is trade-off vs "how to pass spaces as arguments?"

Maybe allow strings to be passed enclosed in ""?
Then it is simple to add whitespace.

But the use of "" should be optional in all other cases.
And the "" should be stripped.

	Sam

Re: [PATCH v4 07/31] kconfig: add built-in function support

[Masahiro Yamada]

Hi Sam,


2018-05-21 15:16 GMT+09:00 Sam Ravnborg <sam@ravnborg.org>:
> Hi Masahiro
>
>> >> +     char *(*func)(int argc, char *argv[], int old_argc, char *old_argv[]);
>> >> +};
>> > If a typedef was provided for the function then ...
>>
>>
>> Yes, I can do this,
>> but I may rather consider to simplify the code.
>
> Simplify is better.
>
>> > Will the following be equal:
>> >
>> >         $(foo,abc,$(x),$(y))
>> >         $(foo, abc, $(x), $(y))
>> >
>> > make is rather annoying as space is significant, but there seems no good reason
>> > for kconfig to inheritate this.
>> > So unless there are good arguments consider alloing the spaces.
>> > If the current implmentation already supports optional spaces then I just missed
>> > it whie reviewing.
>>
>>
>> I have been thinking of trimming the leading whitespaces.
>> (https://patchwork.kernel.org/patch/10405549/)
>>
>> This is trade-off vs "how to pass spaces as arguments?"
>
> Maybe allow strings to be passed enclosed in ""?
> Then it is simple to add whitespace.
>
> But the use of "" should be optional in all other cases.
> And the "" should be stripped.
>

Hmm, your suggestion is more shell-oriented parsing.


In Make, there is no concept of quoting
because it does not touch single-quote, double-quote, whitespaces etc. at all.


$(info "'@@"''  '" ' "' )

will print the message as they are.


This simplifies both the grammar and the parser implementation.

If we expand the quoting by Kconfig,
we need more careful consideration.


[1] In the following, would "hello   world" be expanded by Kconfig or by shell?
$(shell, echo "hello   world")

[2] Is a quoted comma delimiter or not?
$(if,",",$(A))


If remember I first examined shell-oriented expansion,
but I stopped.

Probably, I found it was much more complex,
then I chose Make-like simpler one.



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v4 07/31] kconfig: add built-in function support

[Sam Ravnborg]

Hi Masahiro,

> Hmm, your suggestion is more shell-oriented parsing.
...
> Probably, I found it was much more complex,
> then I chose Make-like simpler one.

Agree that the simple one should be the best choice for now.
We shall not add anything more complex unless it is really needed.

	Sam

Re: [PATCH v4 03/31] kconfig: reference environment variables directly and remove 'option env='

[Ulf Magnusson]

On Mon, May 21, 2018 at 6:43 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> Hi.
>
>
>
> 2018-05-21 0:46 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>
>> s/environments/environment variables/
>
> Will fix.
>
>
>>
>>> +        * They will be written out to include/config/auto.conf.cmd
>>> +        */
>>> +       env_add(name, value);
>>> +
>>> +       return xstrdup(value);
>>> +}
>>> +
>>> +void env_write_dep(FILE *f, const char *autoconfig_name)
>>> +{
>>> +       struct env *e, *tmp;
>>> +
>>> +       list_for_each_entry_safe(e, tmp, &env_list, node) {
>>> +               fprintf(f, "ifneq \"$(%s)\" \"%s\"\n", e->name, e->value);
>>> +               fprintf(f, "%s: FORCE\n", autoconfig_name);
>>> +               fprintf(f, "endif\n");
>>> +               env_del(e);
>>> +       }
>>> +}
>>> +
>>> +static char *eval_clause(const char *in)
>>> +{
>>> +       char *res, *name;
>>> +
>>> +       /*
>>> +        * Returns an empty string because '$()' should be evaluated
>>> +        * to a null string.
>>> +        */
>>
>> Do you know of cases where this is more useful than erroring out?
>>
>> Not saying it doesn't make sense. Just curious.
>
>
> I just followed how Make works.
>
> Anyway, eval_clause() will return null string for null input.
> I will remove that hunk.
>
>
>
>
>>> +       if (!*in)
>>> +               return xstrdup("");
>>> +
>>> +       name = expand_string(in);
>>> +
>>> +       res = env_expand(name);
>>> +       free(name);
>>> +
>>> +       return res;
>>> +}
>>> +
>>> +/*
>>> + * Expand a string that follows '$'
>>> + *
>>> + * For example, if the input string is
>>> + *     ($(FOO)$($(BAR)))$(BAZ)
>>> + * this helper evaluates
>>> + *     $($(FOO)$($(BAR)))
>>> + * and returns the resulted string, then updates 'str' to point to the next
>>
>> s/resulted/resulting/
>>
>> Maybe something like this would make the behavior a bit clearer:
>>
>>   ...and returns a new string containing the expansion, also advancing
>>   'str' to point to the next character after... (note that this function does
>>   a recursive expansion) ...
>
>
> Is this OK?
>
> /*
>  * Expand a string that follows '$'
>  *
>  * For example, if the input string is
>  *     ($(FOO)$($(BAR)))$(BAZ)
>  * this helper evaluates
>  *     $($(FOO)$($(BAR)))
>  * and returns a new string containing the expansion (note that the string is
>  * recursively expanded), also advancing 'str' to point to the next character
>  * after the corresponding closing parenthesis, in this case, *str will be
>  *     $(BAR)
>  */

Looks fine to me.

>
>
>
>>> + * character after the corresponding closing parenthesis, in this case, *str
>>> + * will be
>>> + *     $(BAR)
>>> + */
>>> +char *expand_dollar(const char **str)
>>> +{
>>> +       const char *p = *str;
>>> +       const char *q;
>>> +       char *tmp, *out;
>>> +       int nest = 0;
>>> +
>>> +       /* '$$' represents an escaped '$' */
>>> +       if (*p == '$') {
>>> +               *str = p + 1;
>>> +               return xstrdup("$");
>>> +       }
>>> +
>>> +       /*
>>> +        * Kconfig does not support single-letter variable as in $A
>>> +        * or curly braces as in ${CC}.
>>> +        */
>>> +       if (*p != '(')
>>> +               pperror("syntax error: '$' not followed by '('", p);
>>
>> Could say "not followed by '(' by or '$'".
>
> Will do.
>
>
>>> +
>>> +       p++;
>>> +       q = p;
>>> +       while (*q) {
>>> +               if (*q == '(') {
>>> +                       nest++;
>>> +               } else if (*q == ')') {
>>> +                       if (nest-- == 0)
>>> +                               break;
>>> +               }
>>> +               q++;
>>> +       }
>>> +
>>> +       if (!*q)
>>> +               pperror("unterminated reference to '%s': missing ')'", p);
>>> +
>>> +       tmp = xmalloc(q - p + 1);
>>> +       memcpy(tmp, p, q - p);
>>> +       tmp[q - p] = 0;
>>> +       *str = q + 1;
>>> +       out = eval_clause(tmp);
>>> +       free(tmp);
>>> +
>>> +       return out;
>>
>> This might be a bit clearer, since the 'str' update and the expansion
>> are independent:
>
> Indeed, will do.
>
>>
>>   /* Advance 'str' to after the expanded initial portion of the string */
>>   *str = q + 1;
>>
>>   /* Save the "FOO" part of "(FOO)" into 'tmp' and expand it recursively */
>>   tmp = xmalloc(q - p + 1);
>>   memcpy(tmp, p, q - p);
>>   tmp[q - p] = '\0';
>>   out = eval_clause(tmp);
>>   free(tmp);
>>
>>   return out;
>>
>> ...or switched around, but thought putting the 'str' bit first might
>> emphasize that it isn't modified.
>
>
> I prefer advancing the pointer at last.

Yeah, it's a bit less weird in a way...

>
>
>
>
>>> +}
>>> +
>>> +/*
>>> + * Expand variables in the given string.  Undefined variables
>>> + * expand to an empty string.
>>
>> I wonder what the tradeoffs are vs. erroring out here (or leaving
>> undefined variables as-is).
>
>
> I want to stick to the Make-like behavior here and exploit it in some cases.
> We can set some variables in some arch/*/Kconfig,
> but unset in the others.
>
>
>
>
> In some arch/*/Kconfig:
>
> min-gcc-version := 40900
>
>
>
>
> In init/Kconfig:
>
> # GCC 4.5 is required to build Linux Kernel.
> # Some architectures may override it (from arch/*/Kconfig) for their
> requirement.
> min-gcc-version := $(if,$(min-gcc-version),$(min-gcc-version),40500)
>
> ... check the gcc version, then show error
> if it is older than $(min-gcc-version).

OK, makes sense. Fine by me.

>>> + * The returned string must be freed when done.
>>> + */
>>> +char *expand_string(const char *in)
>>> +{
>>> +       const char *prev_in, *p;
>>> +       char *new, *out;
>>> +       size_t outlen;
>>> +
>>> +       out = xmalloc(1);
>>> +       *out = 0;
>>> +
>>> +       while ((p = strchr(in, '$'))) {
>>> +               prev_in = in;
>>> +               in = p + 1;
>>> +               new = expand_dollar(&in);
>>> +               outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
>>> +               out = xrealloc(out, outlen);
>>> +               strncat(out, prev_in, p - prev_in);
>>> +               strcat(out, new);
>>> +               free(new);
>>
>> Some code duplication with expand_one_token() here. Do you think
>> something could be factored out/reorganized?
>
>
> Yes.  For example, the following would work.
> If you prefer that, I can update it in v5.
>
>
>
>
> static char *__expand_string(const char **str, bool (*is_end)(const char *))
> {
>         const char *in, *prev_in, *p;
>         char *new, *out;
>         size_t outlen;
>
>         out = xmalloc(1);
>         *out = 0;
>
>         p = in = *str;
>
>         while (1) {
>                 if (*p == '$') {
>                         prev_in = in;
>                         in = p + 1;
>                         new = expand_dollar(&in);
>                         outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
>                         out = xrealloc(out, outlen);
>                         strncat(out, prev_in, p - prev_in);
>                         strcat(out, new);
>                         free(new);
>                         p = in;
>                         continue;
>                 }
>
>                 if (is_end(p))
>                         break;
>
>                 p++;
>         }
>
>         outlen = strlen(out) + (p - in) + 1;
>         out = xrealloc(out, outlen);
>         strncat(out, in, p - in);
>
>         *str = p;
>
>         return out;
> }
>
> static bool is_end_of_str(const char *s)
> {
>         return !*s;
> }
>
> char *expand_string(const char *in)
> {
>         return __expand_string(&in, is_end_of_str);
> }
>
> static bool is_end_of_token(const char *s)
> {
>         return !(isalnum(*s) || *s == '_' || *s == '-' || *s == '.' ||
> *s == '/');
> }
>
> char *expand_one_token(const char **str)
> {
>         return __expand_string(str, is_end_of_token);
> }

Yep, something like that would be nicer I think.

This variant might work too (untested):

  dollar_i = p;
  p++;
  expansion = expand_dollar(&p);

  out = xrealloc(out, strlen(out) + (dollar_i - in)
                      + strlen(expansion) + 1);
  strncat(out, in, dollar_i - in);
  strcat(out, expansion);
  free(expansion);

  in = p;

  continue;

The p++ would disappear if expand_dollar() took a pointer to the '$'.

Having some string buffer helpers might be nice, but definitely out of scope.

>
>
>
>
>
>> I wonder if it might be cleaner to have expand_dollar() take a pointer
>> to the '$'. Might even out in terms of the +1's you'd have to add, as
>> all callers manually bump the pointer before the call now. I haven't
>> tried implementing it though, so hard to say.
>
>
> If I do this, I'd want to add a sanity check to expand_dollar().
> I'd rather like to avoid unneeded code.
>
>
> static char *expand_dollar(...)
> {
>         assert(*p == '$');
>         p++;
>
>         /* '$$' represents an escaped '$' */
>         if (*p == '$') {
>                *str = p + 1;
>                return xstrdup("$");
>         }
>
>          ...
>
>
> }

Felt a bit more direct to have it point to the start of the thing
being expanded, and the assert() doesn't seem horrible there to me,
but your call.

>
>
>>
>> Some short inline comments similar to above would help too I think.
>>
>
>
>
> --
> Best Regards
> Masahiro Yamada

Cheers,
Ulf

Re: [PATCH v4 03/31] kconfig: reference environment variables directly and remove 'option env='

[Ulf Magnusson]

On Mon, May 21, 2018 at 1:06 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
> On Mon, May 21, 2018 at 6:43 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> Hi.
>>
>>
>>
>> 2018-05-21 0:46 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>>
>>> s/environments/environment variables/
>>
>> Will fix.
>>
>>
>>>
>>>> +        * They will be written out to include/config/auto.conf.cmd
>>>> +        */
>>>> +       env_add(name, value);
>>>> +
>>>> +       return xstrdup(value);
>>>> +}
>>>> +
>>>> +void env_write_dep(FILE *f, const char *autoconfig_name)
>>>> +{
>>>> +       struct env *e, *tmp;
>>>> +
>>>> +       list_for_each_entry_safe(e, tmp, &env_list, node) {
>>>> +               fprintf(f, "ifneq \"$(%s)\" \"%s\"\n", e->name, e->value);
>>>> +               fprintf(f, "%s: FORCE\n", autoconfig_name);
>>>> +               fprintf(f, "endif\n");
>>>> +               env_del(e);
>>>> +       }
>>>> +}
>>>> +
>>>> +static char *eval_clause(const char *in)
>>>> +{
>>>> +       char *res, *name;
>>>> +
>>>> +       /*
>>>> +        * Returns an empty string because '$()' should be evaluated
>>>> +        * to a null string.
>>>> +        */
>>>
>>> Do you know of cases where this is more useful than erroring out?
>>>
>>> Not saying it doesn't make sense. Just curious.
>>
>>
>> I just followed how Make works.
>>
>> Anyway, eval_clause() will return null string for null input.
>> I will remove that hunk.
>>
>>
>>
>>
>>>> +       if (!*in)
>>>> +               return xstrdup("");
>>>> +
>>>> +       name = expand_string(in);
>>>> +
>>>> +       res = env_expand(name);
>>>> +       free(name);
>>>> +
>>>> +       return res;
>>>> +}
>>>> +
>>>> +/*
>>>> + * Expand a string that follows '$'
>>>> + *
>>>> + * For example, if the input string is
>>>> + *     ($(FOO)$($(BAR)))$(BAZ)
>>>> + * this helper evaluates
>>>> + *     $($(FOO)$($(BAR)))
>>>> + * and returns the resulted string, then updates 'str' to point to the next
>>>
>>> s/resulted/resulting/
>>>
>>> Maybe something like this would make the behavior a bit clearer:
>>>
>>>   ...and returns a new string containing the expansion, also advancing
>>>   'str' to point to the next character after... (note that this function does
>>>   a recursive expansion) ...
>>
>>
>> Is this OK?
>>
>> /*
>>  * Expand a string that follows '$'
>>  *
>>  * For example, if the input string is
>>  *     ($(FOO)$($(BAR)))$(BAZ)
>>  * this helper evaluates
>>  *     $($(FOO)$($(BAR)))
>>  * and returns a new string containing the expansion (note that the string is
>>  * recursively expanded), also advancing 'str' to point to the next character
>>  * after the corresponding closing parenthesis, in this case, *str will be
>>  *     $(BAR)
>>  */
>
> Looks fine to me.
>
>>
>>
>>
>>>> + * character after the corresponding closing parenthesis, in this case, *str
>>>> + * will be
>>>> + *     $(BAR)
>>>> + */
>>>> +char *expand_dollar(const char **str)
>>>> +{
>>>> +       const char *p = *str;
>>>> +       const char *q;
>>>> +       char *tmp, *out;
>>>> +       int nest = 0;
>>>> +
>>>> +       /* '$$' represents an escaped '$' */
>>>> +       if (*p == '$') {
>>>> +               *str = p + 1;
>>>> +               return xstrdup("$");
>>>> +       }
>>>> +
>>>> +       /*
>>>> +        * Kconfig does not support single-letter variable as in $A
>>>> +        * or curly braces as in ${CC}.
>>>> +        */
>>>> +       if (*p != '(')
>>>> +               pperror("syntax error: '$' not followed by '('", p);
>>>
>>> Could say "not followed by '(' by or '$'".
>>
>> Will do.
>>
>>
>>>> +
>>>> +       p++;
>>>> +       q = p;
>>>> +       while (*q) {
>>>> +               if (*q == '(') {
>>>> +                       nest++;
>>>> +               } else if (*q == ')') {
>>>> +                       if (nest-- == 0)
>>>> +                               break;
>>>> +               }
>>>> +               q++;
>>>> +       }
>>>> +
>>>> +       if (!*q)
>>>> +               pperror("unterminated reference to '%s': missing ')'", p);
>>>> +
>>>> +       tmp = xmalloc(q - p + 1);
>>>> +       memcpy(tmp, p, q - p);
>>>> +       tmp[q - p] = 0;
>>>> +       *str = q + 1;
>>>> +       out = eval_clause(tmp);
>>>> +       free(tmp);
>>>> +
>>>> +       return out;
>>>
>>> This might be a bit clearer, since the 'str' update and the expansion
>>> are independent:
>>
>> Indeed, will do.
>>
>>>
>>>   /* Advance 'str' to after the expanded initial portion of the string */
>>>   *str = q + 1;
>>>
>>>   /* Save the "FOO" part of "(FOO)" into 'tmp' and expand it recursively */
>>>   tmp = xmalloc(q - p + 1);
>>>   memcpy(tmp, p, q - p);
>>>   tmp[q - p] = '\0';
>>>   out = eval_clause(tmp);
>>>   free(tmp);
>>>
>>>   return out;
>>>
>>> ...or switched around, but thought putting the 'str' bit first might
>>> emphasize that it isn't modified.
>>
>>
>> I prefer advancing the pointer at last.
>
> Yeah, it's a bit less weird in a way...
>
>>
>>
>>
>>
>>>> +}
>>>> +
>>>> +/*
>>>> + * Expand variables in the given string.  Undefined variables
>>>> + * expand to an empty string.
>>>
>>> I wonder what the tradeoffs are vs. erroring out here (or leaving
>>> undefined variables as-is).
>>
>>
>> I want to stick to the Make-like behavior here and exploit it in some cases.
>> We can set some variables in some arch/*/Kconfig,
>> but unset in the others.
>>
>>
>>
>>
>> In some arch/*/Kconfig:
>>
>> min-gcc-version := 40900
>>
>>
>>
>>
>> In init/Kconfig:
>>
>> # GCC 4.5 is required to build Linux Kernel.
>> # Some architectures may override it (from arch/*/Kconfig) for their
>> requirement.
>> min-gcc-version := $(if,$(min-gcc-version),$(min-gcc-version),40500)
>>
>> ... check the gcc version, then show error
>> if it is older than $(min-gcc-version).
>
> OK, makes sense. Fine by me.
>
>>>> + * The returned string must be freed when done.
>>>> + */
>>>> +char *expand_string(const char *in)
>>>> +{
>>>> +       const char *prev_in, *p;
>>>> +       char *new, *out;
>>>> +       size_t outlen;
>>>> +
>>>> +       out = xmalloc(1);
>>>> +       *out = 0;
>>>> +
>>>> +       while ((p = strchr(in, '$'))) {
>>>> +               prev_in = in;
>>>> +               in = p + 1;
>>>> +               new = expand_dollar(&in);
>>>> +               outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
>>>> +               out = xrealloc(out, outlen);
>>>> +               strncat(out, prev_in, p - prev_in);
>>>> +               strcat(out, new);
>>>> +               free(new);
>>>
>>> Some code duplication with expand_one_token() here. Do you think
>>> something could be factored out/reorganized?
>>
>>
>> Yes.  For example, the following would work.
>> If you prefer that, I can update it in v5.
>>
>>
>>
>>
>> static char *__expand_string(const char **str, bool (*is_end)(const char *))
>> {
>>         const char *in, *prev_in, *p;
>>         char *new, *out;
>>         size_t outlen;
>>
>>         out = xmalloc(1);
>>         *out = 0;
>>
>>         p = in = *str;
>>
>>         while (1) {
>>                 if (*p == '$') {
>>                         prev_in = in;
>>                         in = p + 1;
>>                         new = expand_dollar(&in);
>>                         outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
>>                         out = xrealloc(out, outlen);
>>                         strncat(out, prev_in, p - prev_in);
>>                         strcat(out, new);
>>                         free(new);
>>                         p = in;
>>                         continue;
>>                 }
>>
>>                 if (is_end(p))
>>                         break;
>>
>>                 p++;
>>         }
>>
>>         outlen = strlen(out) + (p - in) + 1;
>>         out = xrealloc(out, outlen);
>>         strncat(out, in, p - in);
>>
>>         *str = p;
>>
>>         return out;
>> }
>>
>> static bool is_end_of_str(const char *s)
>> {
>>         return !*s;
>> }
>>
>> char *expand_string(const char *in)
>> {
>>         return __expand_string(&in, is_end_of_str);
>> }
>>
>> static bool is_end_of_token(const char *s)
>> {
>>         return !(isalnum(*s) || *s == '_' || *s == '-' || *s == '.' ||
>> *s == '/');
>> }
>>
>> char *expand_one_token(const char **str)
>> {
>>         return __expand_string(str, is_end_of_token);
>> }
>
> Yep, something like that would be nicer I think.
>
> This variant might work too (untested):
>
>   dollar_i = p;

dollar_p makes more sense.

Cheers,
Ulf

Re: [PATCH v4 07/31] kconfig: add built-in function support

[Ulf Magnusson]

On Sun, May 20, 2018 at 4:50 PM, Sam Ravnborg <sam@ravnborg.org> wrote:
> Will the following be equal:
>
>         $(foo,abc,$(x),$(y))
>         $(foo, abc, $(x), $(y))
>
> make is rather annoying as space is significant, but there seems no good reason
> for kconfig to inheritate this.
> So unless there are good arguments consider alloing the spaces.
> If the current implmentation already supports optional spaces then I just missed
> it whie reviewing.
>
>         Sam

+1 from me.

I also find the rules for whitespace in Make confusing, and always
have to look them up when doing trickier stuff. Maybe they're the
result of people not considering whitespace initially, and stuff
getting tacked on later. GNU Make adds some alternate syntaxes with
quotes.

I was going to mention shell, but it looks like you already did. :)

If we go with Make-like syntax, maybe we could at least have a variant
with fewer whitespace gotchas.

Cheers,
Ulf

Re: [PATCH v4 07/31] kconfig: add built-in function support

[Ulf Magnusson]

On Mon, May 21, 2018 at 4:23 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
> On Sun, May 20, 2018 at 4:50 PM, Sam Ravnborg <sam@ravnborg.org> wrote:
>> Will the following be equal:
>>
>>         $(foo,abc,$(x),$(y))
>>         $(foo, abc, $(x), $(y))
>>
>> make is rather annoying as space is significant, but there seems no good reason
>> for kconfig to inheritate this.
>> So unless there are good arguments consider alloing the spaces.
>> If the current implmentation already supports optional spaces then I just missed
>> it whie reviewing.
>>
>>         Sam
>
> +1 from me.
>
> I also find the rules for whitespace in Make confusing, and always
> have to look them up when doing trickier stuff. Maybe they're the
> result of people not considering whitespace initially, and stuff
> getting tacked on later. GNU Make adds some alternate syntaxes with
> quotes.
>
> I was going to mention shell, but it looks like you already did. :)
>
> If we go with Make-like syntax, maybe we could at least have a variant
> with fewer whitespace gotchas.
>
> Cheers,
> Ulf

Maybe it'd be a pain to implement, but something like $(foo $(x) "two
words" "interpolated $(stuff)") seems pretty nice, with three
arguments there.

For variables too:

  x = foo
  y = "two words"

Or have mandatory quotes, but yeah, bit spammy there maybe.

Cheers,
Ulf

Re: [PATCH v4 07/31] kconfig: add built-in function support

[Ulf Magnusson]

On Mon, May 21, 2018 at 4:32 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
> On Mon, May 21, 2018 at 4:23 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
>> On Sun, May 20, 2018 at 4:50 PM, Sam Ravnborg <sam@ravnborg.org> wrote:
>>> Will the following be equal:
>>>
>>>         $(foo,abc,$(x),$(y))
>>>         $(foo, abc, $(x), $(y))
>>>
>>> make is rather annoying as space is significant, but there seems no good reason
>>> for kconfig to inheritate this.
>>> So unless there are good arguments consider alloing the spaces.
>>> If the current implmentation already supports optional spaces then I just missed
>>> it whie reviewing.
>>>
>>>         Sam
>>
>> +1 from me.
>>
>> I also find the rules for whitespace in Make confusing, and always
>> have to look them up when doing trickier stuff. Maybe they're the
>> result of people not considering whitespace initially, and stuff
>> getting tacked on later. GNU Make adds some alternate syntaxes with
>> quotes.
>>
>> I was going to mention shell, but it looks like you already did. :)
>>
>> If we go with Make-like syntax, maybe we could at least have a variant
>> with fewer whitespace gotchas.
>>
>> Cheers,
>> Ulf
>
> Maybe it'd be a pain to implement, but something like $(foo $(x) "two
> words" "interpolated $(stuff)") seems pretty nice, with three
> arguments there.

Guess that might interact poorly with $(shell foo "bar baz") though.
Kinda nice to have a syntax that doesn't overlap with shell when
building shell commands.

Still wondering if you could get rid of some of the Make gotchas
without losing other stuff...

Cheers,
Ulf

Re: [PATCH v4 07/31] kconfig: add built-in function support

[Masahiro Yamada]

2018-05-22 0:10 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> On Mon, May 21, 2018 at 4:32 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
>> On Mon, May 21, 2018 at 4:23 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
>>> On Sun, May 20, 2018 at 4:50 PM, Sam Ravnborg <sam@ravnborg.org> wrote:
>>>> Will the following be equal:
>>>>
>>>>         $(foo,abc,$(x),$(y))
>>>>         $(foo, abc, $(x), $(y))
>>>>
>>>> make is rather annoying as space is significant, but there seems no good reason
>>>> for kconfig to inheritate this.
>>>> So unless there are good arguments consider alloing the spaces.
>>>> If the current implmentation already supports optional spaces then I just missed
>>>> it whie reviewing.
>>>>
>>>>         Sam
>>>
>>> +1 from me.
>>>
>>> I also find the rules for whitespace in Make confusing, and always
>>> have to look them up when doing trickier stuff. Maybe they're the
>>> result of people not considering whitespace initially, and stuff
>>> getting tacked on later. GNU Make adds some alternate syntaxes with
>>> quotes.
>>>
>>> I was going to mention shell, but it looks like you already did. :)
>>>
>>> If we go with Make-like syntax, maybe we could at least have a variant
>>> with fewer whitespace gotchas.
>>>
>>> Cheers,
>>> Ulf
>>
>> Maybe it'd be a pain to implement, but something like $(foo $(x) "two
>> words" "interpolated $(stuff)") seems pretty nice, with three
>> arguments there.
>
> Guess that might interact poorly with $(shell foo "bar baz") though.
> Kinda nice to have a syntax that doesn't overlap with shell when
> building shell commands.


Right.  I can easily imagine
that would end up with more gotchas due to quoting and escaping.



> Still wondering if you could get rid of some of the Make gotchas
> without losing other stuff...
>
> Cheers,
> Ulf



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v4 07/31] kconfig: add built-in function support

[Ulf Magnusson]

On Tue, May 22, 2018 at 5:11 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2018-05-22 0:10 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>> On Mon, May 21, 2018 at 4:32 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
>>> On Mon, May 21, 2018 at 4:23 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
>>>> On Sun, May 20, 2018 at 4:50 PM, Sam Ravnborg <sam@ravnborg.org> wrote:
>>>>> Will the following be equal:
>>>>>
>>>>>         $(foo,abc,$(x),$(y))
>>>>>         $(foo, abc, $(x), $(y))
>>>>>
>>>>> make is rather annoying as space is significant, but there seems no good reason
>>>>> for kconfig to inheritate this.
>>>>> So unless there are good arguments consider alloing the spaces.
>>>>> If the current implmentation already supports optional spaces then I just missed
>>>>> it whie reviewing.
>>>>>
>>>>>         Sam
>>>>
>>>> +1 from me.
>>>>
>>>> I also find the rules for whitespace in Make confusing, and always
>>>> have to look them up when doing trickier stuff. Maybe they're the
>>>> result of people not considering whitespace initially, and stuff
>>>> getting tacked on later. GNU Make adds some alternate syntaxes with
>>>> quotes.
>>>>
>>>> I was going to mention shell, but it looks like you already did. :)
>>>>
>>>> If we go with Make-like syntax, maybe we could at least have a variant
>>>> with fewer whitespace gotchas.
>>>>
>>>> Cheers,
>>>> Ulf
>>>
>>> Maybe it'd be a pain to implement, but something like $(foo $(x) "two
>>> words" "interpolated $(stuff)") seems pretty nice, with three
>>> arguments there.
>>
>> Guess that might interact poorly with $(shell foo "bar baz") though.
>> Kinda nice to have a syntax that doesn't overlap with shell when
>> building shell commands.
>
>
> Right.  I can easily imagine
> that would end up with more gotchas due to quoting and escaping.

Yeah, you're right. It's probably trying to fix something that isn't
broken. Make's syntax really isn't bad there, just slightly
non-obvious at first...

Think it's fine now. Better to commit to the syntax than trying to be
"helpful" by adding a bunch of random exceptions too. That probably
gives a bigger mess in the end...

Cheers,
Ulf

Re: [PATCH v4 07/31] kconfig: add built-in function support

[Ulf Magnusson]

On Tue, May 22, 2018 at 6:50 AM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
> On Tue, May 22, 2018 at 5:11 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> 2018-05-22 0:10 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>>> On Mon, May 21, 2018 at 4:32 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
>>>> On Mon, May 21, 2018 at 4:23 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
>>>>> On Sun, May 20, 2018 at 4:50 PM, Sam Ravnborg <sam@ravnborg.org> wrote:
>>>>>> Will the following be equal:
>>>>>>
>>>>>>         $(foo,abc,$(x),$(y))
>>>>>>         $(foo, abc, $(x), $(y))
>>>>>>
>>>>>> make is rather annoying as space is significant, but there seems no good reason
>>>>>> for kconfig to inheritate this.
>>>>>> So unless there are good arguments consider alloing the spaces.
>>>>>> If the current implmentation already supports optional spaces then I just missed
>>>>>> it whie reviewing.
>>>>>>
>>>>>>         Sam
>>>>>
>>>>> +1 from me.
>>>>>
>>>>> I also find the rules for whitespace in Make confusing, and always
>>>>> have to look them up when doing trickier stuff. Maybe they're the
>>>>> result of people not considering whitespace initially, and stuff
>>>>> getting tacked on later. GNU Make adds some alternate syntaxes with
>>>>> quotes.
>>>>>
>>>>> I was going to mention shell, but it looks like you already did. :)
>>>>>
>>>>> If we go with Make-like syntax, maybe we could at least have a variant
>>>>> with fewer whitespace gotchas.
>>>>>
>>>>> Cheers,
>>>>> Ulf
>>>>
>>>> Maybe it'd be a pain to implement, but something like $(foo $(x) "two
>>>> words" "interpolated $(stuff)") seems pretty nice, with three
>>>> arguments there.
>>>
>>> Guess that might interact poorly with $(shell foo "bar baz") though.
>>> Kinda nice to have a syntax that doesn't overlap with shell when
>>> building shell commands.
>>
>>
>> Right.  I can easily imagine
>> that would end up with more gotchas due to quoting and escaping.
>
> Yeah, you're right. It's probably trying to fix something that isn't
> broken. Make's syntax really isn't bad there, just slightly
> non-obvious at first...
>
> Think it's fine now. Better to commit to the syntax than trying to be
> "helpful" by adding a bunch of random exceptions too. That probably
> gives a bigger mess in the end...
>
> Cheers,
> Ulf

I'm fine with the comma-after-function-name change though. That just
makes it more consistent.

Cheers,
Ulf

Re: [PATCH v4 00/31] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

Hi reviewers,


2018-05-17 23:22 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:
> 2018-05-17 16:51 GMT+09:00 Nicholas Piggin <npiggin@gmail.com>:
>> On Thu, 17 May 2018 15:16:39 +0900
>> Masahiro Yamada <yamada.masahiro@socionext.com> wrote:
>>
>>> [Introduction]
>>>
>>> The motivation of this work is to move the compiler option tests to
>>> Kconfig from Makefile.  A number of kernel features require the
>>> compiler support.  Enabling such features blindly in Kconfig ends up
>>> with a lot of nasty build-time testing in Makefiles.  If a chosen
>>> feature turns out unsupported by the compiler, what the build system
>>> can do is either to disable it (silently!) or to forcibly break the
>>> build, despite Kconfig has let the user to enable it.  By moving the
>>> compiler capability tests to Kconfig, Kconfig entries will be visible
>>> only when they are available.
>>>
>>> [Major Changes in V4]
>>
>> Do you have a git tree for v4? I can test it with the powerpc patches.
>>
>> The new scripting capability in kconfig has allowed us to already
>> improve the config process on powerpc:
>>
>> https://marc.info/?l=linuxppc-embedded&m=152648110727868&w=2
>>
>> I'm sure there's more clever things we can do with it but I haven't
>> had time to think about it yet. One thing that comes to mind is that
>> It might be nice to show the option as disabled, then the user could
>> upgrade their compiler to get the options they want.
>>
>> Anyway v3 worked fine for me, the documentation is really nice, I
>> could implement the above patch without any problem despite being a
>> kbuild dummy. Thanks for the series, ack from me.
>
>
> For easier review and test,
> I pushed v4 to the following branch:
>
>
> git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
> kconfig-shell-v4
>



Planned change for v5
---------------------

I am thinking of more simplification for v5.
This is my thought.


V4 supported the _lazy_ expansion and 'if' function.


If you have ever looked into GNU Make source code,
you may know some functions lazily expand arguments.

https://git.savannah.gnu.org/cgit/make.git/tree/src/function.c#n2339


"foreach", "if", "or", "and" are specified with "EXP? = 0".

Those 4 functions expand arguments as-needed.

For example,

$(if $(cond),$(yes-case),$(no-case))


The 'if' function receive arguments before expansion,
then expands $(cond) first.

$(yes-case) is expanded only when $(cond)
turns out to be a non-empty string.
In this case, $(no-case) is not evaluated at all.


$(error msg), if it is evaluated,
immediately terminates Makefile parsing



That's why $(error ) in Makefile are generally used
in the either of the following forms:

ifeq ...
  $(error blah blah)
endif

  OR

$(if $(cond),$(error blah blah))





This does not work in user-defined functions.

$(call my-func,$(cond),$(error blah blah))


This always terminates parsing regardless of $(cond).

Because the 'call' function is specified with "EXP? = 1",
arguments are expanded (i.e. evaluated)
before they are passed to the 'call' function.


Let's say, you implemented your helper macros based on the
'if' built-in function, like follows:

# return $(2) or $(3), depending on the gcc version.
gcc-ifversion = $(if $(shell test $(gcc-version) -ge $(1) && echo y),$(2),$(3))


In this case, both $(2) and $(3) are evaluated
before the condition part is checked.
So, we end up with unneeded calculation
where we know $(2) and $(3) are exclusively used.


We can exploit the laziness
only when we use the 'if' function in raw form.
This use cases are limited.

We use helper macros anyway,
and the advantage of the lazy expansion disappear
in user-defined functions.


After consideration, I am reluctant to implement the lazy expansion
(at least until we find it is necessary).

To keep the source code simple,
I do not want to implement what we may or may not need.
At this moment, I am not so sure whether it is useful.


So, here is a planned change in v5:

Kconfig expands arguments before passing them to a function.

This means,
$(error blah blah) would never be useful.


I will remove "error" and "warning" built-in functions.

Instead, I will add "error-on" and "warning-on" built-in functions.

 -  $(error-on,<cond>,<message>)

   If <cond> contains any character,
   it terminates file parsing, showing <message> to stderr.

 -  $(warning-on,<cond>,<message>)

   If <cond> contains any character,
   show <message> to stderr.


I will remove 'if' function too.
If we give up its laziness, we can implement it as
a user-defined function.




-- 
Best Regards
Masahiro Yamada

Re: [PATCH v4 03/31] kconfig: reference environment variables directly and remove 'option env='

[Masahiro Yamada]

2018-05-21 20:06 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:

>>
>> static char *__expand_string(const char **str, bool (*is_end)(const char *))
>> {
>>         const char *in, *prev_in, *p;
>>         char *new, *out;
>>         size_t outlen;
>>
>>         out = xmalloc(1);
>>         *out = 0;
>>
>>         p = in = *str;
>>
>>         while (1) {
>>                 if (*p == '$') {
>>                         prev_in = in;
>>                         in = p + 1;
>>                         new = expand_dollar(&in);
>>                         outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
>>                         out = xrealloc(out, outlen);
>>                         strncat(out, prev_in, p - prev_in);
>>                         strcat(out, new);
>>                         free(new);
>>                         p = in;
>>                         continue;
>>                 }
>>
>>                 if (is_end(p))
>>                         break;
>>
>>                 p++;
>>         }
>>
>>         outlen = strlen(out) + (p - in) + 1;
>>         out = xrealloc(out, outlen);
>>         strncat(out, in, p - in);
>>
>>         *str = p;
>>
>>         return out;
>> }
>>
>> static bool is_end_of_str(const char *s)
>> {
>>         return !*s;
>> }
>>
>> char *expand_string(const char *in)
>> {
>>         return __expand_string(&in, is_end_of_str);
>> }
>>
>> static bool is_end_of_token(const char *s)
>> {
>>         return !(isalnum(*s) || *s == '_' || *s == '-' || *s == '.' ||
>> *s == '/');
>> }
>>
>> char *expand_one_token(const char **str)
>> {
>>         return __expand_string(str, is_end_of_token);
>> }
>
> Yep, something like that would be nicer I think.
>
> This variant might work too (untested):
>
>   dollar_i = p;
>   p++;
>   expansion = expand_dollar(&p);
>
>   out = xrealloc(out, strlen(out) + (dollar_i - in)
>                       + strlen(expansion) + 1);
>   strncat(out, in, dollar_i - in);
>   strcat(out, expansion);
>   free(expansion);
>
>   in = p;
>
>   continue;
>
> The p++ would disappear if expand_dollar() took a pointer to the '$'.
>


I took the variable name "expansion" because
it is more descriptive than "new".


I rewrote like follows:


static char *__expand_string(const char **str, bool (*is_end)(const char *))
{
        const char *in, *p;
        char *expansion, *out;
        size_t in_len, out_len;

        out = xmalloc(1);
        *out = 0;
        out_len = 1;

        p = in = *str;

        while (1) {
                if (*p == '$') {
                        in_len = p - in;
                        p++;
                        expansion = expand_dollar(&p);
                        out_len += in_len + strlen(expansion);
                        out = xrealloc(out, out_len);
                        strncat(out, in, in_len);
                        strcat(out, expansion);
                        free(expansion);
                        in = p;
                        continue;
                }

                if (is_end(p))
                        break;

                p++;
        }

        in_len = p - in;
        out_len += in_len;
        out = xrealloc(out, out_len);
        strncat(out, in, in_len);

        /* Advance 'str' to the end character */
        *str = p;

        return out;
}



I used "out_len" to remember the length of "out"
instead of calculating strlen(out) every time.

I do not need dollar_p.




-- 
Best Regards
Masahiro Yamada

Re: [PATCH v4 19/31] Documentation: kconfig: document a new Kconfig macro language

[Randy Dunlap]

On 05/16/2018 11:16 PM, Masahiro Yamada wrote:
> Add a document for the macro language introduced to Kconfig.
> 
> The motivation of this work is to move the compiler option tests to
> Kconfig from Makefile.  A number of kernel features require the
> compiler support.  Enabling such features blindly in Kconfig ends up
> with a lot of nasty build-time testing in Makefiles.  If a chosen
> feature turns out unsupported by the compiler, what the build system
> can do is either to disable it (silently!) or to forcibly break the
> build, despite Kconfig has let the user to enable it.  By moving the
> compiler capability tests to Kconfig, features unsupported by the
> compiler will be hidden automatically.
> 
> This change was strongly prompted by Linus Torvalds.  You can find
> his suggestions [1] [2] in ML.  The original idea was to add a new
> attribute with 'option shell=...', but I found more generalized text
> expansion would make Kconfig more powerful and lovely.  The basic
> ideas are from Make, but there are some differences.
> 
> [1]: https://lkml.org/lkml/2016/12/9/577
> [2]: https://lkml.org/lkml/2018/2/7/527
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
> 
> Changes in v4:
>  - Update according to the syntax change
> 
> Changes in v3:
>  - Newly added
> 
> Changes in v2: None
> 
>  Documentation/kbuild/kconfig-macro-language.txt | 252 ++++++++++++++++++++++++
>  MAINTAINERS                                     |   2 +-
>  2 files changed, 253 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/kbuild/kconfig-macro-language.txt
> 
> diff --git a/Documentation/kbuild/kconfig-macro-language.txt b/Documentation/kbuild/kconfig-macro-language.txt
> new file mode 100644
> index 0000000..a8dc792
> --- /dev/null
> +++ b/Documentation/kbuild/kconfig-macro-language.txt
> @@ -0,0 +1,252 @@
> +Concept
> +-------
> +
> +The basic idea was inspired by Make. When we look at Make, we notice sort of
> +two languages in one. One language describes dependency graphs consisting of
> +targets and prerequisites. The other is a macro language for performing textual
> +substitution.
> +
> +There is clear distinction between the two language stages. For example, you
> +can write a makefile like follows:
> +
> +    APP := foo
> +    SRC := foo.c
> +    CC := gcc
> +
> +    $(APP): $(SRC)
> +            $(CC) -o $(APP) $(SRC)
> +
> +The macro language replaces the variable references with their expanded form,
> +and handles as if the source file were input like follows:
> +
> +    foo: foo.c
> +            gcc -o foo foo.c
> +
> +Then, Make analyzes the dependency graph and determines the targets to be
> +updated.
> +
> +The idea is quite similar in Kconfig - it is possible to describe a Kconfig
> +file like this:
> +
> +    CC := gcc
> +
> +    config CC_HAS_FOO
> +            def_bool $(shell, $(srctree)/scripts/gcc-check-foo.sh $(CC))
> +
> +The macro language in Kconfig processes the source file into the following
> +intermediate:
> +
> +    config CC_HAS_FOO
> +            def_bool y
> +
> +Then, Kconfig moves onto the evaluation stage to resolve inter-symbol
> +dependency as explained in kconfig-language.txt.
> +
> +
> +Variables
> +---------
> +
> +Like in Make, a variable in Kconfig works as a macro variable.  A macro
> +variable is expanded "in place" to yield a text string that may then be
> +expanded further. To get the value of a variable, enclose the variable name in
> +$( ). The parentheses are required even for single-letter variable names; $X is
> +a syntax error. The curly brace form as in ${CC} is not supported either.
> +
> +There are two types of variables: simply expanded variables and recursively
> +expanded variables.
> +
> +A simply expanded variable is defined using the := assignment operator. Its
> +righthand side is expanded immediately upon reading the line from the Kconfig
> +file.
> +
> +A recursively expanded variable is defined using the = assignment operator.
> +Its righthand side is simply stored as the value of the variable without
> +expanding it in any way. Instead, the expansion is performed when the variable
> +is used.
> +
> +There is another type of assignment operator; += is used to append text to a
> +variable. The righthand side of += is expanded immediately if the lefthand
> +side was originally defined as a simple variable. Otherwise, its evaluation is
> +deferred.
> +
> +The variable reference can take parameters, in the following form:
> +
> +  $(name,arg1,arg2,arg3)
> +
> +You can consider the parameterized reference as a function. (more precisely,
> +"user-defined function" in the contrast to "built-in function" listed below).

                           in contrast to

> +
> +Useful functions must be expanded when they are used since the same function is
> +expanded differently if different parameters are passed. Hence, a user-defined
> +function is defined using the = assignment operator. The parameters are
> +referenced within the body definition with $(1), $(2), etc.
> +
> +In fact, recursively expanded variables and user-defined functions are the same
> +internally. (In other words, "variable" is "function with zero argument".)
> +When we say "variable" in a broad sense, it includes "user-defined function".
> +
> +
> +Built-in functions
> +------------------
> +
> +Like Make, Kconfig provides several built-in functions. Every function takes a
> +particular number of arguments.
> +
> +In Make, every built-in function takes at least one argument. Kconfig allows
> +zero argument for built-in functions, such as $(fileno), $(lineno). You could
> +consider those as "built-in variable", but it is just a matter of how we call
> +it after all. Let's say "built-in function" here to refer to natively supported
> +functionality.
> +
> +Kconfig currently supports the following built-in functions.
> +
> + - $(shell,command)
> +
> +  The "shell" function accepts a single argument that is expanded and passed
> +  to a subshell for execution. The standard output of the command is then read
> +  and returned as the value of the function. Every newline in the output is
> +  replaced with a space. Any trailing newlines are deleted. The standard error
> +  is not returned, nor is any program exit status.
> +
> + - $(info,text)
> +
> +  The "info" function takes a single argument and prints it to stdout.
> +  It evaluates to an empty string.
> +
> + - $(warning,text)
> +
> +  The "warning" function is similar to "info" except that it sends its argument
> +  to stderr and prefixes the output with the name of the current Kconfig file
> +  and the current line number.
> +
> + - $(error,text)
> +
> +  The "error" function is similar to "warning", but it terminates the parsing
> +  immediately.
> +
> + - $(if,condition,then-part[,else-part])
> +
> +  The "if" function takes two or three arguments ('else-part' is optional).
> +  Depending on the value of the condition part, the argument to be expanded
> +  is selected. The condition is true if its expansion contains any characters
> +  except whitespaces. In this case, the then-part is expanded. Otherwise, the
> +  else-part is expanded.
> +
> +  Note:
> +  In Make, the condition is true if it contains any characters including
> +  whitespaces, which is why $(strip ...) is sometimes necessary in the
> +  condition part. Kconfig changed the behavior to make it handier.
> +
> + - $(filename)
> +
> +  The 'filename' takes no argument, and $(filename) is expanded to a file name

                                                                   to the file name

> +  being parsed.
> +
> + - $(lineno)
> +
> +  The 'lineno' takes no argument, and $(lineno) is expanded to a line number

                                                               to the line number

> +  being parsed.
> +
> +
> +Difference of function call syntax
> +----------------------------------
> +
> +Kconfig adopts Make-like macro language, but the function call syntax is
> +slightly different.
> +
> +A function call in Make looks like follows:

                           looks like the following:
or                         looks like this:

> +
> +  $(func-name arg1,arg2,arg3)
> +
> +The function name and the first argument are separated by at least one
> +whitespace. Then, leading whitespaces are trimmed from the first argument,
> +but whitespaces in the other arguments are kept. You need to use a kind of
> +trick to start the first parameter with spaces. For example, if you want
> +to make "info" function print "  hello", you can write like follows:
> +
> +  $(info $(space)$(space)hello)
> +
> +Kconfig uses only commas for delimiters, and keeps all whitespaces in the
> +function call. Some people prefer putting a space after each comma delimiter:
> +
> +  $(func-name, arg1, arg2, arg3)
> +
> +In this case, "func-name" will receive " arg1", " arg2", " arg3". The presence
> +of leading spaces may really matter depending on the function. The same applies
> +to Make - for example, $(subst .c, .o, $(sources)) is a typical mistake.
> +
> +In Make, a user-defined function is referenced by using a built-in function,
> +'call', like this:
> +
> +    $(call my-func,arg1,arg2,arg3)
> +
> +Kconfig invokes user-defined functions and built-in functions in the same way.
> +The omission of 'call' makes the syntax shorter.
> +
> +In Make, some functions exceptionally treat commas verbatim instead of argument

or:                        essentially (?)

> +separators. For example, $(shell echo hello, world) evaluates to "hello, world".
> +Likewise, $(info hello, world) prints "hello, world" to stdout. You could say
> +this is _useful_ inconsistency.
> +
> +For simpler implementation and grammatical consistency, Kconfig always treats
> +commas that appear in the $( ) form as delimiters. It means
> +
> +  $(shell, echo hello, world)
> +
> +is an error because it is passing two parameters where the 'shell' function
> +accepts only one. To pass commas in arguments, you can use the following trick:
> +
> +  comma := ,
> +  $(shell, echo hello$(comma) world)


ugh :)
I would prefer:
     $(shell, echo "hello, world")

> +
> +
> +Caveats
> +-------
> +
> +A variable (or function) cannot be expanded across tokens. So, you cannot use
> +a variable as a shorthand for an expression that consists of multiple tokens.
> +The following works:
> +
> +    RANGE_MIN := 1
> +    RANGE_MAX := 3
> +
> +    config FOO
> +            int "foo"
> +            range $(RANGE_MIN) $(RANGE_MAX)
> +
> +But, the following does not work:
> +
> +    RANGES := 1 3
> +
> +    config FOO
> +            int "foo"
> +            range $(RANGES)
> +
> +A variable cannot be expanded to any keyword in Kconfig.  The following does
> +not work:
> +
> +    MY_TYPE := tristate
> +
> +    config FOO
> +            $(MY_TYPE) "foo"
> +            default y
> +
> +Obviously from the design, $(shell command) is expanded in the textual
> +substitution phase. You cannot pass symbols to the 'shell' function.
> +The following does not work as expected.
> +
> +    config ENDIAN_FLAG
> +            string
> +            default "-mbig-endian" if CPU_BIG_ENDIAN
> +            default "-mlittle-endian" if CPU_LITTLE_ENDIAN
> +
> +    config CC_HAS_ENDIAN_FLAG
> +            def_bool $(shell $(srctree)/scripts/gcc-check-flag ENDIAN_FLAG)
> +
> +Instead, you can do like follows so that any function call is statically
> +expanded.
> +
> +    config CC_HAS_ENDIAN_FLAG
> +            bool
> +            default $(shell $(srctree)/scripts/gcc-check-flag -mbig-endian) if CPU_BIG_ENDIAN
> +            default $(shell $(srctree)/scripts/gcc-check-flag -mlittle-endian) if CPU_LITTLE_ENDIAN


You may add:
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>

thanks,
-- 
~Randy

Re: [PATCH v4 03/31] kconfig: reference environment variables directly and remove 'option env='

[Ulf Magnusson]

On Thu, May 24, 2018 at 6:45 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2018-05-21 20:06 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>
>>>
>>> static char *__expand_string(const char **str, bool (*is_end)(const char *))
>>> {
>>>         const char *in, *prev_in, *p;
>>>         char *new, *out;
>>>         size_t outlen;
>>>
>>>         out = xmalloc(1);
>>>         *out = 0;
>>>
>>>         p = in = *str;
>>>
>>>         while (1) {
>>>                 if (*p == '$') {
>>>                         prev_in = in;
>>>                         in = p + 1;
>>>                         new = expand_dollar(&in);
>>>                         outlen = strlen(out) + (p - prev_in) + strlen(new) + 1;
>>>                         out = xrealloc(out, outlen);
>>>                         strncat(out, prev_in, p - prev_in);
>>>                         strcat(out, new);
>>>                         free(new);
>>>                         p = in;
>>>                         continue;
>>>                 }
>>>
>>>                 if (is_end(p))
>>>                         break;
>>>
>>>                 p++;
>>>         }
>>>
>>>         outlen = strlen(out) + (p - in) + 1;
>>>         out = xrealloc(out, outlen);
>>>         strncat(out, in, p - in);
>>>
>>>         *str = p;
>>>
>>>         return out;
>>> }
>>>
>>> static bool is_end_of_str(const char *s)
>>> {
>>>         return !*s;
>>> }
>>>
>>> char *expand_string(const char *in)
>>> {
>>>         return __expand_string(&in, is_end_of_str);
>>> }
>>>
>>> static bool is_end_of_token(const char *s)
>>> {
>>>         return !(isalnum(*s) || *s == '_' || *s == '-' || *s == '.' ||
>>> *s == '/');
>>> }
>>>
>>> char *expand_one_token(const char **str)
>>> {
>>>         return __expand_string(str, is_end_of_token);
>>> }
>>
>> Yep, something like that would be nicer I think.
>>
>> This variant might work too (untested):
>>
>>   dollar_i = p;
>>   p++;
>>   expansion = expand_dollar(&p);
>>
>>   out = xrealloc(out, strlen(out) + (dollar_i - in)
>>                       + strlen(expansion) + 1);
>>   strncat(out, in, dollar_i - in);
>>   strcat(out, expansion);
>>   free(expansion);
>>
>>   in = p;
>>
>>   continue;
>>
>> The p++ would disappear if expand_dollar() took a pointer to the '$'.
>>
>
>
> I took the variable name "expansion" because
> it is more descriptive than "new".
>
>
> I rewrote like follows:
>
>
> static char *__expand_string(const char **str, bool (*is_end)(const char *))
> {
>         const char *in, *p;
>         char *expansion, *out;
>         size_t in_len, out_len;
>
>         out = xmalloc(1);
>         *out = 0;
>         out_len = 1;
>
>         p = in = *str;
>
>         while (1) {
>                 if (*p == '$') {
>                         in_len = p - in;
>                         p++;
>                         expansion = expand_dollar(&p);
>                         out_len += in_len + strlen(expansion);
>                         out = xrealloc(out, out_len);
>                         strncat(out, in, in_len);
>                         strcat(out, expansion);
>                         free(expansion);
>                         in = p;
>                         continue;
>                 }
>
>                 if (is_end(p))
>                         break;
>
>                 p++;
>         }
>
>         in_len = p - in;
>         out_len += in_len;
>         out = xrealloc(out, out_len);
>         strncat(out, in, in_len);
>
>         /* Advance 'str' to the end character */
>         *str = p;
>
>         return out;
> }
>
>
>
> I used "out_len" to remember the length of "out"
> instead of calculating strlen(out) every time.
>
> I do not need dollar_p.
>
>
>
>
> --
> Best Regards
> Masahiro Yamada

Looks good to me.

Could keep some 'out' pointer to avoid the str(n)cat()s too, but
pretty sure it's overkilling it. Should have some general string
buffer helpers at that point I think. :)

Cheers,
Ulf