[PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

I brushed up the implementation in this version.

In the previous RFC, CC_HAS_ was described by using 'option shell=',
like this:

config CC_HAS_STACKPROTECTOR
        bool
        option shell="$CC -Werror -fstack-protector -c -x c /dev/null"

After I thought a bit more, the following syntax is more grammatical,
and flexible.

config CC_HAS_STACKPROTECTOR
        bool
        default $(shell $CC -Werror -fstack-protector -c -x c /dev/null)

This version supports cc-option, so it can be written as:

config CC_HAS_STACKPROTECTOR
        bool
        default $(cc-option -fstack-protector)

To support this in a clean way, I introduced a new concept 'function'
like we see in Makefiles.

$(shell ...) is a built-in function.  $(cc-option ...) is implemented
as macro (user-defined function).

I also try cleaning of stack-protector, gcc-plugins since the Makefile
is so dirty.

Current limitations:

Dependency on outside scripts.
  For example, scripts/gcc-x86_64-has-stack-protecter.sh is run from
  Kconfig.  When the shell script is updated, should Kconfig be re-run
  automatically?

Inter-option dependency:
  $(call cc-option,...) in Makefile accumulates added options to
  KBUILD_CFLAGS, but it is difficult to do it in Kconfig.
  If a compiler option check is dependent on another option,
  this is a difficult case.  Let's see how significant it is.

Functions are evaluated statically:
  Functions are only expanded when parsing the Kconfig.  So, it can
  not refelect user configuration.  If this is required, $(shell )
  must be dynamically re-calculated depending on other symbols.
  But, this is difficult, and may cause performance issue.



Masahiro Yamada (22):
  kbuild: remove kbuild cache
  kbuild: remove CONFIG_CROSS_COMPILE support
  kconfig: add xstrdup() helper
  kconfig: set SYMBOL_AUTO to the symbol marked with defconfig_list
  kconfig: move and rename sym_expand_string_value()
  kconfig: reference environments directly and remove 'option env='
    syntax
  kconfig: add function support and implement 'shell' function
  kconfig: add 'macro' keyword to support user-defined function
  kconfig: add 'cc-option' macro
  stack-protector: test compiler capability in Kconfig and drop AUTO
    mode
  kconfig: add 'shell-stdout' function
  kconfig: replace $UNAME_RELEASE with function call
  kconfig: expand environments/functions in (main)menu, comment, prompt
  kconfig: show compiler version text in the top comment
  kconfig: add CC_IS_GCC and GCC_VERSION
  kconfig: add CC_IS_CLANG and CLANG_VERSION
  gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT
  kcov: imply GCC_PLUGINS and GCC_PLUGIN_SANCOV instead of select'ing
    them
  gcc-plugins: always build plugins with C++
  gcc-plugins: move GCC version check for PowerPC to Kconfig
  gcc-plugins: test GCC plugin support in Kconfig
  gcc-plugins: enable GCC_PLUGINS for COMPILE_TEST

Sami Tolvanen (1):
  kbuild: add clang-version.sh

 Documentation/kbuild/kconfig-language.txt |   8 -
 Kconfig                                   |   4 +-
 Makefile                                  | 103 ++----------
 arch/Kconfig                              |  43 +++--
 arch/powerpc/Kconfig                      |   2 +-
 arch/sh/Kconfig                           |   4 +-
 arch/sparc/Kconfig                        |   4 +-
 arch/tile/Kconfig                         |   2 +-
 arch/um/Kconfig.common                    |   4 -
 arch/x86/Kconfig                          |  12 +-
 arch/x86/um/Kconfig                       |   4 +-
 init/Kconfig                              |  44 +++---
 kernel/gcov/Kconfig                       |  18 +--
 kernel/gcov/Makefile                      |   2 -
 lib/Kconfig.debug                         |   7 +-
 scripts/Kbuild.include                    | 101 ++----------
 scripts/Makefile.gcc-plugins              |  95 ++++-------
 scripts/clang-version.sh                  |  31 ++++
 scripts/gcc-plugin.sh                     |  37 +----
 scripts/gcc-plugins/Makefile              |  15 +-
 scripts/gcc-x86_32-has-stack-protector.sh |   7 +-
 scripts/gcc-x86_64-has-stack-protector.sh |   5 -
 scripts/kconfig/confdata.c                |  33 +---
 scripts/kconfig/function.c                | 251 ++++++++++++++++++++++++++++++
 scripts/kconfig/kconf_id.c                |   2 +-
 scripts/kconfig/kxgettext.c               |   2 +-
 scripts/kconfig/lkc.h                     |   6 +-
 scripts/kconfig/lkc_proto.h               |   7 +-
 scripts/kconfig/menu.c                    |   6 +-
 scripts/kconfig/symbol.c                  | 139 +++--------------
 scripts/kconfig/util.c                    | 186 ++++++++++++++++++++--
 scripts/kconfig/zconf.l                   |  40 ++++-
 scripts/kconfig/zconf.y                   |  48 +++---
 33 files changed, 687 insertions(+), 585 deletions(-)
 create mode 100755 scripts/clang-version.sh
 create mode 100644 scripts/kconfig/function.c

-- 
2.7.4

[PATCH 01/23] kbuild: remove kbuild cache

[Masahiro Yamada]

The kbuild cache was introduced to remember the result of shell
commands, some of which are expensive to compute, such as
$(call cc-option,...).

However, this turned out not so clear as I had first expected.
Actually, it is problematic.  For example, "$(CC) -print-file-name"
is cached.  If the compiler is updated, the stale search path causes
build error, which is difficult to figure out.  Another problem
scenario is cache files could be touched while install targets are
running under the root permission.  We can patch them if desired,
but the build infrastructure is getting uglier and uglier.

Now, we are going to move compiler flag tests to the configuration
phase.  If this is completed, the result of compiler tests will be
naturally cached in the .config file.  We will not have performance
issues of incremental building since this testing only happens at
Kconfig time.

To start this work with a cleaner code base, remove the kbuild
cache first.

Revert the following commits:
Commit 9a234a2e384349 ("kbuild: create directory for make cache only when necessary")
Commit e17c400ae194945ee ("kbuild: shrink .cache.mk when it exceeds 1000 lines")
Commit 4e56207130eda9 ("kbuild: Cache a few more calls to the compiler")
Commit 3298b690b21cdb ("kbuild: Add a cache for generated variables")

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 Makefile               |   5 +--
 scripts/Kbuild.include | 101 +++++++------------------------------------------
 2 files changed, 16 insertions(+), 90 deletions(-)

diff --git a/Makefile b/Makefile
index 79ad2bf..a27e6d8 100644
--- a/Makefile
+++ b/Makefile
@@ -655,7 +655,7 @@ KBUILD_CFLAGS += $(call cc-ifversion, -lt, 0409, \
 KBUILD_CFLAGS	+= $(call cc-option,--param=allow-store-data-races=0)
 
 # check for 'asm goto'
-ifeq ($(call shell-cached,$(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC) $(KBUILD_CFLAGS)), y)
+ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC) $(KBUILD_CFLAGS)), y)
 	KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
 	KBUILD_AFLAGS += -DCC_HAVE_ASM_GOTO
 endif
@@ -810,7 +810,7 @@ KBUILD_CFLAGS	+= $(call cc-option,-fdata-sections,)
 endif
 
 # arch Makefile may override CC so keep this after arch Makefile is included
-NOSTDINC_FLAGS += -nostdinc -isystem $(call shell-cached,$(CC) -print-file-name=include)
+NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)
 CHECKFLAGS     += $(NOSTDINC_FLAGS)
 
 # warn about C99 declaration after statement
@@ -1601,7 +1601,6 @@ clean: $(clean-dirs)
 		-o -name '.*.d' -o -name '.*.tmp' -o -name '*.mod.c' \
 		-o -name '*.symtypes' -o -name 'modules.order' \
 		-o -name modules.builtin -o -name '.tmp_*.o.*' \
-		-o -name .cache.mk \
 		-o -name '*.c.[012]*.*' \
 		-o -name '*.ll' \
 		-o -name '*.gcno' \) -type f -print | xargs rm -f
diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index 065324a..9776946 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -8,8 +8,6 @@ squote  := '
 empty   :=
 space   := $(empty) $(empty)
 space_escape := _-_SPACE_-_
-right_paren := )
-left_paren := (
 
 ###
 # Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
@@ -82,71 +80,6 @@ cc-cross-prefix =  \
 			echo $(c);                                    \
 		fi)))
 
-# Tools for caching Makefile variables that are "expensive" to compute.
-#
-# Here we want to help deal with variables that take a long time to compute
-# by making it easy to store these variables in a cache.
-#
-# The canonical example here is testing for compiler flags.  On a simple system
-# each call to the compiler takes 10 ms, but on a system with a compiler that's
-# called through various wrappers it can take upwards of 100 ms.  If we have
-# 100 calls to the compiler this can take 1 second (on a simple system) or 10
-# seconds (on a complicated system).
-#
-# The "cache" will be in Makefile syntax and can be directly included.
-# Any time we try to reference a variable that's not in the cache we'll
-# calculate it and store it in the cache for next time.
-
-# Include values from last time
-make-cache := $(if $(KBUILD_EXTMOD),$(KBUILD_EXTMOD)/,$(if $(obj),$(obj)/)).cache.mk
-$(make-cache): ;
--include $(make-cache)
-
-cached-data := $(filter __cached_%, $(.VARIABLES))
-
-# If cache exceeds 1000 lines, shrink it down to 500.
-ifneq ($(word 1000,$(cached-data)),)
-$(shell tail -n 500 $(make-cache) > $(make-cache).tmp; \
-	mv $(make-cache).tmp $(make-cache))
-endif
-
-create-cache-dir := $(if $(KBUILD_SRC),$(if $(cache-data),,1))
-
-# Usage: $(call __sanitize-opt,Hello=Hola$(comma)Goodbye Adios)
-#
-# Convert all '$', ')', '(', '\', '=', ' ', ',', ':' to '_'
-__sanitize-opt = $(subst $$,_,$(subst $(right_paren),_,$(subst $(left_paren),_,$(subst \,_,$(subst =,_,$(subst $(space),_,$(subst $(comma),_,$(subst :,_,$(1)))))))))
-
-# Usage:   $(call shell-cached,shell_command)
-# Example: $(call shell-cached,md5sum /usr/bin/gcc)
-#
-# If we've already seen a call to this exact shell command (even in a
-# previous invocation of make!) we'll return the value.  If not, we'll
-# compute it and store the result for future runs.
-#
-# This is a bit of voodoo, but basic explanation is that if the variable
-# was undefined then we'll evaluate the shell command and store the result
-# into the variable.  We'll then store that value in the cache and finally
-# output the value.
-#
-# NOTE: The $$(2) here isn't actually a parameter to __run-and-store.  We
-# happen to know that the caller will have their shell command in $(2) so the
-# result of "call"ing this will produce a reference to that $(2).  The reason
-# for this strangeness is to avoid an extra level of eval (and escaping) of
-# $(2).
-define __run-and-store
-ifeq ($(origin $(1)),undefined)
-  $$(eval $(1) := $$(shell $$(2)))
-ifeq ($(create-cache-dir),1)
-  $$(shell mkdir -p $(dir $(make-cache)))
-  $$(eval create-cache-dir :=)
-endif
-  $$(shell echo '$(1) := $$($(1))' >> $(make-cache))
-endif
-endef
-__shell-cached = $(eval $(call __run-and-store,$(1)))$($(1))
-shell-cached = $(call __shell-cached,__cached_$(call __sanitize-opt,$(1)),$(1))
-
 # output directory for tests below
 TMPOUT := $(if $(KBUILD_EXTMOD),$(firstword $(KBUILD_EXTMOD))/)
 
@@ -154,36 +87,30 @@ TMPOUT := $(if $(KBUILD_EXTMOD),$(firstword $(KBUILD_EXTMOD))/)
 # Usage: option = $(call try-run, $(CC)...-o "$$TMP",option-ok,otherwise)
 # Exit code chooses option. "$$TMP" serves as a temporary file and is
 # automatically cleaned up.
-__try-run = set -e;			\
+try-run = $(shell set -e;		\
 	TMP="$(TMPOUT).$$$$.tmp";	\
 	TMPO="$(TMPOUT).$$$$.o";	\
 	if ($(1)) >/dev/null 2>&1;	\
 	then echo "$(2)";		\
 	else echo "$(3)";		\
 	fi;				\
-	rm -f "$$TMP" "$$TMPO"
-
-try-run = $(shell $(__try-run))
-
-# try-run-cached
-# This works like try-run, but the result is cached.
-try-run-cached = $(call shell-cached,$(__try-run))
+	rm -f "$$TMP" "$$TMPO")
 
 # as-option
 # Usage: cflags-y += $(call as-option,-Wa$(comma)-isa=foo,)
 
-as-option = $(call try-run-cached,\
+as-option = $(call try-run,\
 	$(CC) $(KBUILD_CFLAGS) $(1) -c -x assembler /dev/null -o "$$TMP",$(1),$(2))
 
 # as-instr
 # Usage: cflags-y += $(call as-instr,instr,option1,option2)
 
-as-instr = $(call try-run-cached,\
+as-instr = $(call try-run,\
 	printf "%b\n" "$(1)" | $(CC) $(KBUILD_AFLAGS) -c -x assembler -o "$$TMP" -,$(2),$(3))
 
 # __cc-option
 # Usage: MY_CFLAGS += $(call __cc-option,$(CC),$(MY_CFLAGS),-march=winchip-c6,-march=i586)
-__cc-option = $(call try-run-cached,\
+__cc-option = $(call try-run,\
 	$(1) -Werror $(2) $(3) -c -x c /dev/null -o "$$TMP",$(3),$(4))
 
 # Do not attempt to build with gcc plugins during cc-option tests.
@@ -203,23 +130,23 @@ hostcc-option = $(call __cc-option, $(HOSTCC),\
 
 # cc-option-yn
 # Usage: flag := $(call cc-option-yn,-march=winchip-c6)
-cc-option-yn = $(call try-run-cached,\
+cc-option-yn = $(call try-run,\
 	$(CC) -Werror $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) $(1) -c -x c /dev/null -o "$$TMP",y,n)
 
 # cc-disable-warning
 # Usage: cflags-y += $(call cc-disable-warning,unused-but-set-variable)
-cc-disable-warning = $(call try-run-cached,\
+cc-disable-warning = $(call try-run,\
 	$(CC) -Werror $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) -W$(strip $(1)) -c -x c /dev/null -o "$$TMP",-Wno-$(strip $(1)))
 
 # cc-name
 # Expands to either gcc or clang
-cc-name = $(call shell-cached,$(CC) -v 2>&1 | grep -q "clang version" && echo clang || echo gcc)
+cc-name = $(shell $(CC) -v 2>&1 | grep -q "clang version" && echo clang || echo gcc)
 
 # cc-version
-cc-version = $(call shell-cached,$(CONFIG_SHELL) $(srctree)/scripts/gcc-version.sh $(CC))
+cc-version = $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-version.sh $(CC))
 
 # cc-fullversion
-cc-fullversion = $(call shell-cached,$(CONFIG_SHELL) \
+cc-fullversion = $(shell $(CONFIG_SHELL) \
 	$(srctree)/scripts/gcc-version.sh -p $(CC))
 
 # cc-ifversion
@@ -232,23 +159,23 @@ cc-if-fullversion = $(shell [ $(cc-fullversion) $(1) $(2) ] && echo $(3) || echo
 
 # cc-ldoption
 # Usage: ldflags += $(call cc-ldoption, -Wl$(comma)--hash-style=both)
-cc-ldoption = $(call try-run-cached,\
+cc-ldoption = $(call try-run,\
 	$(CC) $(1) $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) -nostdlib -x c /dev/null -o "$$TMP",$(1),$(2))
 
 # ld-option
 # Usage: LDFLAGS += $(call ld-option, -X)
-ld-option = $(call try-run-cached,\
+ld-option = $(call try-run,\
 	$(CC) $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) -x c /dev/null -c -o "$$TMPO"; \
 	$(LD) $(LDFLAGS) $(1) "$$TMPO" -o "$$TMP",$(1),$(2))
 
 # ar-option
 # Usage: KBUILD_ARFLAGS := $(call ar-option,D)
 # Important: no spaces around options
-ar-option = $(call try-run-cached, $(AR) rc$(1) "$$TMP",$(1),$(2))
+ar-option = $(call try-run, $(AR) rc$(1) "$$TMP",$(1),$(2))
 
 # ld-version
 # Note this is mainly for HJ Lu's 3 number binutil versions
-ld-version = $(call shell-cached,$(LD) --version | $(srctree)/scripts/ld-version.sh)
+ld-version = $(shell $(LD) --version | $(srctree)/scripts/ld-version.sh)
 
 # ld-ifversion
 # Usage:  $(call ld-ifversion, -ge, 22252, y)
-- 
2.7.4

[PATCH 02/23] kbuild: remove CONFIG_CROSS_COMPILE support

[Masahiro Yamada]

Kbuild provides a couple of ways to specify CROSS_COMPILE:

[1] Command line
[2] Environment
[3] arch/*/Makefile (only some architectures)
[4] CONFIG_CROSS_COMPILE

[4] is problematic for the compiler capability tests in Kconfig.
CONFIG_CROSS_COMPILE allows users to change the compiler prefix from
'make menuconfig', etc.  It means, the compiler options would have
to be all re-calculated everytime CONFIG_CROSS_COMPILE is changed.

To avoid complexity and performance issues, I'd like to evaluate
the shell commands statically, i.e. only parsing Kconfig files.

I guess the majority is [1] or [2].  Currently, there are only
4 defconfig files that specify CONFIG_CROSS_COMPILE.
  arch/arm/configs/lpc18xx_defconfig
  arch/hexagon/configs/comet_defconfig
  arch/openrisc/configs/or1ksim_defconfig
  arch/openrisc/configs/simple_smp_defconfig

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 Makefile     | 3 ---
 init/Kconfig | 9 ---------
 2 files changed, 12 deletions(-)

diff --git a/Makefile b/Makefile
index a27e6d8..94a957e 100644
--- a/Makefile
+++ b/Makefile
@@ -307,12 +307,9 @@ SUBARCH := $(shell uname -m | sed -e s/i.86/x86/ -e s/x86_64/x86/ \
 # CROSS_COMPILE can be set on the command line
 # make CROSS_COMPILE=ia64-linux-
 # Alternatively CROSS_COMPILE can be set in the environment.
-# A third alternative is to store a setting in .config so that plain
-# "make" in the configured kernel build directory always uses that.
 # Default value for CROSS_COMPILE is not to prefix executables
 # Note: Some architectures assign CROSS_COMPILE in their arch/*/Makefile
 ARCH		?= $(SUBARCH)
-CROSS_COMPILE	?= $(CONFIG_CROSS_COMPILE:"%"=%)
 
 # Architecture as present in compile.h
 UTS_MACHINE 	:= $(ARCH)
diff --git a/init/Kconfig b/init/Kconfig
index e37f4b2..df18492 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -54,15 +54,6 @@ config INIT_ENV_ARG_LIMIT
 	  Maximum of each of the number of arguments and environment
 	  variables passed to init from the kernel command line.
 
-
-config CROSS_COMPILE
-	string "Cross-compiler tool prefix"
-	help
-	  Same as running 'make CROSS_COMPILE=prefix-' but stored for
-	  default make runs in this kernel build directory.  You don't
-	  need to set this unless you want the configured kernel build
-	  directory to select the cross-compiler automatically.
-
 config COMPILE_TEST
 	bool "Compile also drivers which will not load"
 	depends on !UML
-- 
2.7.4

[PATCH 03/23] kconfig: add xstrdup() helper

[Masahiro Yamada]

We already have xmalloc(), xcalloc(), and xrealloc(().  Add xstrdup()
as well to save tedious error handling.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 scripts/kconfig/confdata.c  |  2 +-
 scripts/kconfig/kxgettext.c |  2 +-
 scripts/kconfig/lkc.h       |  1 +
 scripts/kconfig/symbol.c    |  4 ++--
 scripts/kconfig/util.c      | 11 +++++++++++
 scripts/kconfig/zconf.y     |  2 +-
 6 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
index 5c12dc9..df26c7b 100644
--- a/scripts/kconfig/confdata.c
+++ b/scripts/kconfig/confdata.c
@@ -178,7 +178,7 @@ static int conf_set_sym_val(struct symbol *sym, int def, int def_flags, char *p)
 	case S_HEX:
 	done:
 		if (sym_string_valid(sym, p)) {
-			sym->def[def].val = strdup(p);
+			sym->def[def].val = xstrdup(p);
 			sym->flags |= def_flags;
 		} else {
 			if (def != S_DEF_AUTO)
diff --git a/scripts/kconfig/kxgettext.c b/scripts/kconfig/kxgettext.c
index 2858738..240880a 100644
--- a/scripts/kconfig/kxgettext.c
+++ b/scripts/kconfig/kxgettext.c
@@ -101,7 +101,7 @@ static struct message *message__new(const char *msg, char *option,
 	if (self->files == NULL)
 		goto out_fail;
 
-	self->msg = strdup(msg);
+	self->msg = xstrdup(msg);
 	if (self->msg == NULL)
 		goto out_fail_msg;
 
diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
index 4e23feb..2d5ec2d 100644
--- a/scripts/kconfig/lkc.h
+++ b/scripts/kconfig/lkc.h
@@ -115,6 +115,7 @@ int file_write_dep(const char *name);
 void *xmalloc(size_t size);
 void *xcalloc(size_t nmemb, size_t size);
 void *xrealloc(void *p, size_t size);
+char *xstrdup(const char *s);
 
 struct gstr {
 	size_t len;
diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
index cca9663..2220bc4 100644
--- a/scripts/kconfig/symbol.c
+++ b/scripts/kconfig/symbol.c
@@ -183,7 +183,7 @@ static void sym_validate_range(struct symbol *sym)
 		sprintf(str, "%lld", val2);
 	else
 		sprintf(str, "0x%llx", val2);
-	sym->curr.val = strdup(str);
+	sym->curr.val = xstrdup(str);
 }
 
 static void sym_set_changed(struct symbol *sym)
@@ -849,7 +849,7 @@ struct symbol *sym_lookup(const char *name, int flags)
 				   : !(symbol->flags & (SYMBOL_CONST|SYMBOL_CHOICE))))
 				return symbol;
 		}
-		new_name = strdup(name);
+		new_name = xstrdup(name);
 	} else {
 		new_name = NULL;
 		hash = 0;
diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index b98a79e..c6f6e21 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -154,3 +154,14 @@ void *xrealloc(void *p, size_t size)
 	fprintf(stderr, "Out of memory.\n");
 	exit(1);
 }
+
+char *xstrdup(const char *s)
+{
+	char *p;
+
+	p = strdup(s);
+	if (p)
+		return p;
+	fprintf(stderr, "Out of memory.\n");
+	exit(1);
+}
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index 4893676..ad6305b 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -127,7 +127,7 @@ no_mainmenu_stmt: /* empty */
 	 * later regardless of whether it comes from the 'prompt' in
 	 * mainmenu_stmt or here
 	 */
-	menu_add_prompt(P_MENU, strdup("Linux Kernel Configuration"), NULL);
+	menu_add_prompt(P_MENU, xstrdup("Linux Kernel Configuration"), NULL);
 };
 
 
-- 
2.7.4

[PATCH 04/23] kconfig: set SYMBOL_AUTO to the symbol marked with defconfig_list

[Masahiro Yamada]

The 'defconfig_list' is a weird attribute.  If the '.config' is
missing, conf_read_simple() iterates over all visible defaults,
then it uses the first one for which fopen() succeeds.

config DEFCONFIG_LIST
	string
	depends on !UML
	option defconfig_list
	default "/lib/modules/$UNAME_RELEASE/.config"
	default "/etc/kernel-config"
	default "/boot/config-$UNAME_RELEASE"
	default "$ARCH_DEFCONFIG"
	default "arch/$ARCH/defconfig"

However, like other symbols, the first visible default is always
written out to the .config file.  This might be different from what
has been actually used.

For example, on my machine, the third one "/boot/config-$UNAME_RELEASE"
is opened, like follows:

  $ rm .config
  $ make oldconfig 2>/dev/null
  scripts/kconfig/conf  --oldconfig Kconfig
  #
  # using defaults found in /boot/config-4.4.0-112-generic
  #
  *
  * Restart config...
  *
  *
  * IRQ subsystem
  *
  Expose irq internals in debugfs (GENERIC_IRQ_DEBUGFS) [N/y/?] (NEW)

However, the resulted .config file contains the first one since it is
visible:

  $ grep CONFIG_DEFCONFIG_LIST .config
  CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

In order to stop confusing people, prevent this CONFIG option from
being written to the .config file.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

I'd like to fix the root case of this weirdness later.
(and other 'option' attributes as well)

But, this series is focusing a more important work in a bigger picture.

For now, I decided to just hide CONFIG_DEFCONFIG_LIST
from the .config file.


 scripts/kconfig/menu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
index 9922285..36cd3e1 100644
--- a/scripts/kconfig/menu.c
+++ b/scripts/kconfig/menu.c
@@ -212,6 +212,7 @@ void menu_add_option(int token, char *arg)
 			sym_defconfig_list = current_entry->sym;
 		else if (sym_defconfig_list != current_entry->sym)
 			zconf_error("trying to redefine defconfig symbol");
+		sym_defconfig_list->flags |= SYMBOL_AUTO;
 		break;
 	case T_OPT_ENV:
 		prop_add_env(arg);
-- 
2.7.4

[PATCH 05/23] kconfig: move and rename sym_expand_string_value()

[Masahiro Yamada]

This helper expands symbols contained in a string.  I am about to
change it to expand environments instead of symbols.  Also, I will
add function expansion later.

Rename it to expand_string_value(), and move it to util.c, which is
a more suitable place.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 scripts/kconfig/lkc.h       |  1 +
 scripts/kconfig/lkc_proto.h |  1 -
 scripts/kconfig/symbol.c    | 53 -------------------------------------------
 scripts/kconfig/util.c      | 55 ++++++++++++++++++++++++++++++++++++++++++++-
 scripts/kconfig/zconf.y     |  2 +-
 5 files changed, 56 insertions(+), 56 deletions(-)

diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
index 2d5ec2d..fb2503a 100644
--- a/scripts/kconfig/lkc.h
+++ b/scripts/kconfig/lkc.h
@@ -110,6 +110,7 @@ void menu_finalize(struct menu *parent);
 void menu_set_type(int type);
 
 /* util.c */
+char *expand_string_value(const char *in);
 struct file *file_lookup(const char *name);
 int file_write_dep(const char *name);
 void *xmalloc(size_t size);
diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
index 9dc8abf..9884adc 100644
--- a/scripts/kconfig/lkc_proto.h
+++ b/scripts/kconfig/lkc_proto.h
@@ -31,7 +31,6 @@ extern struct symbol * symbol_hash[SYMBOL_HASHSIZE];
 
 struct symbol * sym_lookup(const char *name, int flags);
 struct symbol * sym_find(const char *name);
-char *sym_expand_string_value(const char *in);
 const char * sym_escape_string_value(const char *in);
 struct symbol ** sym_re_search(const char *pattern);
 const char * sym_type_name(enum symbol_type type);
diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
index 2220bc4..e4ccf56 100644
--- a/scripts/kconfig/symbol.c
+++ b/scripts/kconfig/symbol.c
@@ -894,59 +894,6 @@ struct symbol *sym_find(const char *name)
 	return symbol;
 }
 
-/*
- * Expand symbol's names embedded in the string given in argument. Symbols'
- * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
- * the empty string.
- */
-char *sym_expand_string_value(const char *in)
-{
-	const char *src;
-	char *res;
-	size_t reslen;
-
-	/*
-	 * Note: 'in' might come from a token that's about to be
-	 * freed, so make sure to always allocate a new string
-	 */
-	reslen = strlen(in) + 1;
-	res = xmalloc(reslen);
-	res[0] = '\0';
-
-	while ((src = strchr(in, '$'))) {
-		char *p, name[SYMBOL_MAXLENGTH];
-		const char *symval = "";
-		struct symbol *sym;
-		size_t newlen;
-
-		strncat(res, in, src - in);
-		src++;
-
-		p = name;
-		while (isalnum(*src) || *src == '_')
-			*p++ = *src++;
-		*p = '\0';
-
-		sym = sym_find(name);
-		if (sym != NULL) {
-			sym_calc_value(sym);
-			symval = sym_get_string_value(sym);
-		}
-
-		newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
-		if (newlen > reslen) {
-			reslen = newlen;
-			res = xrealloc(res, reslen);
-		}
-
-		strcat(res, symval);
-		in = src;
-	}
-	strcat(res, in);
-
-	return res;
-}
-
 const char *sym_escape_string_value(const char *in)
 {
 	const char *p;
diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index c6f6e21..22201a4 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -10,11 +10,64 @@
 #include <string.h>
 #include "lkc.h"
 
+/*
+ * Expand symbol's names embedded in the string given in argument. Symbols'
+ * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
+ * the empty string.
+ */
+char *expand_string_value(const char *in)
+{
+	const char *src;
+	char *res;
+	size_t reslen;
+
+	/*
+	 * Note: 'in' might come from a token that's about to be
+	 * freed, so make sure to always allocate a new string
+	 */
+	reslen = strlen(in) + 1;
+	res = xmalloc(reslen);
+	res[0] = '\0';
+
+	while ((src = strchr(in, '$'))) {
+		char *p, name[SYMBOL_MAXLENGTH];
+		const char *symval = "";
+		struct symbol *sym;
+		size_t newlen;
+
+		strncat(res, in, src - in);
+		src++;
+
+		p = name;
+		while (isalnum(*src) || *src == '_')
+			*p++ = *src++;
+		*p = '\0';
+
+		sym = sym_find(name);
+		if (sym != NULL) {
+			sym_calc_value(sym);
+			symval = sym_get_string_value(sym);
+		}
+
+		newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
+		if (newlen > reslen) {
+			reslen = newlen;
+			res = xrealloc(res, reslen);
+		}
+
+		strcat(res, symval);
+		in = src;
+	}
+	strcat(res, in);
+
+	return res;
+}
+
 /* file already present in list? If not add it */
 struct file *file_lookup(const char *name)
 {
 	struct file *file;
-	char *file_name = sym_expand_string_value(name);
+	char *file_name = expand_string_value(name);
 
 	for (file = file_list; file; file = file->next) {
 		if (!strcmp(name, file->name)) {
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index ad6305b..262c464 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -547,7 +547,7 @@ void conf_parse(const char *name)
 
 	tmp = rootmenu.prompt->text;
 	rootmenu.prompt->text = _(rootmenu.prompt->text);
-	rootmenu.prompt->text = sym_expand_string_value(rootmenu.prompt->text);
+	rootmenu.prompt->text = expand_string_value(rootmenu.prompt->text);
 	free((char*)tmp);
 
 	menu_finalize(&rootmenu);
-- 
2.7.4

[PATCH 06/23] kconfig: reference environments directly and remove 'option env=' syntax

[Masahiro Yamada]

To get an environment value, Kconfig needs to define a symbol using
"option env=" syntax.  It is tedious to add a config entry for each
environment given that we need more environments such as 'CC', 'AS',
'srctree' etc. to evaluate the compiler capability in Kconfig.

Adding '$' to symbols is weird.  Kconfig can reference symbols directly
like this:

  config FOO
          string
          default BAR

So, I want to use the following syntax to get environment 'BAR' from
the system:

  config FOO
          string
          default $BAR

Looking at the code, the symbols prefixed with 'S' are expanded by:
 - conf_expand_value()
   This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
 - expand_string_value()
   This is used to expand strings in 'source' and 'mainmenu'

All of them are independent of user configuration, i.e. fixed values.
So, this kind of syntax should be moved to simply take the environment.

This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.

sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
should be be given from the environment.

ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
by 'default ARCH_DEFCONFIG'.

An environment can appear anywhere a symbol reference can appear.
(It is expanded by sym_lookup().)  If an expression (which is derived
from symbols) is a string, environments in the string are also expanded.

For example, the following code works.

Example code:

  config TOOLCHAIN_LIST
          string
          default "My tools: CC=$CC, AS=$AS, CPP=$CPP"

Result:

  $ make -s alldefconfig && tail -n 1 .config
  CONFIG_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

I tested all 'make *config' for arch architectures.
I confirmed this commit still produced the same result.


 Documentation/kbuild/kconfig-language.txt |   8 --
 Kconfig                                   |   4 -
 Makefile                                  |   3 +-
 arch/sh/Kconfig                           |   4 +-
 arch/sparc/Kconfig                        |   4 +-
 arch/tile/Kconfig                         |   2 +-
 arch/um/Kconfig.common                    |   4 -
 arch/x86/Kconfig                          |   4 +-
 arch/x86/um/Kconfig                       |   4 +-
 init/Kconfig                              |  10 +-
 scripts/kconfig/confdata.c                |  31 +-----
 scripts/kconfig/kconf_id.c                |   1 -
 scripts/kconfig/lkc.h                     |   4 -
 scripts/kconfig/menu.c                    |   3 -
 scripts/kconfig/symbol.c                  |  84 ++++------------
 scripts/kconfig/util.c                    | 156 +++++++++++++++++++++---------
 scripts/kconfig/zconf.l                   |   2 +-
 scripts/kconfig/zconf.y                   |   1 -
 18 files changed, 144 insertions(+), 185 deletions(-)

diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
index f5b9493..0e966e8 100644
--- a/Documentation/kbuild/kconfig-language.txt
+++ b/Documentation/kbuild/kconfig-language.txt
@@ -198,14 +198,6 @@ applicable everywhere (see syntax).
     enables the third modular state for all config symbols.
     At most one symbol may have the "modules" option set.
 
-  - "env"=<value>
-    This imports the environment variable into Kconfig. It behaves like
-    a default, except that the value comes from the environment, this
-    also means that the behaviour when mixing it with normal defaults is
-    undefined at this point. The symbol is currently not exported back
-    to the build environment (if this is desired, it can be done via
-    another symbol).
-
   - "allnoconfig_y"
     This declares the symbol as one that should have the value y when
     using "allnoconfig". Used for symbols that hide other symbols.
diff --git a/Kconfig b/Kconfig
index 8c4c1cb..e6ece5b 100644
--- a/Kconfig
+++ b/Kconfig
@@ -5,8 +5,4 @@
 #
 mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
 
-config SRCARCH
-	string
-	option env="SRCARCH"
-
 source "arch/$SRCARCH/Kconfig"
diff --git a/Makefile b/Makefile
index 94a957e..9a8c689 100644
--- a/Makefile
+++ b/Makefile
@@ -275,7 +275,8 @@ include scripts/Kbuild.include
 # Read KERNELRELEASE from include/config/kernel.release (if it exists)
 KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
 KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
-export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
+UNAME_RELEASE := $(shell uname --release)
+export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
 
 # SUBARCH tells the usermode build what the underlying arch is.  That is set
 # first, and if a usermode build is happening, the "ARCH=um" on the command
diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
index 97fe293..89fc2f6 100644
--- a/arch/sh/Kconfig
+++ b/arch/sh/Kconfig
@@ -57,7 +57,7 @@ config SUPERH
 	  <http://www.linux-sh.org/>.
 
 config SUPERH32
-	def_bool ARCH = "sh"
+	def_bool $ARCH = "sh"
 	select HAVE_KPROBES
 	select HAVE_KRETPROBES
 	select HAVE_IOREMAP_PROT if MMU && !X2TLB
@@ -76,7 +76,7 @@ config SUPERH32
 	select HAVE_CC_STACKPROTECTOR
 
 config SUPERH64
-	def_bool ARCH = "sh64"
+	def_bool $ARCH = "sh64"
 	select HAVE_EXIT_THREAD
 	select KALLSYMS
 
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 6bf594a..fed3d82 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -1,6 +1,6 @@
 config 64BIT
-	bool "64-bit kernel" if ARCH = "sparc"
-	default ARCH = "sparc64"
+	bool "64-bit kernel" if $ARCH = "sparc"
+	default $ARCH = "sparc64"
 	help
 	  SPARC is a family of RISC microprocessors designed and marketed by
 	  Sun Microsystems, incorporated.  They are very widely found in Sun
diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
index ef9d403..fed372e 100644
--- a/arch/tile/Kconfig
+++ b/arch/tile/Kconfig
@@ -119,7 +119,7 @@ config HVC_TILE
 # Building with ARCH=tilegx (or ARCH=tile) implies using the
 # 64-bit TILE-Gx toolchain, so force CONFIG_TILEGX on.
 config TILEGX
-	def_bool ARCH != "tilepro"
+	def_bool $ARCH != "tilepro"
 	select ARCH_SUPPORTS_ATOMIC_RMW
 	select GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
 	select HAVE_ARCH_JUMP_LABEL
diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
index c68add8..07f84c8 100644
--- a/arch/um/Kconfig.common
+++ b/arch/um/Kconfig.common
@@ -54,10 +54,6 @@ config HZ
 	int
 	default 100
 
-config SUBARCH
-	string
-	option env="SUBARCH"
-
 config NR_CPUS
 	int
 	range 1 1
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index a528c14..54d943a 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1,8 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
 # Select 32 or 64 bit
 config 64BIT
-	bool "64-bit kernel" if ARCH = "x86"
-	default ARCH != "i386"
+	bool "64-bit kernel" if $ARCH = "x86"
+	default $ARCH != "i386"
 	---help---
 	  Say yes to build a 64-bit kernel - formerly known as x86_64
 	  Say no to build a 32-bit kernel - formerly known as i386
diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
index 13ed827..d355413 100644
--- a/arch/x86/um/Kconfig
+++ b/arch/x86/um/Kconfig
@@ -16,8 +16,8 @@ config UML_X86
 	select GENERIC_FIND_FIRST_BIT
 
 config 64BIT
-	bool "64-bit kernel" if SUBARCH = "x86"
-	default SUBARCH != "i386"
+	bool "64-bit kernel" if $SUBARCH = "x86"
+	default $SUBARCH != "i386"
 
 config X86_32
 	def_bool !64BIT
diff --git a/init/Kconfig b/init/Kconfig
index df18492..b4814e6 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1,11 +1,3 @@
-config ARCH
-	string
-	option env="ARCH"
-
-config KERNELVERSION
-	string
-	option env="KERNELVERSION"
-
 config DEFCONFIG_LIST
 	string
 	depends on !UML
@@ -13,7 +5,7 @@ config DEFCONFIG_LIST
 	default "/lib/modules/$UNAME_RELEASE/.config"
 	default "/etc/kernel-config"
 	default "/boot/config-$UNAME_RELEASE"
-	default "$ARCH_DEFCONFIG"
+	default ARCH_DEFCONFIG
 	default "arch/$ARCH/defconfig"
 
 config CONSTRUCTORS
diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
index df26c7b..98c2014 100644
--- a/scripts/kconfig/confdata.c
+++ b/scripts/kconfig/confdata.c
@@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
 	return name ? name : "include/config/auto.conf";
 }
 
-static char *conf_expand_value(const char *in)
-{
-	struct symbol *sym;
-	const char *src;
-	static char res_value[SYMBOL_MAXLENGTH];
-	char *dst, name[SYMBOL_MAXLENGTH];
-
-	res_value[0] = 0;
-	dst = name;
-	while ((src = strchr(in, '$'))) {
-		strncat(res_value, in, src - in);
-		src++;
-		dst = name;
-		while (isalnum(*src) || *src == '_')
-			*dst++ = *src++;
-		*dst = 0;
-		sym = sym_lookup(name, 0);
-		sym_calc_value(sym);
-		strcat(res_value, sym_get_string_value(sym));
-		in = src;
-	}
-	strcat(res_value, in);
-
-	return res_value;
-}
-
 char *conf_get_default_confname(void)
 {
 	struct stat buf;
 	static char fullname[PATH_MAX+1];
 	char *env, *name;
 
-	name = conf_expand_value(conf_defname);
+	name = expand_string_value(conf_defname);
 	env = getenv(SRCTREE);
 	if (env) {
 		sprintf(fullname, "%s/%s", env, name);
@@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
 			if (expr_calc_value(prop->visible.expr) == no ||
 			    prop->expr->type != E_SYMBOL)
 				continue;
-			name = conf_expand_value(prop->expr->left.sym->name);
+			sym_calc_value(prop->expr->left.sym);
+			name = sym_get_string_value(prop->expr->left.sym);
 			in = zconf_fopen(name);
 			if (in) {
 				conf_message(_("using defaults found in %s"),
diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
index 3ea9c5f..b3e0ea0 100644
--- a/scripts/kconfig/kconf_id.c
+++ b/scripts/kconfig/kconf_id.c
@@ -32,7 +32,6 @@ static struct kconf_id kconf_id_array[] = {
 	{ "on",			T_ON,			TF_PARAM },
 	{ "modules",		T_OPT_MODULES,		TF_OPTION },
 	{ "defconfig_list",	T_OPT_DEFCONFIG_LIST,	TF_OPTION },
-	{ "env",		T_OPT_ENV,		TF_OPTION },
 	{ "allnoconfig_y",	T_OPT_ALLNOCONFIG_Y,	TF_OPTION },
 };
 
diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
index fb2503a..0ff3256 100644
--- a/scripts/kconfig/lkc.h
+++ b/scripts/kconfig/lkc.h
@@ -58,7 +58,6 @@ enum conf_def_mode {
 
 #define T_OPT_MODULES		1
 #define T_OPT_DEFCONFIG_LIST	2
-#define T_OPT_ENV		3
 #define T_OPT_ALLNOCONFIG_Y	4
 
 struct kconf_id {
@@ -134,9 +133,6 @@ void str_printf(struct gstr *gs, const char *fmt, ...);
 const char *str_get(struct gstr *gs);
 
 /* symbol.c */
-extern struct expr *sym_env_list;
-
-void sym_init(void);
 void sym_clear_all_valid(void);
 struct symbol *sym_choice_default(struct symbol *sym);
 const char *sym_get_string_default(struct symbol *sym);
diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
index 36cd3e1..a9d0ccc 100644
--- a/scripts/kconfig/menu.c
+++ b/scripts/kconfig/menu.c
@@ -214,9 +214,6 @@ void menu_add_option(int token, char *arg)
 			zconf_error("trying to redefine defconfig symbol");
 		sym_defconfig_list->flags |= SYMBOL_AUTO;
 		break;
-	case T_OPT_ENV:
-		prop_add_env(arg);
-		break;
 	case T_OPT_ALLNOCONFIG_Y:
 		current_entry->sym->flags |= SYMBOL_ALLNOCONFIG_Y;
 		break;
diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
index e4ccf56..96ea8a9 100644
--- a/scripts/kconfig/symbol.c
+++ b/scripts/kconfig/symbol.c
@@ -33,33 +33,6 @@ struct symbol *sym_defconfig_list;
 struct symbol *modules_sym;
 tristate modules_val;
 
-struct expr *sym_env_list;
-
-static void sym_add_default(struct symbol *sym, const char *def)
-{
-	struct property *prop = prop_alloc(P_DEFAULT, sym);
-
-	prop->expr = expr_alloc_symbol(sym_lookup(def, SYMBOL_CONST));
-}
-
-void sym_init(void)
-{
-	struct symbol *sym;
-	struct utsname uts;
-	static bool inited = false;
-
-	if (inited)
-		return;
-	inited = true;
-
-	uname(&uts);
-
-	sym = sym_lookup("UNAME_RELEASE", 0);
-	sym->type = S_STRING;
-	sym->flags |= SYMBOL_AUTO;
-	sym_add_default(sym, uts.release);
-}
-
 enum symbol_type sym_get_type(struct symbol *sym)
 {
 	enum symbol_type type = sym->type;
@@ -828,28 +801,38 @@ static unsigned strhash(const char *s)
 
 struct symbol *sym_lookup(const char *name, int flags)
 {
-	struct symbol *symbol;
+	struct symbol *symbol = NULL;
 	char *new_name;
 	int hash;
 
 	if (name) {
-		if (name[0] && !name[1]) {
-			switch (name[0]) {
-			case 'y': return &symbol_yes;
-			case 'm': return &symbol_mod;
-			case 'n': return &symbol_no;
+		new_name = expand_string_value(name);
+		if (new_name[0] && !new_name[1]) {
+			switch (new_name[0]) {
+			case 'y':
+				symbol = &symbol_yes;
+				break;
+			case 'm':
+				symbol = &symbol_mod;
+				break;
+			case 'n':
+				symbol = &symbol_no;
+				break;
+			}
+			if (symbol) {
+				free(new_name);
+				return symbol;
 			}
 		}
-		hash = strhash(name) % SYMBOL_HASHSIZE;
+		hash = strhash(new_name) % SYMBOL_HASHSIZE;
 
 		for (symbol = symbol_hash[hash]; symbol; symbol = symbol->next) {
 			if (symbol->name &&
-			    !strcmp(symbol->name, name) &&
+			    !strcmp(symbol->name, new_name) &&
 			    (flags ? symbol->flags & flags
 				   : !(symbol->flags & (SYMBOL_CONST|SYMBOL_CHOICE))))
 				return symbol;
 		}
-		new_name = xstrdup(name);
 	} else {
 		new_name = NULL;
 		hash = 0;
@@ -1336,32 +1319,3 @@ const char *prop_get_type_name(enum prop_type type)
 	}
 	return "unknown";
 }
-
-static void prop_add_env(const char *env)
-{
-	struct symbol *sym, *sym2;
-	struct property *prop;
-	char *p;
-
-	sym = current_entry->sym;
-	sym->flags |= SYMBOL_AUTO;
-	for_all_properties(sym, prop, P_ENV) {
-		sym2 = prop_get_symbol(prop);
-		if (strcmp(sym2->name, env))
-			menu_warn(current_entry, "redefining environment symbol from %s",
-				  sym2->name);
-		return;
-	}
-
-	prop = prop_alloc(P_ENV, sym);
-	prop->expr = expr_alloc_symbol(sym_lookup(env, SYMBOL_CONST));
-
-	sym_env_list = expr_alloc_one(E_LIST, sym_env_list);
-	sym_env_list->right.sym = sym;
-
-	p = getenv(env);
-	if (p)
-		sym_add_default(sym, p);
-	else
-		menu_warn(current_entry, "environment variable %s undefined", env);
-}
diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index 22201a4..dddf85b 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -8,16 +8,98 @@
 #include <stdarg.h>
 #include <stdlib.h>
 #include <string.h>
+
+#include "list.h"
 #include "lkc.h"
 
+static LIST_HEAD(env_list);
+
+struct env {
+	char *name;
+	char *value;
+	struct list_head node;
+};
+
+static struct env *env_list_lookup(const char *name)
+{
+	struct env *e;
+
+	list_for_each_entry(e, &env_list, node) {
+		if (!strcmp(name, e->name))
+			return e;
+	}
+
+	return NULL;
+}
+
+static void env_list_add(const char *name, const char *value)
+{
+	struct env *e;
+
+	e = xmalloc(sizeof(*e));
+	e->name = xstrdup(name);
+	e->value = xstrdup(value);
+
+	list_add_tail(&e->node, &env_list);
+}
+
+static void env_list_del(struct env *e)
+{
+	list_del(&e->node);
+	free(e->name);
+	free(e->value);
+	free(e);
+}
+
+/* the returned pointer must be freed when done */
+static char *env_expand(const char *name)
+{
+	struct env *e;
+	const char *value;
+
+	e = env_list_lookup(name);
+	if (e)
+		return xstrdup(e->value);
+
+	value = getenv(name);
+	if (!value) {
+		fprintf(stderr, "environment variable \"%s\" undefined\n", name);
+		value = "";
+	}
+
+	/*
+	 * we need to remember all referenced environments.
+	 * They will be written out to include/config/auto.conf.cmd
+	 */
+	env_list_add(name, value);
+
+	return xstrdup(value);
+}
+
+/* works like env_expand, but 'name' does not need to be null-terminated */
+static char *env_expand_n(const char *name, size_t n)
+{
+	char *tmp, *res;
+
+	tmp = xmalloc(n + 1);
+	memcpy(tmp, name, n);
+	*(tmp + n) = '\0';
+
+	res = env_expand(tmp);
+
+	free(tmp);
+
+	return res;
+}
+
 /*
- * Expand symbol's names embedded in the string given in argument. Symbols'
- * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
+ * Expand environments embedded in the string given in argument. Environments
+ * to be expanded shall be prefixed by a '$'. Unknown environment expands to
  * the empty string.
  */
 char *expand_string_value(const char *in)
 {
-	const char *src;
+	const char *p, *q;
 	char *res;
 	size_t reslen;
 
@@ -25,39 +107,28 @@ char *expand_string_value(const char *in)
 	 * Note: 'in' might come from a token that's about to be
 	 * freed, so make sure to always allocate a new string
 	 */
-	reslen = strlen(in) + 1;
-	res = xmalloc(reslen);
-	res[0] = '\0';
-
-	while ((src = strchr(in, '$'))) {
-		char *p, name[SYMBOL_MAXLENGTH];
-		const char *symval = "";
-		struct symbol *sym;
-		size_t newlen;
-
-		strncat(res, in, src - in);
-		src++;
-
-		p = name;
-		while (isalnum(*src) || *src == '_')
-			*p++ = *src++;
-		*p = '\0';
-
-		sym = sym_find(name);
-		if (sym != NULL) {
-			sym_calc_value(sym);
-			symval = sym_get_string_value(sym);
-		}
+	res = xmalloc(1);
+	*res = '\0';
 
-		newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
-		if (newlen > reslen) {
-			reslen = newlen;
-			res = xrealloc(res, reslen);
-		}
+	while ((p = strchr(in, '$'))) {
+		char *new;
+
+		q = p + 1;
+		while (isalnum(*q) || *q == '_')
+			q++;
 
-		strcat(res, symval);
-		in = src;
+		new = env_expand_n(p + 1, q - p - 1);
+
+		reslen = strlen(res) + (p - in) + strlen(new) + 1;
+		res = xrealloc(res, reslen);
+		strncat(res, in, p - in);
+		strcat(res, new);
+		free(new);
+		in = q;
 	}
+
+	reslen = strlen(res) + strlen(in) + 1;
+	res = xrealloc(res, reslen);
 	strcat(res, in);
 
 	return res;
@@ -87,8 +158,7 @@ struct file *file_lookup(const char *name)
 /* write a dependency file as used by kbuild to track dependencies */
 int file_write_dep(const char *name)
 {
-	struct symbol *sym, *env_sym;
-	struct expr *e;
+	struct env *env, *tmp;
 	struct file *file;
 	FILE *out;
 
@@ -107,20 +177,12 @@ int file_write_dep(const char *name)
 	fprintf(out, "\n%s: \\\n"
 		     "\t$(deps_config)\n\n", conf_get_autoconfig_name());
 
-	expr_list_for_each_sym(sym_env_list, e, sym) {
-		struct property *prop;
-		const char *value;
-
-		prop = sym_get_env_prop(sym);
-		env_sym = prop_get_symbol(prop);
-		if (!env_sym)
-			continue;
-		value = getenv(env_sym->name);
-		if (!value)
-			value = "";
-		fprintf(out, "ifneq \"$(%s)\" \"%s\"\n", env_sym->name, value);
+	list_for_each_entry_safe(env, tmp, &env_list, node) {
+		fprintf(out, "ifneq \"$(%s)\" \"%s\"\n",
+			env->name, getenv(env->name) ?: "");
 		fprintf(out, "%s: FORCE\n", conf_get_autoconfig_name());
 		fprintf(out, "endif\n");
+		env_list_del(env);
 	}
 
 	fprintf(out, "\n$(deps_config): ;\n");
diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
index 02de6fe..0d89ea6 100644
--- a/scripts/kconfig/zconf.l
+++ b/scripts/kconfig/zconf.l
@@ -75,7 +75,7 @@ static void warn_ignored_character(char chr)
 }
 %}
 
-n	[A-Za-z0-9_-]
+n	[$A-Za-z0-9_-]
 
 %%
 	int str = 0;
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index 262c464..784083d 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -534,7 +534,6 @@ void conf_parse(const char *name)
 
 	zconf_initscan(name);
 
-	sym_init();
 	_menu_init();
 
 	if (getenv("ZCONF_DEBUG"))
-- 
2.7.4

[PATCH 07/23] kconfig: add function support and implement 'shell' function

[Masahiro Yamada]

This commit adds a new concept 'function' to Kconfig.  A function call
resembles a variable reference with arguments, and looks like this:

  $(function arg1, arg2, arg3, ...)

(Actually, this syntax was inspired by Makefile.)

Real examples will look like this:

  $(shell true)
  $(cc-option -fstackprotector)

This commit adds the basic infrastructure to add, delete, evaluate
functions.

Also, add the first built-in function $(shell ...).  This evaluates
to 'y' if the given command exits with 0, 'n' otherwise.

I am also planning to support user-defined functions (a.k.a 'macro')
for cases where hard-coding is not preferred.

If you want to try this feature, the hello-world code is someting below.

Example code:

  config CC_IS_GCC
          bool
          default $(shell $CC --version | grep -q gcc)

  config CC_IS_CLANG
          bool
          default $(shell $CC --version | grep -q clang)

  config CC_HAS_OZ
          bool
          default $(shell $CC -Werror -Oz -c -x c /dev/null -o /dev/null)

Result:

  $ make -s alldefconfig && tail -n 3 .config
  CONFIG_CC_IS_GCC=y
  # CONFIG_CC_IS_CLANG is not set
  # CONFIG_CC_HAS_OZ is not set

  $ make CC=clang -s alldefconfig && tail -n 3 .config
  # CONFIG_CC_IS_GCC is not set
  CONFIG_CC_IS_CLANG=y
  CONFIG_CC_HAS_OZ=y

A function call can appear anywhere a symbol reference can appear.
So, the following code is possible.

Example code:

  config CC_NAME
          string
          default "gcc" if $(shell $CC --version | grep -q gcc)
          default "clang" if $(shell $CC --version | grep -q clang)
          default "unknown compiler"

Result:

  $ make -s alldefconfig && tail -n 1 .config
  CONFIG_CC_NAME="gcc"

  $ make CC=clang -s alldefconfig && tail -n 1 .config
  CONFIG_CC_NAME="clang"

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Reminder for myself:
Update Documentation/kbuild/kconfig-language.txt


 scripts/kconfig/function.c  | 149 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/kconfig/lkc_proto.h |   5 ++
 scripts/kconfig/util.c      |  46 +++++++++++---
 scripts/kconfig/zconf.l     |  38 ++++++++++-
 scripts/kconfig/zconf.y     |   9 +++
 5 files changed, 238 insertions(+), 9 deletions(-)
 create mode 100644 scripts/kconfig/function.c

diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
new file mode 100644
index 0000000..60e59be
--- /dev/null
+++ b/scripts/kconfig/function.c
@@ -0,0 +1,149 @@
+// SPDX-License-Identifier: GPL-2.0
+//
+// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "list.h"
+
+#define FUNCTION_MAX_ARGS		10
+
+static LIST_HEAD(function_list);
+
+struct function {
+	const char *name;
+	char *(*func)(int argc, char *argv[]);
+	struct list_head node;
+};
+
+static struct function *func_lookup(const char *name)
+{
+	struct function *f;
+
+	list_for_each_entry(f, &function_list, node) {
+		if (!strcmp(name, f->name))
+			return f;
+	}
+
+	return NULL;
+}
+
+static void func_add(const char *name, char *(*func)(int argc, char *argv[]))
+{
+	struct function *f;
+
+	f = func_lookup(name);
+	if (f) {
+		fprintf(stderr, "%s: function already exists. ignored.\n", name);
+		return;
+	}
+
+	f = xmalloc(sizeof(*f));
+	f->name = name;
+	f->func = func;
+
+	list_add_tail(&f->node, &function_list);
+}
+
+static void func_del(struct function *f)
+{
+	list_del(&f->node);
+	free(f);
+}
+
+static char *func_call(int argc, char *argv[])
+{
+	struct function *f;
+
+	f = func_lookup(argv[0]);
+	if (!f) {
+		fprintf(stderr, "%s: function not found\n", argv[0]);
+		return NULL;
+	}
+
+	return f->func(argc, argv);
+}
+
+static char *func_eval(const char *func)
+{
+	char *expanded, *saveptr, *str, *token, *res;
+	const char *delim;
+	int argc = 0;
+	char *argv[FUNCTION_MAX_ARGS];
+
+	expanded = expand_string_value(func);
+
+	str = expanded;
+	delim = " ";
+
+	while ((token = strtok_r(str, delim, &saveptr))) {
+		argv[argc++] = token;
+		str = NULL;
+		delim = ",";
+	}
+
+	res = func_call(argc, argv);
+
+	free(expanded);
+
+	return res ?: xstrdup("");
+}
+
+char *func_eval_n(const char *func, size_t n)
+{
+	char *tmp, *res;
+
+	tmp = xmalloc(n + 1);
+	memcpy(tmp, func, n);
+	*(tmp + n) = '\0';
+
+	res = func_eval(tmp);
+
+	free(tmp);
+
+	return res;
+}
+
+/* built-in functions */
+static char *do_shell(int argc, char *argv[])
+{
+	static const char *pre = "(";
+	static const char *post = ") >/dev/null 2>&1";
+	char *cmd;
+	int ret;
+
+	if (argc != 2)
+		return NULL;
+
+	/*
+	 * Surround the command with ( ) in case it is piped commands.
+	 * Also, redirect stdout and stderr to /dev/null.
+	 */
+	cmd = xmalloc(strlen(pre) + strlen(argv[1]) + strlen(post) + 1);
+	strcpy(cmd, pre);
+	strcat(cmd, argv[1]);
+	strcat(cmd, post);
+
+	ret = system(cmd);
+
+	free(cmd);
+
+	return xstrdup(ret == 0 ? "y" : "n");
+}
+
+void func_init(void)
+{
+	/* register built-in functions */
+	func_add("shell", do_shell);
+}
+
+void func_exit(void)
+{
+	struct function *f, *tmp;
+
+	/* unregister all functions */
+	list_for_each_entry_safe(f, tmp, &function_list, node)
+		func_del(f);
+}
diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
index 9884adc..09a4f53 100644
--- a/scripts/kconfig/lkc_proto.h
+++ b/scripts/kconfig/lkc_proto.h
@@ -48,5 +48,10 @@ const char * sym_get_string_value(struct symbol *sym);
 
 const char * prop_get_type_name(enum prop_type type);
 
+/* function.c */
+char *func_eval_n(const char *func, size_t n);
+void func_init(void);
+void func_exit(void);
+
 /* expr.c */
 void expr_print(struct expr *e, void (*fn)(void *, struct symbol *, const char *), void *data, int prevtoken);
diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index dddf85b..ed89fb9 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -93,9 +93,10 @@ static char *env_expand_n(const char *name, size_t n)
 }
 
 /*
- * Expand environments embedded in the string given in argument. Environments
- * to be expanded shall be prefixed by a '$'. Unknown environment expands to
- * the empty string.
+ * Expand environments and functions embedded in the string given in argument.
+ * Environments to be expanded shall be prefixed by a '$'. Functions to be
+ * evaluated shall be surrounded by $(). Unknown environment/function expands
+ * to the empty string.
  */
 char *expand_string_value(const char *in)
 {
@@ -113,11 +114,40 @@ char *expand_string_value(const char *in)
 	while ((p = strchr(in, '$'))) {
 		char *new;
 
-		q = p + 1;
-		while (isalnum(*q) || *q == '_')
-			q++;
-
-		new = env_expand_n(p + 1, q - p - 1);
+		/*
+		 * If the next character is '(', it is a function.
+		 * Otherwise, environment.
+		 */
+		if (*(p + 1) == '(') {
+			int nest = 0;
+
+			q = p + 2;
+			while (1) {
+				if (*q == '\0') {
+					fprintf(stderr,
+						"unterminated function: %s\n",
+						p);
+					new = xstrdup("");
+					break;
+				} else if (*q == '(') {
+					nest++;
+				} else if (*q == ')') {
+					if (nest-- == 0) {
+						new = func_eval_n(p + 2,
+								  q - p - 2);
+						q++;
+						break;
+					}
+				}
+				q++;
+			}
+		} else {
+			q = p + 1;
+			while (isalnum(*q) || *q == '_')
+				q++;
+
+			new = env_expand_n(p + 1, q - p - 1);
+		}
 
 		reslen = strlen(res) + (p - in) + strlen(new) + 1;
 		res = xrealloc(res, reslen);
diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
index 0d89ea6..f433ab0 100644
--- a/scripts/kconfig/zconf.l
+++ b/scripts/kconfig/zconf.l
@@ -1,7 +1,7 @@
 %option nostdinit noyywrap never-interactive full ecs
 %option 8bit nodefault perf-report perf-report
 %option noinput
-%x COMMAND HELP STRING PARAM
+%x COMMAND HELP STRING PARAM FUNCTION
 %{
 /*
  * Copyright (C) 2002 Roman Zippel <zippel@linux-m68k.org>
@@ -25,6 +25,7 @@ static struct {
 
 static char *text;
 static int text_size, text_asize;
+static int function_nest;
 
 struct buffer {
 	struct buffer *parent;
@@ -138,6 +139,12 @@ n	[$A-Za-z0-9_-]
 		new_string();
 		BEGIN(STRING);
 	}
+	"$("	{
+		new_string();
+		append_string(yytext, yyleng);
+		function_nest = 0;
+		BEGIN(FUNCTION);
+	}
 	\n	BEGIN(INITIAL); current_file->lineno++; return T_EOL;
 	({n}|[/.])+	{
 		const struct kconf_id *id = kconf_id_lookup(yytext, yyleng);
@@ -196,6 +203,35 @@ n	[$A-Za-z0-9_-]
 	}
 }
 
+<FUNCTION>{
+	[^()\n]* {
+		append_string(yytext, yyleng);
+	}
+	"(" {
+		append_string(yytext, yyleng);
+		function_nest++;
+	}
+	")" {
+		append_string(yytext, yyleng);
+		if (function_nest-- == 0) {
+			BEGIN(PARAM);
+			yylval.string = text;
+			return T_WORD;
+		}
+	}
+	\n {
+		fprintf(stderr,
+			"%s:%d:warning: multi-line function not supported\n",
+			zconf_curname(), zconf_lineno());
+		current_file->lineno++;
+		BEGIN(INITIAL);
+		return T_EOL;
+	}
+	<<EOF>>	{
+		BEGIN(INITIAL);
+	}
+}
+
 <HELP>{
 	[ \t]+	{
 		ts = 0;
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index 784083d..d9977de 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -534,11 +534,19 @@ void conf_parse(const char *name)
 
 	zconf_initscan(name);
 
+	func_init();
 	_menu_init();
 
 	if (getenv("ZCONF_DEBUG"))
 		yydebug = 1;
 	yyparse();
+
+	/*
+	 * Currently, functions are evaluated only when Kconfig files are
+	 * parsed. We can free functions here.
+	 */
+	func_exit();
+
 	if (yynerrs)
 		exit(1);
 	if (!modules_sym)
@@ -778,4 +786,5 @@ void zconfdump(FILE *out)
 #include "confdata.c"
 #include "expr.c"
 #include "symbol.c"
+#include "function.c"
 #include "menu.c"
-- 
2.7.4

[PATCH 08/23] kconfig: add 'macro' keyword to support user-defined function

[Masahiro Yamada]

Now, we got a basic ability to test compiler capability in Kconfig.

config CC_HAS_STACKPROTECTOR
        bool
        default $(shell $CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null)

This works, but it is ugly to repeat this long boilerplate.

We want to describe like this:

config CC_HAS_STACKPROTECTOR
        bool
        default $(cc-option -fstack-protector)

It is straight-forward to implement a new function, but I do not like
to hard-code specialized functions like this.  Hence, here is another
feature to add functions from Kconfig files.

A user-defined function can be defined as a string type symbol with
a special keyword 'macro'.  It can be referenced in the same way as
built-in functions.  This feature was also inspired by Makefile where
user-defined functions are referenced by $(call func-name, args...),
but I omitted the 'call' to makes it shorter.

The macro definition can contain $(1), $(2), ... which will be replaced
with arguments from the caller.

Example code:

  config cc-option
          string
          macro $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)

  config CC_HAS_STACKPROTECTOR
          bool
          default $(cc-option -fstack-protector)

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Reminder for myself:
Update Documentation/kbuild/kconfig-language.txt

 scripts/kconfig/function.c  | 66 +++++++++++++++++++++++++++++++++++++++++----
 scripts/kconfig/kconf_id.c  |  1 +
 scripts/kconfig/lkc_proto.h |  1 +
 scripts/kconfig/zconf.y     |  8 ++++++
 4 files changed, 71 insertions(+), 5 deletions(-)

diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
index 60e59be..f7f154d 100644
--- a/scripts/kconfig/function.c
+++ b/scripts/kconfig/function.c
@@ -14,7 +14,8 @@ static LIST_HEAD(function_list);
 
 struct function {
 	const char *name;
-	char *(*func)(int argc, char *argv[]);
+	char *(*func)(struct function *f, int argc, char *argv[]);
+	void *priv;
 	struct list_head node;
 };
 
@@ -30,7 +31,9 @@ static struct function *func_lookup(const char *name)
 	return NULL;
 }
 
-static void func_add(const char *name, char *(*func)(int argc, char *argv[]))
+static void func_add(const char *name,
+		     char *(*func)(struct function *f, int argc, char *argv[]),
+		     void *priv)
 {
 	struct function *f;
 
@@ -43,6 +46,7 @@ static void func_add(const char *name, char *(*func)(int argc, char *argv[]))
 	f = xmalloc(sizeof(*f));
 	f->name = name;
 	f->func = func;
+	f->priv = priv;
 
 	list_add_tail(&f->node, &function_list);
 }
@@ -50,6 +54,7 @@ static void func_add(const char *name, char *(*func)(int argc, char *argv[]))
 static void func_del(struct function *f)
 {
 	list_del(&f->node);
+	free(f->priv);
 	free(f);
 }
 
@@ -63,7 +68,7 @@ static char *func_call(int argc, char *argv[])
 		return NULL;
 	}
 
-	return f->func(argc, argv);
+	return f->func(f, argc, argv);
 }
 
 static char *func_eval(const char *func)
@@ -106,8 +111,59 @@ char *func_eval_n(const char *func, size_t n)
 	return res;
 }
 
+/* run user-defined function */
+static char *do_macro(struct function *f, int argc, char *argv[])
+{
+	char *new;
+	char *src, *p, *res;
+	size_t newlen;
+	int n;
+
+	new = xmalloc(1);
+	*new = 0;
+
+	/*
+	 * This is a format string. $(1), $(2), ... must be replaced with
+	 * function arguments.
+	 */
+	src = f->priv;
+	p = src;
+
+	while ((p = strstr(p, "$("))) {
+		if (isdigit(p[2]) && p[3] == ')') {
+			n = p[2] - '0';
+			if (n < argc) {
+				newlen = strlen(new) + (p - src) +
+							strlen(argv[n]) + 1;
+				new = xrealloc(new, newlen);
+				strncat(new, src, p - src);
+				strcat(new, argv[n]);
+				src = p + 4;
+			}
+			p += 2;
+		}
+		p += 2;
+	}
+
+	newlen = strlen(new) + strlen(src) + 1;
+	new = xrealloc(new, newlen);
+	strcat(new, src);
+
+	res = expand_string_value(new);
+
+	free(new);
+
+	return res;
+}
+
+/* add user-defined function (macro) */
+void func_add_macro(const char *name, char *macro)
+{
+	func_add(name, do_macro, macro);
+}
+
 /* built-in functions */
-static char *do_shell(int argc, char *argv[])
+static char *do_shell(struct function *f, int argc, char *argv[])
 {
 	static const char *pre = "(";
 	static const char *post = ") >/dev/null 2>&1";
@@ -136,7 +192,7 @@ static char *do_shell(int argc, char *argv[])
 void func_init(void)
 {
 	/* register built-in functions */
-	func_add("shell", do_shell);
+	func_add("shell", do_shell, NULL);
 }
 
 void func_exit(void)
diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
index b3e0ea0..5a1357d 100644
--- a/scripts/kconfig/kconf_id.c
+++ b/scripts/kconfig/kconf_id.c
@@ -28,6 +28,7 @@ static struct kconf_id kconf_id_array[] = {
 	{ "imply",		T_IMPLY,		TF_COMMAND },
 	{ "range",		T_RANGE,		TF_COMMAND },
 	{ "visible",		T_VISIBLE,		TF_COMMAND },
+	{ "macro",		T_MACRO,		TF_COMMAND },
 	{ "option",		T_OPTION,		TF_COMMAND },
 	{ "on",			T_ON,			TF_PARAM },
 	{ "modules",		T_OPT_MODULES,		TF_OPTION },
diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
index 09a4f53..25caca3 100644
--- a/scripts/kconfig/lkc_proto.h
+++ b/scripts/kconfig/lkc_proto.h
@@ -50,6 +50,7 @@ const char * prop_get_type_name(enum prop_type type);
 
 /* function.c */
 char *func_eval_n(const char *func, size_t n);
+void func_add_macro(const char *name, char *macro);
 void func_init(void);
 void func_exit(void);
 
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index d9977de..19452b6 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -65,6 +65,7 @@ static struct menu *current_menu, *current_entry;
 %token <id>T_IMPLY
 %token <id>T_RANGE
 %token <id>T_VISIBLE
+%token <id>T_MACRO
 %token <id>T_OPTION
 %token <id>T_ON
 %token <string> T_WORD
@@ -199,6 +200,7 @@ config_option_list:
 	| config_option_list config_option
 	| config_option_list symbol_option
 	| config_option_list depends
+	| config_option_list macro
 	| config_option_list help
 	| config_option_list option_error
 	| config_option_list T_EOL
@@ -246,6 +248,12 @@ config_option: T_RANGE symbol symbol if_expr T_EOL
 	printd(DEBUG_PARSE, "%s:%d:range\n", zconf_curname(), zconf_lineno());
 };
 
+macro: T_MACRO T_WORD T_EOL
+{
+	current_entry->sym->flags |= SYMBOL_AUTO;
+	func_add_macro(current_entry->sym->name, $2);
+}
+
 symbol_option: T_OPTION symbol_option_list T_EOL
 ;
 
-- 
2.7.4

[PATCH 09/23] kconfig: add 'cc-option' macro

[Masahiro Yamada]

This will be the most frequently used macro.  It evaluates to 'y' if
the given argument is supported by the compiler, or 'n' otherwise.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 init/Kconfig | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/init/Kconfig b/init/Kconfig
index b4814e6..f026a62 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -8,6 +8,10 @@ config DEFCONFIG_LIST
 	default ARCH_DEFCONFIG
 	default "arch/$ARCH/defconfig"
 
+config cc-option
+	string
+	macro $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
+
 config CONSTRUCTORS
 	bool
 	depends on !UML
-- 
2.7.4

[PATCH 10/23] stack-protector: test compiler capability in Kconfig and drop AUTO mode

[Masahiro Yamada]

Add CC_HAS_STACKPROTECTOR(_STRONG) to test if the compiler supports
-fstack-protector(-strong) option.

X86 has additional shell scripts in case the compiler supports the
option, but generates broken code.  I added CC_HAS_SANE_STACKPROTECTOR
to test this.  I had to add -m32 to gcc-x86_32-has-stack-protector.sh
to make it work correctly.

If the compiler does not support the option, the menu is automatically
hidden.  If _STRONG is not supported, it will fall back to _REGULAR.
This means, _AUTO is implicitly supported in the dependency solver of
Kconfig, hence removed.

I also turned the 'choice' into only two boolean symbols.  The use of
'choice' is not a good idea here, because all of all{yes,mod,no}config
would choose the first visible value, while we want allnoconfig to
disable as many features as possible.

I did not add CC_HAS_STACKPROTECTOR_NONE in the hope that GCC versions
we support will recognize -fno-stack-protector.

If this turns out to be a problem, it will be possible to do this:

stackp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector
stackp-flags-$(CONFIG_CC_STACKPROTECTOR)          := -fstack-protector
stackp-flags-$(CONFIG_CC_STACKPROTECTOR_STRONG)   := -fstack-protector-strong

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 Makefile                                  | 93 ++-----------------------------
 arch/Kconfig                              | 37 ++++++------
 arch/x86/Kconfig                          |  8 ++-
 scripts/gcc-x86_32-has-stack-protector.sh |  7 +--
 scripts/gcc-x86_64-has-stack-protector.sh |  5 --
 5 files changed, 30 insertions(+), 120 deletions(-)

diff --git a/Makefile b/Makefile
index 9a8c689..e9fc7c9 100644
--- a/Makefile
+++ b/Makefile
@@ -675,55 +675,11 @@ ifneq ($(CONFIG_FRAME_WARN),0)
 KBUILD_CFLAGS += $(call cc-option,-Wframe-larger-than=${CONFIG_FRAME_WARN})
 endif
 
-# This selects the stack protector compiler flag. Testing it is delayed
-# until after .config has been reprocessed, in the prepare-compiler-check
-# target.
-ifdef CONFIG_CC_STACKPROTECTOR_AUTO
-  stackp-flag := $(call cc-option,-fstack-protector-strong,$(call cc-option,-fstack-protector))
-  stackp-name := AUTO
-else
-ifdef CONFIG_CC_STACKPROTECTOR_REGULAR
-  stackp-flag := -fstack-protector
-  stackp-name := REGULAR
-else
-ifdef CONFIG_CC_STACKPROTECTOR_STRONG
-  stackp-flag := -fstack-protector-strong
-  stackp-name := STRONG
-else
-  # If either there is no stack protector for this architecture or
-  # CONFIG_CC_STACKPROTECTOR_NONE is selected, we're done, and $(stackp-name)
-  # is empty, skipping all remaining stack protector tests.
-  #
-  # Force off for distro compilers that enable stack protector by default.
-  KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
-endif
-endif
-endif
-# Find arch-specific stack protector compiler sanity-checking script.
-ifdef stackp-name
-ifneq ($(stackp-flag),)
-  stackp-path := $(srctree)/scripts/gcc-$(SRCARCH)_$(BITS)-has-stack-protector.sh
-  stackp-check := $(wildcard $(stackp-path))
-  # If the wildcard test matches a test script, run it to check functionality.
-  ifdef stackp-check
-    ifneq ($(shell $(CONFIG_SHELL) $(stackp-check) $(CC) $(KBUILD_CPPFLAGS) $(biarch)),y)
-      stackp-broken := y
-    endif
-  endif
-  ifndef stackp-broken
-    # If the stack protector is functional, enable code that depends on it.
-    KBUILD_CPPFLAGS += -DCONFIG_CC_STACKPROTECTOR
-    # Either we've already detected the flag (for AUTO) or we'll fail the
-    # build in the prepare-compiler-check rule (for specific flag).
-    KBUILD_CFLAGS += $(stackp-flag)
-  else
-    # We have to make sure stack protector is unconditionally disabled if
-    # the compiler is broken (in case we're going to continue the build in
-    # AUTO mode).
-    KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
-  endif
-endif
-endif
+stackp-flags-y					:= -fno-stack-protector
+stackp-flags-$(CONFIG_CC_STACKPROTECTOR)	:= -fstack-protector
+stackp-flags-$(CONFIG_CC_STACKPROTECTOR_STRONG)	:= -fstack-protector-strong
+
+KBUILD_CFLAGS += $(stackp-flags-y)
 
 ifeq ($(cc-name),clang)
 KBUILD_CPPFLAGS += $(call cc-option,-Qunused-arguments,)
@@ -1079,7 +1035,7 @@ endif
 # prepare2 creates a makefile if using a separate output directory.
 # From this point forward, .config has been reprocessed, so any rules
 # that need to depend on updated CONFIG_* values can be checked here.
-prepare2: prepare3 prepare-compiler-check outputmakefile asm-generic
+prepare2: prepare3 outputmakefile asm-generic
 
 prepare1: prepare2 $(version_h) include/generated/utsrelease.h \
                    include/config/auto.conf
@@ -1105,43 +1061,6 @@ uapi-asm-generic:
 PHONY += prepare-objtool
 prepare-objtool: $(objtool_target)
 
-# Check for CONFIG flags that require compiler support. Abort the build
-# after .config has been processed, but before the kernel build starts.
-#
-# For security-sensitive CONFIG options, we don't want to fallback and/or
-# silently change which compiler flags will be used, since that leads to
-# producing kernels with different security feature characteristics
-# depending on the compiler used. (For example, "But I selected
-# CC_STACKPROTECTOR_STRONG! Why did it build with _REGULAR?!")
-PHONY += prepare-compiler-check
-prepare-compiler-check: FORCE
-# Make sure compiler supports requested stack protector flag.
-ifdef stackp-name
-  # Warn about CONFIG_CC_STACKPROTECTOR_AUTO having found no option.
-  ifeq ($(stackp-flag),)
-	@echo CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-		  Compiler does not support any known stack-protector >&2
-  else
-  # Fail if specifically requested stack protector is missing.
-  ifeq ($(call cc-option, $(stackp-flag)),)
-	@echo Cannot use CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-		  $(stackp-flag) not supported by compiler >&2 && exit 1
-  endif
-  endif
-endif
-# Make sure compiler does not have buggy stack-protector support. If a
-# specific stack-protector was requested, fail the build, otherwise warn.
-ifdef stackp-broken
-  ifeq ($(stackp-name),AUTO)
-	@echo CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-                  $(stackp-flag) available but compiler is broken: disabling >&2
-  else
-	@echo Cannot use CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
-                  $(stackp-flag) available but compiler is broken >&2 && exit 1
-  endif
-endif
-	@:
-
 # Generate some files
 # ---------------------------------------------------------------------------
 
diff --git a/arch/Kconfig b/arch/Kconfig
index 76c0b54..9b7a628 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -535,13 +535,21 @@ config HAVE_CC_STACKPROTECTOR
 	bool
 	help
 	  An arch should select this symbol if:
-	  - its compiler supports the -fstack-protector option
 	  - it has implemented a stack canary (e.g. __stack_chk_guard)
 
-choice
-	prompt "Stack Protector buffer overflow detection"
+config CC_HAS_STACKPROTECTOR
+	bool
+	default $(cc-option -fstack-protector)
+
+config CC_HAS_STACKPROTECTOR_STRONG
+	bool
+	default $(cc-option -fstack-protector-strong)
+
+config CC_STACKPROTECTOR
+	bool "Stack Protector buffer overflow detection"
 	depends on HAVE_CC_STACKPROTECTOR
-	default CC_STACKPROTECTOR_AUTO
+	depends on CC_HAS_STACKPROTECTOR
+	default y
 	help
 	  This option turns on the "stack-protector" GCC feature. This
 	  feature puts, at the beginning of functions, a canary value on
@@ -551,14 +559,6 @@ choice
 	  overwrite the canary, which gets detected and the attack is then
 	  neutralized via a kernel panic.
 
-config CC_STACKPROTECTOR_NONE
-	bool "None"
-	help
-	  Disable "stack-protector" GCC feature.
-
-config CC_STACKPROTECTOR_REGULAR
-	bool "Regular"
-	help
 	  Functions will have the stack-protector canary logic added if they
 	  have an 8-byte or larger character array on the stack.
 
@@ -570,7 +570,10 @@ config CC_STACKPROTECTOR_REGULAR
 	  by about 0.3%.
 
 config CC_STACKPROTECTOR_STRONG
-	bool "Strong"
+	bool "Strong Stack Protector"
+	depends on CC_HAS_STACKPROTECTOR_STRONG
+	depends on CC_STACKPROTECTOR
+	default y
 	help
 	  Functions will have the stack-protector canary logic added in any
 	  of the following conditions:
@@ -588,14 +591,6 @@ config CC_STACKPROTECTOR_STRONG
 	  about 20% of all kernel functions, which increases the kernel code
 	  size by about 2%.
 
-config CC_STACKPROTECTOR_AUTO
-	bool "Automatic"
-	help
-	  If the compiler supports it, the best available stack-protector
-	  option will be chosen.
-
-endchoice
-
 config THIN_ARCHIVES
 	def_bool y
 	help
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 54d943a..498694d 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -124,7 +124,6 @@ config X86
 	select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64
 	select HAVE_ARCH_VMAP_STACK		if X86_64
 	select HAVE_ARCH_WITHIN_STACK_FRAMES
-	select HAVE_CC_STACKPROTECTOR
 	select HAVE_CMPXCHG_DOUBLE
 	select HAVE_CMPXCHG_LOCAL
 	select HAVE_CONTEXT_TRACKING		if X86_64
@@ -340,6 +339,13 @@ config PGTABLE_LEVELS
 	default 2
 
 source "init/Kconfig"
+
+config CC_HAS_SANE_STACKPROTECTOR
+	bool
+	default $(shell $srctree/scripts/gcc-x86_64-has-stack-protector.sh $CC) if 64BIT
+	default $(shell $srctree/scripts/gcc-x86_32-has-stack-protector.sh $CC)
+	select HAVE_CC_STACKPROTECTOR
+
 source "kernel/Kconfig.freezer"
 
 menu "Processor type and features"
diff --git a/scripts/gcc-x86_32-has-stack-protector.sh b/scripts/gcc-x86_32-has-stack-protector.sh
index 6b2aeef..f5c1194 100755
--- a/scripts/gcc-x86_32-has-stack-protector.sh
+++ b/scripts/gcc-x86_32-has-stack-protector.sh
@@ -1,9 +1,4 @@
 #!/bin/sh
 # SPDX-License-Identifier: GPL-2.0
 
-echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -O0 -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
-if [ "$?" -eq "0" ] ; then
-	echo y
-else
-	echo n
-fi
+echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -m32 -O0 -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
diff --git a/scripts/gcc-x86_64-has-stack-protector.sh b/scripts/gcc-x86_64-has-stack-protector.sh
index 4a48bdc..3755af0 100755
--- a/scripts/gcc-x86_64-has-stack-protector.sh
+++ b/scripts/gcc-x86_64-has-stack-protector.sh
@@ -2,8 +2,3 @@
 # SPDX-License-Identifier: GPL-2.0
 
 echo "int foo(void) { char X[200]; return 3; }" | $* -S -x c -c -O0 -mcmodel=kernel -fno-PIE -fstack-protector - -o - 2> /dev/null | grep -q "%gs"
-if [ "$?" -eq "0" ] ; then
-	echo y
-else
-	echo n
-fi
-- 
2.7.4

[PATCH 11/23] kconfig: add 'shell-stdout' function

[Masahiro Yamada]

This is the second built-in function, which retrieves the first line
of stdout from the given shell command.

Example code:

  config CC_IS_GCC
          bool
          default $(shell $CC --version | grep -q gcc)

  config GCC_VERSION
          int
          default $(shell-stdout $srctree/scripts/gcc-version.sh $CC | sed 's/^0*//') if CC_IS_GCC
          default 0

Result:

  $ make -s alldefconfig && tail -n 2 .config
  CONFIG_CC_IS_GCC=y
  CONFIG_GCC_VERSION=504

  $ make CC=clang -s alldefconfig && tail -n 2 .config
  # CONFIG_CC_IS_GCC is not set
  CONFIG_GCC_VERSION=0

By the way, function calls can be nested, so the following works.

Example code:

  config FOO
          bool
          default $(shell $(shell-stdout echo $COMMAND_IN_CAPITAL | tr [A-Z] [a-z]))

Result:
  $ make -s COMMAND=TRUE alldefconfig && tail -n 1 .config
  CONFIG_FOO=y
  $ make -s COMMAND=FALSE alldefconfig && tail -n 1 .config
  # CONFIG_FOO is not set

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 scripts/kconfig/function.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
index f7f154d..266f4ec 100644
--- a/scripts/kconfig/function.c
+++ b/scripts/kconfig/function.c
@@ -189,10 +189,56 @@ static char *do_shell(struct function *f, int argc, char *argv[])
 	return xstrdup(ret == 0 ? "y" : "n");
 }
 
+static char *do_shell_stdout(struct function *f, int argc, char *argv[])
+{
+	static const char *pre = "(";
+	static const char *post = ") 2>/dev/null";
+	FILE *p;
+	char buf[256];
+	char *cmd;
+	int ret;
+
+	if (argc != 2)
+		return NULL;
+
+	/*
+	 * Surround the command with ( ) in case it is piped commands.
+	 * Also, redirect stderr to /dev/null.
+	 */
+	cmd = xmalloc(strlen(pre) + strlen(argv[1]) + strlen(post) + 1);
+	strcpy(cmd, pre);
+	strcat(cmd, argv[1]);
+	strcat(cmd, post);
+
+	p = popen(cmd, "r");
+	if (!p) {
+		perror(cmd);
+		goto free;
+	}
+	if (fgets(buf, sizeof(buf), p)) {
+		size_t len = strlen(buf);
+
+		if (buf[len - 1] == '\n')
+			buf[len - 1] = '\0';
+	} else {
+		buf[0] = '\0';
+	}
+
+	ret = pclose(p);
+	if (ret == -1)
+		perror(cmd);
+
+free:
+	free(cmd);
+
+	return xstrdup(buf);
+}
+
 void func_init(void)
 {
 	/* register built-in functions */
 	func_add("shell", do_shell, NULL);
+	func_add("shell-stdout", do_shell_stdout, NULL);
 }
 
 void func_exit(void)
-- 
2.7.4

[PATCH 12/23] kconfig: replace $UNAME_RELEASE with function call

[Masahiro Yamada]

With 'shell-stdout' supported, we can make it self-contained in
Kconfig.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 Makefile     | 3 +--
 init/Kconfig | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index e9fc7c9..6491404 100644
--- a/Makefile
+++ b/Makefile
@@ -275,8 +275,7 @@ include scripts/Kbuild.include
 # Read KERNELRELEASE from include/config/kernel.release (if it exists)
 KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
 KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
-UNAME_RELEASE := $(shell uname --release)
-export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
+export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
 
 # SUBARCH tells the usermode build what the underlying arch is.  That is set
 # first, and if a usermode build is happening, the "ARCH=um" on the command
diff --git a/init/Kconfig b/init/Kconfig
index f026a62..837a584 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2,9 +2,9 @@ config DEFCONFIG_LIST
 	string
 	depends on !UML
 	option defconfig_list
-	default "/lib/modules/$UNAME_RELEASE/.config"
+	default "/lib/modules/$(shell-stdout uname --release)/.config"
 	default "/etc/kernel-config"
-	default "/boot/config-$UNAME_RELEASE"
+	default "/boot/config-$(shell-stdout uname --release)"
 	default ARCH_DEFCONFIG
 	default "arch/$ARCH/defconfig"
 
-- 
2.7.4

[PATCH 13/23] kconfig: expand environments/functions in (main)menu, comment, prompt

[Masahiro Yamada]

Expand the prompt passed to menu_add_prompt().  This affects
'mainmenu', 'menu', 'prompt', 'comment'.

Another good thing is, I am fixing the memory leak for the case
without mainmenu.  The 'mainmenu' should be independent of user
configuration.  So, its prompt can be expanded in the first parse
phase.  The ugly hack for no_mainmenu_stmt is gone.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 scripts/kconfig/menu.c  |  2 +-
 scripts/kconfig/zconf.y | 30 +++++++++++-------------------
 2 files changed, 12 insertions(+), 20 deletions(-)

diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
index a9d0ccc..462b995 100644
--- a/scripts/kconfig/menu.c
+++ b/scripts/kconfig/menu.c
@@ -176,7 +176,7 @@ static struct property *menu_add_prop(enum prop_type type, char *prompt, struct
 
 struct property *menu_add_prompt(enum prop_type type, char *prompt, struct expr *dep)
 {
-	return menu_add_prop(type, prompt, NULL, dep);
+	return menu_add_prop(type, expand_string_value(prompt), NULL, dep);
 }
 
 void menu_add_visibility(struct expr *expr)
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index 19452b6..ef006be 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -110,28 +110,16 @@ static struct menu *current_menu, *current_entry;
 %%
 input: nl start | start;
 
-start: mainmenu_stmt stmt_list | no_mainmenu_stmt stmt_list;
+start: mainmenu_stmt stmt_list | stmt_list;
 
 /* mainmenu entry */
 
 mainmenu_stmt: T_MAINMENU prompt nl
 {
 	menu_add_prompt(P_MENU, $2, NULL);
+	free($2);
 };
 
-/* Default main menu, if there's no mainmenu entry */
-
-no_mainmenu_stmt: /* empty */
-{
-	/*
-	 * Hack: Keep the main menu title on the heap so we can safely free it
-	 * later regardless of whether it comes from the 'prompt' in
-	 * mainmenu_stmt or here
-	 */
-	menu_add_prompt(P_MENU, xstrdup("Linux Kernel Configuration"), NULL);
-};
-
-
 stmt_list:
 	  /* empty */
 	| stmt_list common_stmt
@@ -217,6 +205,7 @@ config_option: T_TYPE prompt_stmt_opt T_EOL
 config_option: T_PROMPT prompt if_expr T_EOL
 {
 	menu_add_prompt(P_PROMPT, $2, $3);
+	free($2);
 	printd(DEBUG_PARSE, "%s:%d:prompt\n", zconf_curname(), zconf_lineno());
 };
 
@@ -316,6 +305,7 @@ choice_option_list:
 choice_option: T_PROMPT prompt if_expr T_EOL
 {
 	menu_add_prompt(P_PROMPT, $2, $3);
+	free($2);
 	printd(DEBUG_PARSE, "%s:%d:prompt\n", zconf_curname(), zconf_lineno());
 };
 
@@ -385,6 +375,7 @@ menu: T_MENU prompt T_EOL
 {
 	menu_add_entry(NULL);
 	menu_add_prompt(P_MENU, $2, NULL);
+	free($2);
 	printd(DEBUG_PARSE, "%s:%d:menu\n", zconf_curname(), zconf_lineno());
 };
 
@@ -424,6 +415,7 @@ comment: T_COMMENT prompt T_EOL
 {
 	menu_add_entry(NULL);
 	menu_add_prompt(P_COMMENT, $2, NULL);
+	free($2);
 	printd(DEBUG_PARSE, "%s:%d:comment\n", zconf_curname(), zconf_lineno());
 };
 
@@ -489,6 +481,7 @@ prompt_stmt_opt:
 	| prompt if_expr
 {
 	menu_add_prompt(P_PROMPT, $1, $2);
+	free($1);
 };
 
 prompt:	  T_WORD
@@ -536,7 +529,6 @@ word_opt: /* empty */			{ $$ = NULL; }
 
 void conf_parse(const char *name)
 {
-	const char *tmp;
 	struct symbol *sym;
 	int i;
 
@@ -560,10 +552,10 @@ void conf_parse(const char *name)
 	if (!modules_sym)
 		modules_sym = sym_find( "n" );
 
-	tmp = rootmenu.prompt->text;
-	rootmenu.prompt->text = _(rootmenu.prompt->text);
-	rootmenu.prompt->text = expand_string_value(rootmenu.prompt->text);
-	free((char*)tmp);
+	if (!menu_has_prompt(&rootmenu)) {
+		current_entry = &rootmenu;
+		menu_add_prompt(P_MENU, "Linux Kernel Configuration", NULL);
+	}
 
 	menu_finalize(&rootmenu);
 	for_all_symbols(i, sym) {
-- 
2.7.4

[PATCH 14/23] kconfig: show compiler version text in the top comment

[Masahiro Yamada]

The kernel configuration phase is now tightly coupled with the compiler
in use.  It will be nice to show the compiler information in Kconfig.

The compiler information will be displayed like this:

  $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- config
  scripts/kconfig/conf  --oldaskconfig Kconfig
  *
  * Linux/arm64 4.16.0-rc1 Kernel Configuration
  *
  *
  * Compiler: aarch64-linux-gnu-gcc (Linaro GCC 7.2-2017.11) 7.2.1 20171011
  *
  *
  * General setup
  *
  Compile also drivers which will not load (COMPILE_TEST) [N/y/?]

If you use GUI methods such as menuconfig, it will be displayed in the
top menu.

This is simply implemented by using 'comment'.  So, it will be saved
into the .config file as well.

This commit has a very important meaning.  If the compile is upgraded,
Kconfig must be re-run since different compilers have different sets
of supported options.

All referenced environments are written to include/config/auto.conf.cmd
so that any environment change triggers silentoldconfig, and prompt
the user to input new values if needed.

With this commit, something like follows will be added to
include/config/auto.conf.cmd

  ifneq "$(CC_VERSION_TEXT)" "aarch64-linux-gnu-gcc (Linaro GCC 7.2-2017.11) 7.2.1 20171011"
  include/config/auto.conf: FORCE
  endif

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 Kconfig  | 2 ++
 Makefile | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/Kconfig b/Kconfig
index e6ece5b..99ed4b8 100644
--- a/Kconfig
+++ b/Kconfig
@@ -5,4 +5,6 @@
 #
 mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
 
+comment "Compiler: $CC_VERSION_TEXT"
+
 source "arch/$SRCARCH/Kconfig"
diff --git a/Makefile b/Makefile
index 6491404..b8df04b 100644
--- a/Makefile
+++ b/Makefile
@@ -438,6 +438,8 @@ export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_LDFLAGS_MODULE
 export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL
 export KBUILD_ARFLAGS
 
+export CC_VERSION_TEXT := $(shell $(CC) --version | head -n 1)
+
 # When compiling out-of-tree modules, put MODVERDIR in the module
 # tree rather than in the kernel tree. The kernel tree might
 # even be read-only.
-- 
2.7.4

[PATCH 15/23] kconfig: add CC_IS_GCC and GCC_VERSION

[Masahiro Yamada]

This will be useful to specify the required compiler version,
like this:

config FOO
        bool "Use Foo"
        depends on GCC_VERSION >= 408000
        help
          This feature requires GCC 4.8 or newer.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 init/Kconfig | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/init/Kconfig b/init/Kconfig
index 837a584..f2da5e9 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -12,6 +12,15 @@ config cc-option
 	string
 	macro $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
 
+config CC_IS_GCC
+	bool
+	default $(shell $CC --version | grep -q gcc)
+
+config GCC_VERSION
+	int
+	default $(shell-stdout $srctree/scripts/gcc-version.sh -p $CC | sed 's/^0*//') if CC_IS_GCC
+	default 0
+
 config CONSTRUCTORS
 	bool
 	depends on !UML
-- 
2.7.4

[PATCH 16/23] kbuild: add clang-version.sh

[Masahiro Yamada]

From: Sami Tolvanen <samitolvanen@google.com>

Based on gcc-version.sh, clang-version.sh prints out the correct
version of clang.

Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Tested-by: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 scripts/clang-version.sh | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100755 scripts/clang-version.sh

diff --git a/scripts/clang-version.sh b/scripts/clang-version.sh
new file mode 100755
index 0000000..9780efa
--- /dev/null
+++ b/scripts/clang-version.sh
@@ -0,0 +1,33 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+#
+# clang-version [-p] clang-command
+#
+# Prints the compiler version of `clang-command' in a canonical 4-digit form
+# such as `0500' for clang-5.0 etc.
+#
+# With the -p option, prints the patchlevel as well, for example `050001' for
+# clang-5.0.1 etc.
+#
+
+if [ "$1" = "-p" ] ; then
+	with_patchlevel=1;
+	shift;
+fi
+
+compiler="$*"
+
+if [ ${#compiler} -eq 0 ]; then
+	echo "Error: No compiler specified."
+	printf "Usage:\n\t$0 <clang-command>\n"
+	exit 1
+fi
+
+MAJOR=$(echo __clang_major__ | $compiler -E -x c - | tail -n 1)
+MINOR=$(echo __clang_minor__ | $compiler -E -x c - | tail -n 1)
+if [ "x$with_patchlevel" != "x" ] ; then
+	PATCHLEVEL=$(echo __clang_patchlevel__ | $compiler -E -x c - | tail -n 1)
+	printf "%02d%02d%02d\\n" $MAJOR $MINOR $PATCHLEVEL
+else
+	printf "%02d%02d\\n" $MAJOR $MINOR
+fi
-- 
2.7.4

[PATCH 17/23] kconfig: add CC_IS_CLANG and CLANG_VERSION

[Masahiro Yamada]

This will be useful to describe the clang version dependency.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 init/Kconfig             |  8 ++++++++
 scripts/clang-version.sh | 24 +++++++++++-------------
 2 files changed, 19 insertions(+), 13 deletions(-)

diff --git a/init/Kconfig b/init/Kconfig
index f2da5e9..1b751bc 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -21,6 +21,14 @@ config GCC_VERSION
 	default $(shell-stdout $srctree/scripts/gcc-version.sh -p $CC | sed 's/^0*//') if CC_IS_GCC
 	default 0
 
+config CC_IS_CLANG
+	bool
+	default $(shell $CC --version | grep -q clang)
+
+config CLANG_VERSION
+	int
+	default $(shell-stdout $srctree/scripts/clang-version.sh $CC)
+
 config CONSTRUCTORS
 	bool
 	depends on !UML
diff --git a/scripts/clang-version.sh b/scripts/clang-version.sh
index 9780efa..d8002d2 100755
--- a/scripts/clang-version.sh
+++ b/scripts/clang-version.sh
@@ -10,24 +10,22 @@
 # clang-5.0.1 etc.
 #
 
-if [ "$1" = "-p" ] ; then
-	with_patchlevel=1;
-	shift;
-fi
-
 compiler="$*"
 
 if [ ${#compiler} -eq 0 ]; then
-	echo "Error: No compiler specified."
-	printf "Usage:\n\t$0 <clang-command>\n"
+	echo "Error: No compiler specified." >&2
+	printf "Usage:\n\t$0 <clang-command>\n" >&2
+	echo 0
+	exit 1
+fi
+
+if !( $compiler --version | grep -q clang) ; then
+	echo "This is not clang." >&2
+	echo 0
 	exit 1
 fi
 
 MAJOR=$(echo __clang_major__ | $compiler -E -x c - | tail -n 1)
 MINOR=$(echo __clang_minor__ | $compiler -E -x c - | tail -n 1)
-if [ "x$with_patchlevel" != "x" ] ; then
-	PATCHLEVEL=$(echo __clang_patchlevel__ | $compiler -E -x c - | tail -n 1)
-	printf "%02d%02d%02d\\n" $MAJOR $MINOR $PATCHLEVEL
-else
-	printf "%02d%02d\\n" $MAJOR $MINOR
-fi
+PATCHLEVEL=$(echo __clang_patchlevel__ | $compiler -E -x c - | tail -n 1)
+printf "%d%02d%02d\\n" $MAJOR $MINOR $PATCHLEVEL
-- 
2.7.4

[PATCH 18/23] gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT

[Masahiro Yamada]

CONFIG_GCOV_FORMAT_AUTODETECT compiles either gcc_3_4.c or gcc_4_7.c
according to your GCC version.

We can achieve the equivalent behavior by setting reasonable default
in the 'choice' with the knowledge of the compiler version.

Users are still allowed to change it if they need to do so.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 kernel/gcov/Kconfig  | 18 ++++++------------
 kernel/gcov/Makefile |  2 --
 2 files changed, 6 insertions(+), 14 deletions(-)

diff --git a/kernel/gcov/Kconfig b/kernel/gcov/Kconfig
index 1276aab..c56bee7 100644
--- a/kernel/gcov/Kconfig
+++ b/kernel/gcov/Kconfig
@@ -53,20 +53,14 @@ config GCOV_PROFILE_ALL
 choice
 	prompt "Specify GCOV format"
 	depends on GCOV_KERNEL
-	default GCOV_FORMAT_AUTODETECT
+	default GCOV_FORMAT_3_4 if CC_IS_GCC && GCC_VERSION < 40700
+	default GCOV_FORMAT_4_7
 	---help---
-	The gcov format is usually determined by the GCC version, but there are
+	The gcov format is usually determined by the GCC version, and the
+	default is chosen according to your GCC version. However, there are
 	exceptions where format changes are integrated in lower-version GCCs.
-	In such a case use this option to adjust the format used in the kernel
-	accordingly.
-
-	If unsure, choose "Autodetect".
-
-config GCOV_FORMAT_AUTODETECT
-	bool "Autodetect"
-	---help---
-	Select this option to use the format that corresponds to your GCC
-	version.
+	In such a case, change this option to adjust the format used in the
+	kernel accordingly.
 
 config GCOV_FORMAT_3_4
 	bool "GCC 3.4 format"
diff --git a/kernel/gcov/Makefile b/kernel/gcov/Makefile
index c6c50e5..ff06d64 100644
--- a/kernel/gcov/Makefile
+++ b/kernel/gcov/Makefile
@@ -4,5 +4,3 @@ ccflags-y := -DSRCTREE='"$(srctree)"' -DOBJTREE='"$(objtree)"'
 obj-y := base.o fs.o
 obj-$(CONFIG_GCOV_FORMAT_3_4) += gcc_3_4.o
 obj-$(CONFIG_GCOV_FORMAT_4_7) += gcc_4_7.o
-obj-$(CONFIG_GCOV_FORMAT_AUTODETECT) += $(call cc-ifversion, -lt, 0407, \
-							gcc_3_4.o, gcc_4_7.o)
-- 
2.7.4

[PATCH 19/23] kcov: imply GCC_PLUGINS and GCC_PLUGIN_SANCOV instead of select'ing them

[Masahiro Yamada]

As Documentation/kbuild/kconfig-language.txt notes, 'select' should be
used with care - it forces a lower limit of another symbol, ignoring
the dependency.  In this case, KCOV can select GCC_PLUGINS even if
'depends on HAVE_GCC_PLUGINS' is unmet.

'imply' is modest enough to observe the dependency, and this makes
sense.  If you enable KCOV, you will probably want to enable
GCC_PLUGIN_SANCOV, but it should not break the dependency.

I also remove unneeded code, I just happened to notice.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 lib/Kconfig.debug | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 6088408..1216ce6 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -743,8 +743,8 @@ config KCOV
 	bool "Code coverage for fuzzing"
 	depends on ARCH_HAS_KCOV
 	select DEBUG_FS
-	select GCC_PLUGINS if !COMPILE_TEST
-	select GCC_PLUGIN_SANCOV if !COMPILE_TEST
+	imply GCC_PLUGINS
+	imply GCC_PLUGIN_SANCOV
 	help
 	  KCOV exposes kernel code coverage information in a form suitable
 	  for coverage-guided fuzzing (randomized testing).
@@ -758,7 +758,6 @@ config KCOV
 config KCOV_ENABLE_COMPARISONS
 	bool "Enable comparison operands collection by KCOV"
 	depends on KCOV
-	default n
 	help
 	  KCOV also exposes operands of every comparison in the instrumented
 	  code along with operand sizes and PCs of the comparison instructions.
@@ -768,7 +767,7 @@ config KCOV_ENABLE_COMPARISONS
 config KCOV_INSTRUMENT_ALL
 	bool "Instrument all code by default"
 	depends on KCOV
-	default y if KCOV
+	default y
 	help
 	  If you are doing generic system call fuzzing (like e.g. syzkaller),
 	  then you will want to instrument the whole kernel and you should
-- 
2.7.4

[PATCH 20/23] gcc-plugins: always build plugins with C++

[Masahiro Yamada]

If the target compiler is GCC 4.8 or newer, plugins are compiled with
HOSTCXX.  Otherwise, gcc-plugin.sh will select HOSTCC or HOSTCXX.

To simpily things, let's decide GCC 4.8 is the requirement for GCC
pulgins.  With this, plugins are always built with HOSTCXX.  This is a
feature of advanced users, so this requirement whould not be not a big
issue.

Add 'depends on GCC_VERSION >= 40800' to GCC_PLUGINS, and remove the
code that was needed to build plugins with HOSTCC.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 arch/Kconfig                 |  1 +
 scripts/Makefile.gcc-plugins | 11 +++--------
 scripts/gcc-plugin.sh        | 38 +++-----------------------------------
 scripts/gcc-plugins/Makefile | 15 ++++-----------
 4 files changed, 11 insertions(+), 54 deletions(-)

diff --git a/arch/Kconfig b/arch/Kconfig
index 9b7a628..9bd4e1f 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -410,6 +410,7 @@ config HAVE_GCC_PLUGINS
 menuconfig GCC_PLUGINS
 	bool "GCC plugins"
 	depends on HAVE_GCC_PLUGINS
+	depends on GCC_VERSION >= 40800
 	depends on !COMPILE_TEST
 	help
 	  GCC plugins are loadable modules that provide extra features to the
diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index b2a95af..b0f9108 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -1,7 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 ifdef CONFIG_GCC_PLUGINS
-  __PLUGINCC := $(call cc-ifversion, -ge, 0408, $(HOSTCXX), $(HOSTCC))
-  PLUGINCC := $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-plugin.sh "$(__PLUGINCC)" "$(HOSTCXX)" "$(CC)")
+  PLUGINCC := $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-plugin.sh $(HOSTCXX) $(CC))
 
   SANCOV_PLUGIN := -fplugin=$(objtree)/scripts/gcc-plugins/sancov_plugin.so
 
@@ -65,12 +64,8 @@ ifdef CONFIG_GCC_PLUGINS
 	  @echo "Cannot use CONFIG_GCC_PLUGINS: plugin support on gcc <= 5.1 is buggy on powerpc, please upgrade to gcc 5.2 or newer" >&2 && exit 1
         endif
       endif
-      ifeq ($(call cc-ifversion, -ge, 0405, y), y)
-	$(Q)$(srctree)/scripts/gcc-plugin.sh --show-error "$(__PLUGINCC)" "$(HOSTCXX)" "$(CC)" || true
-	@echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
-      else
-	@echo "Cannot use CONFIG_GCC_PLUGINS: your gcc version does not support plugins, you should upgrade it to at least gcc 4.5" >&2 && exit 1
-      endif
+      $(Q)$(srctree)/scripts/gcc-plugin.sh --show-error $(HOSTCXX) $(CC) || true
+      @echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
     endif
   endif
 endif
diff --git a/scripts/gcc-plugin.sh b/scripts/gcc-plugin.sh
index d3caefe..0edbdae 100755
--- a/scripts/gcc-plugin.sh
+++ b/scripts/gcc-plugin.sh
@@ -8,42 +8,10 @@ if [ "$1" = "--show-error" ] ; then
 	shift || true
 fi
 
-gccplugins_dir=$($3 -print-file-name=plugin)
-plugincc=$($1 -E -x c++ - -o /dev/null -I"${srctree}"/gcc-plugins -I"${gccplugins_dir}"/include 2>&1 <<EOF
-#include "gcc-common.h"
-#if BUILDING_GCC_VERSION >= 4008 || defined(ENABLE_BUILD_WITH_CXX)
-#warning $2 CXX
-#else
-#warning $1 CC
-#endif
-EOF
-)
-
-if [ $? -ne 0 ]
-then
-	if [ -n "$SHOW_ERROR" ] ; then
-		echo "${plugincc}" >&2
-	fi
-	exit 1
-fi
-
-case "$plugincc" in
-	*"$1 CC"*)
-		echo "$1"
-		exit 0
-		;;
-
-	*"$2 CXX"*)
-		# the c++ compiler needs another test, see below
-		;;
-
-	*)
-		exit 1
-		;;
-esac
+gccplugins_dir=$($2 -print-file-name=plugin)
 
 # we need a c++ compiler that supports the designated initializer GNU extension
-plugincc=$($2 -c -x c++ -std=gnu++98 - -fsyntax-only -I"${srctree}"/gcc-plugins -I"${gccplugins_dir}"/include 2>&1 <<EOF
+plugincc=$($1 -c -x c++ -std=gnu++98 - -fsyntax-only -I"${srctree}"/gcc-plugins -I"${gccplugins_dir}"/include 2>&1 <<EOF
 #include "gcc-common.h"
 class test {
 public:
@@ -56,7 +24,7 @@ EOF
 
 if [ $? -eq 0 ]
 then
-	echo "$2"
+	echo "$1"
 	exit 0
 fi
 
diff --git a/scripts/gcc-plugins/Makefile b/scripts/gcc-plugins/Makefile
index e2ff425..47d5f69 100644
--- a/scripts/gcc-plugins/Makefile
+++ b/scripts/gcc-plugins/Makefile
@@ -1,17 +1,10 @@
 # SPDX-License-Identifier: GPL-2.0
 GCC_PLUGINS_DIR := $(shell $(CC) -print-file-name=plugin)
 
-ifeq ($(PLUGINCC),$(HOSTCC))
-  HOSTLIBS := hostlibs
-  HOST_EXTRACFLAGS += -I$(GCC_PLUGINS_DIR)/include -I$(src) -std=gnu99 -ggdb
-  export HOST_EXTRACFLAGS
-else
-  HOSTLIBS := hostcxxlibs
-  HOST_EXTRACXXFLAGS += -I$(GCC_PLUGINS_DIR)/include -I$(src) -std=gnu++98 -fno-rtti
-  HOST_EXTRACXXFLAGS += -fno-exceptions -fasynchronous-unwind-tables -ggdb
-  HOST_EXTRACXXFLAGS += -Wno-narrowing -Wno-unused-variable
-  export HOST_EXTRACXXFLAGS
-endif
+HOSTLIBS := hostcxxlibs
+HOST_EXTRACXXFLAGS += -I$(GCC_PLUGINS_DIR)/include -I$(src) -std=gnu++98 -fno-rtti
+HOST_EXTRACXXFLAGS += -fno-exceptions -fasynchronous-unwind-tables -ggdb
+HOST_EXTRACXXFLAGS += -Wno-narrowing -Wno-unused-variable
 
 ifneq ($(CFLAGS_KCOV), $(SANCOV_PLUGIN))
   GCC_PLUGIN := $(filter-out $(SANCOV_PLUGIN), $(GCC_PLUGIN))
-- 
2.7.4

[PATCH 21/23] gcc-plugins: move GCC version check for PowerPC to Kconfig

[Masahiro Yamada]

For PowerPC, GCC 5.2 is the requirement for GCC plugins.  Move the
version check to Kconfig, and remove the ugly checker.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 arch/powerpc/Kconfig         | 2 +-
 scripts/Makefile.gcc-plugins | 8 --------
 2 files changed, 1 insertion(+), 9 deletions(-)

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 73ce5dd..b8474c2 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -195,7 +195,7 @@ config PPC
 	select HAVE_FTRACE_MCOUNT_RECORD
 	select HAVE_FUNCTION_GRAPH_TRACER
 	select HAVE_FUNCTION_TRACER
-	select HAVE_GCC_PLUGINS
+	select HAVE_GCC_PLUGINS			if GCC_VERSION >= 50200
 	select HAVE_GENERIC_GUP
 	select HAVE_HW_BREAKPOINT		if PERF_EVENTS && (PPC_BOOK3S || PPC_8xx)
 	select HAVE_IDE
diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index b0f9108..25da4c0 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -56,14 +56,6 @@ gcc-plugins-check: FORCE
 ifdef CONFIG_GCC_PLUGINS
   ifeq ($(PLUGINCC),)
     ifneq ($(GCC_PLUGINS_CFLAGS),)
-      # Various gccs between 4.5 and 5.1 have bugs on powerpc due to missing
-      # header files. gcc <= 4.6 doesn't work at all, gccs from 4.8 to 5.1 have
-      # issues with 64-bit targets.
-      ifeq ($(ARCH),powerpc)
-        ifeq ($(call cc-ifversion, -le, 0501, y), y)
-	  @echo "Cannot use CONFIG_GCC_PLUGINS: plugin support on gcc <= 5.1 is buggy on powerpc, please upgrade to gcc 5.2 or newer" >&2 && exit 1
-        endif
-      endif
       $(Q)$(srctree)/scripts/gcc-plugin.sh --show-error $(HOSTCXX) $(CC) || true
       @echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
     endif
-- 
2.7.4

[PATCH 22/23] gcc-plugins: test GCC plugin support in Kconfig

[Masahiro Yamada]

Run scripts/gcc-plugin.sh from Kconfig.  Users can enable GCC_PLUGINS
only when it is supported.

I dropped 'depends on GCC_VERSION >= 40800'.  I guess gcc-plugin.sh
will not pass with GCC 4.7 or older.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 arch/Kconfig                 |  6 +++-
 scripts/Makefile.gcc-plugins | 82 ++++++++++++++++----------------------------
 scripts/gcc-plugin.sh        |  1 -
 3 files changed, 34 insertions(+), 55 deletions(-)

diff --git a/arch/Kconfig b/arch/Kconfig
index 9bd4e1f..d567bd1 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -407,10 +407,14 @@ config HAVE_GCC_PLUGINS
 	  An arch should select this symbol if it supports building with
 	  GCC plugins.
 
+config CC_HAS_GCC_PLUGINS
+	bool
+	default $(shell $srctree/scripts/gcc-plugin.sh $HOSTCXX $CC)
+
 menuconfig GCC_PLUGINS
 	bool "GCC plugins"
 	depends on HAVE_GCC_PLUGINS
-	depends on GCC_VERSION >= 40800
+	depends on CC_HAS_GCC_PLUGINS
 	depends on !COMPILE_TEST
 	help
 	  GCC plugins are loadable modules that provide extra features to the
diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index 25da4c0..19d0d5b 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -1,71 +1,47 @@
 # SPDX-License-Identifier: GPL-2.0
-ifdef CONFIG_GCC_PLUGINS
-  PLUGINCC := $(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-plugin.sh $(HOSTCXX) $(CC))
-
-  SANCOV_PLUGIN := -fplugin=$(objtree)/scripts/gcc-plugins/sancov_plugin.so
+SANCOV_PLUGIN := -fplugin=$(objtree)/scripts/gcc-plugins/sancov_plugin.so
 
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_CYC_COMPLEXITY)	+= cyc_complexity_plugin.so
+gcc-plugin-$(CONFIG_GCC_PLUGIN_CYC_COMPLEXITY)	+= cyc_complexity_plugin.so
 
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= latent_entropy_plugin.so
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= -DLATENT_ENTROPY_PLUGIN
-  ifdef CONFIG_GCC_PLUGIN_LATENT_ENTROPY
-    DISABLE_LATENT_ENTROPY_PLUGIN			+= -fplugin-arg-latent_entropy_plugin-disable
-  endif
+gcc-plugin-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= latent_entropy_plugin.so
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_LATENT_ENTROPY)	+= -DLATENT_ENTROPY_PLUGIN
+ifdef CONFIG_GCC_PLUGIN_LATENT_ENTROPY
+  DISABLE_LATENT_ENTROPY_PLUGIN			+= -fplugin-arg-latent_entropy_plugin-disable
+endif
 
-  ifdef CONFIG_GCC_PLUGIN_SANCOV
-    ifeq ($(CFLAGS_KCOV),)
-      # It is needed because of the gcc-plugin.sh and gcc version checks.
-      gcc-plugin-$(CONFIG_GCC_PLUGIN_SANCOV)           += sancov_plugin.so
+ifdef CONFIG_GCC_PLUGIN_SANCOV
+  ifeq ($(CFLAGS_KCOV),)
+    # It is needed because of the gcc-plugin.sh and gcc version checks.
+    gcc-plugin-$(CONFIG_GCC_PLUGIN_SANCOV)           += sancov_plugin.so
 
-      ifneq ($(PLUGINCC),)
-        CFLAGS_KCOV := $(SANCOV_PLUGIN)
-      else
-        $(warning warning: cannot use CONFIG_KCOV: -fsanitize-coverage=trace-pc is not supported by compiler)
-      endif
-    endif
+    CFLAGS_KCOV := $(SANCOV_PLUGIN)
   endif
+endif
 
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= structleak_plugin.so
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_VERBOSE)	+= -fplugin-arg-structleak_plugin-verbose
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)	+= -fplugin-arg-structleak_plugin-byref-all
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= -DSTRUCTLEAK_PLUGIN
-
-  gcc-plugin-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= randomize_layout_plugin.so
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= -DRANDSTRUCT_PLUGIN
-  gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT_PERFORMANCE)	+= -fplugin-arg-randomize_layout_plugin-performance-mode
+gcc-plugin-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= structleak_plugin.so
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_VERBOSE)	+= -fplugin-arg-structleak_plugin-verbose
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)	+= -fplugin-arg-structleak_plugin-byref-all
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_STRUCTLEAK)	+= -DSTRUCTLEAK_PLUGIN
 
-  GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y))
+gcc-plugin-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= randomize_layout_plugin.so
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT)	+= -DRANDSTRUCT_PLUGIN
+gcc-plugin-cflags-$(CONFIG_GCC_PLUGIN_RANDSTRUCT_PERFORMANCE)	+= -fplugin-arg-randomize_layout_plugin-performance-mode
 
-  export PLUGINCC GCC_PLUGINS_CFLAGS GCC_PLUGIN GCC_PLUGIN_SUBDIR
-  export SANCOV_PLUGIN DISABLE_LATENT_ENTROPY_PLUGIN
+GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y))
 
-  ifneq ($(PLUGINCC),)
-    # SANCOV_PLUGIN can be only in CFLAGS_KCOV because avoid duplication.
-    GCC_PLUGINS_CFLAGS := $(filter-out $(SANCOV_PLUGIN), $(GCC_PLUGINS_CFLAGS))
-  endif
+export GCC_PLUGINS_CFLAGS GCC_PLUGIN GCC_PLUGIN_SUBDIR
+export SANCOV_PLUGIN DISABLE_LATENT_ENTROPY_PLUGIN
 
-  KBUILD_CFLAGS += $(GCC_PLUGINS_CFLAGS)
-  GCC_PLUGIN := $(gcc-plugin-y)
-  GCC_PLUGIN_SUBDIR := $(gcc-plugin-subdir-y)
-endif
+# SANCOV_PLUGIN can be only in CFLAGS_KCOV because avoid duplication.
+GCC_PLUGINS_CFLAGS := $(filter-out $(SANCOV_PLUGIN), $(GCC_PLUGINS_CFLAGS))
 
-# If plugins aren't supported, abort the build before hard-to-read compiler
-# errors start getting spewed by the main build.
-PHONY += gcc-plugins-check
-gcc-plugins-check: FORCE
-ifdef CONFIG_GCC_PLUGINS
-  ifeq ($(PLUGINCC),)
-    ifneq ($(GCC_PLUGINS_CFLAGS),)
-      $(Q)$(srctree)/scripts/gcc-plugin.sh --show-error $(HOSTCXX) $(CC) || true
-      @echo "Cannot use CONFIG_GCC_PLUGINS: your gcc installation does not support plugins, perhaps the necessary headers are missing?" >&2 && exit 1
-    endif
-  endif
-endif
-	@:
+KBUILD_CFLAGS += $(GCC_PLUGINS_CFLAGS)
+GCC_PLUGIN := $(gcc-plugin-y)
+GCC_PLUGIN_SUBDIR := $(gcc-plugin-subdir-y)
 
 # Actually do the build, if requested.
 PHONY += gcc-plugins
-gcc-plugins: scripts_basic gcc-plugins-check
+gcc-plugins: scripts_basic
 ifdef CONFIG_GCC_PLUGINS
 	$(Q)$(MAKE) $(build)=scripts/gcc-plugins
 endif
diff --git a/scripts/gcc-plugin.sh b/scripts/gcc-plugin.sh
index 0edbdae..b18c69c 100755
--- a/scripts/gcc-plugin.sh
+++ b/scripts/gcc-plugin.sh
@@ -24,7 +24,6 @@ EOF
 
 if [ $? -eq 0 ]
 then
-	echo "$1"
 	exit 0
 fi
 
-- 
2.7.4

[PATCH 23/23] gcc-plugins: enable GCC_PLUGINS for COMPILE_TEST

[Masahiro Yamada]

The plugin availability is checked in Kconfig, so all{yes,mod}config
will not be bothered.  Remove 'depends on !COMPILE_TEST'.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

 arch/Kconfig | 1 -
 1 file changed, 1 deletion(-)

diff --git a/arch/Kconfig b/arch/Kconfig
index d567bd1..5b5610b 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -415,7 +415,6 @@ menuconfig GCC_PLUGINS
 	bool "GCC plugins"
 	depends on HAVE_GCC_PLUGINS
 	depends on CC_HAS_GCC_PLUGINS
-	depends on !COMPILE_TEST
 	help
 	  GCC plugins are loadable modules that provide extra features to the
 	  compiler. They are useful for runtime instrumentation and static analysis.
-- 
2.7.4

Re: [PATCH 11/23] kconfig: add 'shell-stdout' function

[Linus Torvalds]

On Fri, Feb 16, 2018 at 10:38 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> This is the second built-in function, which retrieves the first line
> of stdout from the given shell command.

This is the only part I really don't much like in your patch series.

Most of it is just lovely and looks very nice and powerful, but the
difference between "$(shell ..." and "$(shell-stdout ..." to me is
bvery ugly.

Can we *please* make "shell-stdout" go away, and make this just be "shell"?

The rule would be very simple:

 - if the result of the shell command is a failure, the result is 'n'

 - otherwise, the result is the first line of stdout

 - if the result is empty, we replace it with 'y'.

So doing $(shell true) would be 100% equivalent to $(shell echo y),
and you could still do that

          default $(shell $CC --version | grep -q gcc)

because it would just automatically do the right thing.

Basically, the only difference is how $(shell ) works in the success
case: the result won't necessarily be 'y', it will be whatever output.
But if you want to always turn it into 'y' (say, you don't have a "-q"
flag for the grep equivalent above), you can always do so with

          default $(shell $CC --version | noqgrep gcc > /dev/null)

So it seems to me that there is never any fundamental reason why we'd
want both "shell" and "shell-stdout", since "shell-stdout" is
fundamentally more powerful than "shell", and can always be used as
such (and just renamed to "shell").

Because I really think that it's just much prettier and more intuitive
to be able to say

        default "/boot/config-$(shell uname --release)"

without that "-stdout" thing.

Hmm?

But I do want to say how much I liked this series. Just this part
seemed to result in uglier Kconfig scripts.

         Linus

Re: [PATCH 08/23] kconfig: add 'macro' keyword to support user-defined function

[Nicolas Pitre]

On Sat, 17 Feb 2018, Masahiro Yamada wrote:

> Now, we got a basic ability to test compiler capability in Kconfig.
> 
> config CC_HAS_STACKPROTECTOR
>         bool
>         default $(shell $CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null)
> 
> This works, but it is ugly to repeat this long boilerplate.
> 
> We want to describe like this:
> 
> config CC_HAS_STACKPROTECTOR
>         bool
>         default $(cc-option -fstack-protector)
> 
> It is straight-forward to implement a new function, but I do not like
> to hard-code specialized functions like this.  Hence, here is another
> feature to add functions from Kconfig files.
> 
> A user-defined function can be defined as a string type symbol with
> a special keyword 'macro'.  It can be referenced in the same way as
> built-in functions.  This feature was also inspired by Makefile where
> user-defined functions are referenced by $(call func-name, args...),
> but I omitted the 'call' to makes it shorter.
> 
> The macro definition can contain $(1), $(2), ... which will be replaced
> with arguments from the caller.
> 
> Example code:
> 
>   config cc-option
>           string
>           macro $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)

I think this syntax for defining a macro shouldn't start with the 
"config" keyword, unless you want it to be part of the config symbol 
space and land it in .config. And typing it as a "string" while it 
actually returns y/n (hence a bool) is also strange.

What about this instead:

macro cc-option
	bool $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)

This makes it easier to extend as well if need be.


Nicolas

Re: [PATCH 08/23] kconfig: add 'macro' keyword to support user-defined function

[Ulf Magnusson]

On Fri, Feb 16, 2018 at 02:49:31PM -0500, Nicolas Pitre wrote:
> On Sat, 17 Feb 2018, Masahiro Yamada wrote:
> 
> > Now, we got a basic ability to test compiler capability in Kconfig.
> > 
> > config CC_HAS_STACKPROTECTOR
> >         bool
> >         default $(shell $CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null)
> > 
> > This works, but it is ugly to repeat this long boilerplate.
> > 
> > We want to describe like this:
> > 
> > config CC_HAS_STACKPROTECTOR
> >         bool
> >         default $(cc-option -fstack-protector)
> > 
> > It is straight-forward to implement a new function, but I do not like
> > to hard-code specialized functions like this.  Hence, here is another
> > feature to add functions from Kconfig files.
> > 
> > A user-defined function can be defined as a string type symbol with
> > a special keyword 'macro'.  It can be referenced in the same way as
> > built-in functions.  This feature was also inspired by Makefile where
> > user-defined functions are referenced by $(call func-name, args...),
> > but I omitted the 'call' to makes it shorter.
> > 
> > The macro definition can contain $(1), $(2), ... which will be replaced
> > with arguments from the caller.
> > 
> > Example code:
> > 
> >   config cc-option
> >           string
> >           macro $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
> 
> I think this syntax for defining a macro shouldn't start with the 
> "config" keyword, unless you want it to be part of the config symbol 
> space and land it in .config. And typing it as a "string" while it 
> actually returns y/n (hence a bool) is also strange.
> 
> What about this instead:
> 
> macro cc-option
> 	bool $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
> 
> This makes it easier to extend as well if need be.
> 
> 
> Nicolas

I haven't gone over the patchset in detail yet and might be missing
something here, but if this is just meant to be a textual shorthand,
then why give it a type at all?

Do you think a simpler syntax like this would make sense?

	macro cc-option "$(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)"

That's the most general version, where you could use it for other stuff
besides $(shell ...) as well, just to keep parity.

You could then always just expand $() as a string, and maybe spit out
"n" and "y" in the cases Linus suggested for $(shell ...). The existing
logic for constant symbols should then take care of converting that into
a tristate value where appropriate.

If you go with that and want to support $() outside quotes, then

	$(foo)

would just be a shorthand for

	"$(foo)"

Are there any cases where something more advanced than that might be
warranted (e.g., macros that expand to complete expressions)? It seems
pretty nice and nonmagical otherwise.

Cheers,
Ulf

Re: [PATCH 08/23] kconfig: add 'macro' keyword to support user-defined function

[Nicolas Pitre]

On Sat, 17 Feb 2018, Ulf Magnusson wrote:

> On Fri, Feb 16, 2018 at 02:49:31PM -0500, Nicolas Pitre wrote:
> > On Sat, 17 Feb 2018, Masahiro Yamada wrote:
> > 
> > > Now, we got a basic ability to test compiler capability in Kconfig.
> > > 
> > > config CC_HAS_STACKPROTECTOR
> > >         bool
> > >         default $(shell $CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null)
> > > 
> > > This works, but it is ugly to repeat this long boilerplate.
> > > 
> > > We want to describe like this:
> > > 
> > > config CC_HAS_STACKPROTECTOR
> > >         bool
> > >         default $(cc-option -fstack-protector)
> > > 
> > > It is straight-forward to implement a new function, but I do not like
> > > to hard-code specialized functions like this.  Hence, here is another
> > > feature to add functions from Kconfig files.
> > > 
> > > A user-defined function can be defined as a string type symbol with
> > > a special keyword 'macro'.  It can be referenced in the same way as
> > > built-in functions.  This feature was also inspired by Makefile where
> > > user-defined functions are referenced by $(call func-name, args...),
> > > but I omitted the 'call' to makes it shorter.
> > > 
> > > The macro definition can contain $(1), $(2), ... which will be replaced
> > > with arguments from the caller.
> > > 
> > > Example code:
> > > 
> > >   config cc-option
> > >           string
> > >           macro $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
> > 
> > I think this syntax for defining a macro shouldn't start with the 
> > "config" keyword, unless you want it to be part of the config symbol 
> > space and land it in .config. And typing it as a "string" while it 
> > actually returns y/n (hence a bool) is also strange.
> > 
> > What about this instead:
> > 
> > macro cc-option
> > 	bool $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
> > 
> > This makes it easier to extend as well if need be.
> > 
> > 
> > Nicolas
> 
> I haven't gone over the patchset in detail yet and might be missing
> something here, but if this is just meant to be a textual shorthand,
> then why give it a type at all?

It is meant to be like a user-defined function.

> Do you think a simpler syntax like this would make sense?
> 
> 	macro cc-option "$(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)"
> 
> That's the most general version, where you could use it for other stuff
> besides $(shell ...) as well, just to keep parity.

This is not extendable.  Let's imagine that you might want to implement 
some kind of conditionals some day e.g.:

macro complex_test
	bool $(shell foo) if LOCKDEP_SUPPORT
	bool y if DEBUG_DRIVER
	bool n

There is no real advantage to simplify the macro definition to its 
simplest expression, unlike its actual usage.

> Are there any cases where something more advanced than that might be
> warranted (e.g., macros that expand to complete expressions)?

Maybe not now, but there is no need to close the door on the possibility 
either.


Nicolas

Re: [PATCH 08/23] kconfig: add 'macro' keyword to support user-defined function

[Ulf Magnusson]

On Sat, Feb 17, 2018 at 3:30 AM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Sat, 17 Feb 2018, Ulf Magnusson wrote:
>
>> On Fri, Feb 16, 2018 at 02:49:31PM -0500, Nicolas Pitre wrote:
>> > On Sat, 17 Feb 2018, Masahiro Yamada wrote:
>> >
>> > > Now, we got a basic ability to test compiler capability in Kconfig.
>> > >
>> > > config CC_HAS_STACKPROTECTOR
>> > >         bool
>> > >         default $(shell $CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null)
>> > >
>> > > This works, but it is ugly to repeat this long boilerplate.
>> > >
>> > > We want to describe like this:
>> > >
>> > > config CC_HAS_STACKPROTECTOR
>> > >         bool
>> > >         default $(cc-option -fstack-protector)
>> > >
>> > > It is straight-forward to implement a new function, but I do not like
>> > > to hard-code specialized functions like this.  Hence, here is another
>> > > feature to add functions from Kconfig files.
>> > >
>> > > A user-defined function can be defined as a string type symbol with
>> > > a special keyword 'macro'.  It can be referenced in the same way as
>> > > built-in functions.  This feature was also inspired by Makefile where
>> > > user-defined functions are referenced by $(call func-name, args...),
>> > > but I omitted the 'call' to makes it shorter.
>> > >
>> > > The macro definition can contain $(1), $(2), ... which will be replaced
>> > > with arguments from the caller.
>> > >
>> > > Example code:
>> > >
>> > >   config cc-option
>> > >           string
>> > >           macro $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
>> >
>> > I think this syntax for defining a macro shouldn't start with the
>> > "config" keyword, unless you want it to be part of the config symbol
>> > space and land it in .config. And typing it as a "string" while it
>> > actually returns y/n (hence a bool) is also strange.
>> >
>> > What about this instead:
>> >
>> > macro cc-option
>> >     bool $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
>> >
>> > This makes it easier to extend as well if need be.
>> >
>> >
>> > Nicolas
>>
>> I haven't gone over the patchset in detail yet and might be missing
>> something here, but if this is just meant to be a textual shorthand,
>> then why give it a type at all?
>
> It is meant to be like a user-defined function.
>
>> Do you think a simpler syntax like this would make sense?
>>
>>       macro cc-option "$(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)"
>>
>> That's the most general version, where you could use it for other stuff
>> besides $(shell ...) as well, just to keep parity.
>
> This is not extendable.  Let's imagine that you might want to implement
> some kind of conditionals some day e.g.:
>
> macro complex_test
>         bool $(shell foo) if LOCKDEP_SUPPORT
>         bool y if DEBUG_DRIVER
>         bool n

I still don't quite get the semantics here. How would the behavior
change if the type was changed to say string or int in some or all of
the lines?

Since the current model is to evaluate $() while the Kconfig files are
being parsed, would this require evaluating Kconfig expressions during
parsing? There is a relatively clean and (somewhat) easy to understand
parsing/evaluation separation at the moment, which I like.

Do you have anything in mind that would be cleaner and simpler to
implement in this way compared to using plain symbols?

>
> There is no real advantage to simplify the macro definition to its
> simplest expression, unlike its actual usage.

Maybe I'm being grumpy, but this feels like it's adding complexity
rather than reducing it.

I like the rest of this patchset, because the behavior is easy to
understand and fits well with Kconfig's evaluation model: $() is just
a kind of preprocessor that runs during parsing and does value
substitution based on shell commands, possibly along with some helper
macros to avoid repetition.

I think we should think hard about whether we actually need anything
more than that before complicating Kconfig even further "just in
case." If the goal is simplification, then it's bad if we eventually
end up with a bigger mess than the Makefiles.

>
>> Are there any cases where something more advanced than that might be
>> warranted (e.g., macros that expand to complete expressions)?
>
> Maybe not now, but there is no need to close the door on the possibility
> either.
>
>
> Nicolas

Kconfig has no notion of types for expressions by the way. The
simplest way to look at it is that all symbols have a tristate value
(which is n for non-bool/tristate symbols) and a string value. Which
one gets used depends on the context. In A && B, the tristate values
are used, and in A = B the string values are compared.

In something like 'default "foo bar"', "foo bar" is actually a
constant symbol. If we were to drop the straightforward preprocessor
model, then constant symbols would no longer necessarily be constant.
I have a feeling that that might turn Kconfig's internals even
messier.

Constant (and undefined) symbols end up with their name as their
string value by the way, which is why stuff like 'A = "foo"' works.

IMO, let's just go with the simple preprocessor model and let macros
be dumb text substitutions (if we don't want to hardcode
functionality). It's simple and probably good enough, keeps parsing
and evaluation nicely separated, and keeps Kconfig somewhat
comprehensible.

(Note that the preprocessor could still be extended, if we ever need
anything besides $(shell ...). Macros could still just do text
substitution.)

Cheers,
Ulf

Re: [PATCH 08/23] kconfig: add 'macro' keyword to support user-defined function

[Nicolas Pitre]

On Sat, 17 Feb 2018, Ulf Magnusson wrote:

> On Sat, Feb 17, 2018 at 3:30 AM, Nicolas Pitre <nico@fluxnic.net> wrote:
> > On Sat, 17 Feb 2018, Ulf Magnusson wrote:
> >
> >> On Fri, Feb 16, 2018 at 02:49:31PM -0500, Nicolas Pitre wrote:
> >> > On Sat, 17 Feb 2018, Masahiro Yamada wrote:
> >> >
> >> > > Now, we got a basic ability to test compiler capability in Kconfig.
> >> > >
> >> > > config CC_HAS_STACKPROTECTOR
> >> > >         bool
> >> > >         default $(shell $CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null)
> >> > >
> >> > > This works, but it is ugly to repeat this long boilerplate.
> >> > >
> >> > > We want to describe like this:
> >> > >
> >> > > config CC_HAS_STACKPROTECTOR
> >> > >         bool
> >> > >         default $(cc-option -fstack-protector)
> >> > >
> >> > > It is straight-forward to implement a new function, but I do not like
> >> > > to hard-code specialized functions like this.  Hence, here is another
> >> > > feature to add functions from Kconfig files.
> >> > >
> >> > > A user-defined function can be defined as a string type symbol with
> >> > > a special keyword 'macro'.  It can be referenced in the same way as
> >> > > built-in functions.  This feature was also inspired by Makefile where
> >> > > user-defined functions are referenced by $(call func-name, args...),
> >> > > but I omitted the 'call' to makes it shorter.
> >> > >
> >> > > The macro definition can contain $(1), $(2), ... which will be replaced
> >> > > with arguments from the caller.
> >> > >
> >> > > Example code:
> >> > >
> >> > >   config cc-option
> >> > >           string
> >> > >           macro $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
> >> >
> >> > I think this syntax for defining a macro shouldn't start with the
> >> > "config" keyword, unless you want it to be part of the config symbol
> >> > space and land it in .config. And typing it as a "string" while it
> >> > actually returns y/n (hence a bool) is also strange.
> >> >
> >> > What about this instead:
> >> >
> >> > macro cc-option
> >> >     bool $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
> >> >
> >> > This makes it easier to extend as well if need be.
> >> >
> >> >
> >> > Nicolas
> >>
> >> I haven't gone over the patchset in detail yet and might be missing
> >> something here, but if this is just meant to be a textual shorthand,
> >> then why give it a type at all?
> >
> > It is meant to be like a user-defined function.
> >
> >> Do you think a simpler syntax like this would make sense?
> >>
> >>       macro cc-option "$(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)"
> >>
> >> That's the most general version, where you could use it for other stuff
> >> besides $(shell ...) as well, just to keep parity.
> >
> > This is not extendable.  Let's imagine that you might want to implement
> > some kind of conditionals some day e.g.:
> >
> > macro complex_test
> >         bool $(shell foo) if LOCKDEP_SUPPORT
> >         bool y if DEBUG_DRIVER
> >         bool n
> 
> I still don't quite get the semantics here. How would the behavior
> change if the type was changed to say string or int in some or all of
> the lines?

I admit this wouldn't make sense to have multiple different types. In 
this example, the bool keyword acts as syntactic sugar more than 
anything else.

> Since the current model is to evaluate $() while the Kconfig files are
> being parsed, would this require evaluating Kconfig expressions during
> parsing? There is a relatively clean and (somewhat) easy to understand
> parsing/evaluation separation at the moment, which I like.

Agreed. Let's forget about the conditionals then.


Nicolas

Re: [PATCH 08/23] kconfig: add 'macro' keyword to support user-defined function

[Ulf Magnusson]

On Fri, Feb 16, 2018 at 11:44:25PM -0500, Nicolas Pitre wrote:
> On Sat, 17 Feb 2018, Ulf Magnusson wrote:
> 
> > On Sat, Feb 17, 2018 at 3:30 AM, Nicolas Pitre <nico@fluxnic.net> wrote:
> > > On Sat, 17 Feb 2018, Ulf Magnusson wrote:
> > >
> > >> On Fri, Feb 16, 2018 at 02:49:31PM -0500, Nicolas Pitre wrote:
> > >> > On Sat, 17 Feb 2018, Masahiro Yamada wrote:
> > >> >
> > >> > > Now, we got a basic ability to test compiler capability in Kconfig.
> > >> > >
> > >> > > config CC_HAS_STACKPROTECTOR
> > >> > >         bool
> > >> > >         default $(shell $CC -Werror -fstack-protector -c -x c /dev/null -o /dev/null)
> > >> > >
> > >> > > This works, but it is ugly to repeat this long boilerplate.
> > >> > >
> > >> > > We want to describe like this:
> > >> > >
> > >> > > config CC_HAS_STACKPROTECTOR
> > >> > >         bool
> > >> > >         default $(cc-option -fstack-protector)
> > >> > >
> > >> > > It is straight-forward to implement a new function, but I do not like
> > >> > > to hard-code specialized functions like this.  Hence, here is another
> > >> > > feature to add functions from Kconfig files.
> > >> > >
> > >> > > A user-defined function can be defined as a string type symbol with
> > >> > > a special keyword 'macro'.  It can be referenced in the same way as
> > >> > > built-in functions.  This feature was also inspired by Makefile where
> > >> > > user-defined functions are referenced by $(call func-name, args...),
> > >> > > but I omitted the 'call' to makes it shorter.
> > >> > >
> > >> > > The macro definition can contain $(1), $(2), ... which will be replaced
> > >> > > with arguments from the caller.
> > >> > >
> > >> > > Example code:
> > >> > >
> > >> > >   config cc-option
> > >> > >           string
> > >> > >           macro $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
> > >> >
> > >> > I think this syntax for defining a macro shouldn't start with the
> > >> > "config" keyword, unless you want it to be part of the config symbol
> > >> > space and land it in .config. And typing it as a "string" while it
> > >> > actually returns y/n (hence a bool) is also strange.
> > >> >
> > >> > What about this instead:
> > >> >
> > >> > macro cc-option
> > >> >     bool $(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)
> > >> >
> > >> > This makes it easier to extend as well if need be.
> > >> >
> > >> >
> > >> > Nicolas
> > >>
> > >> I haven't gone over the patchset in detail yet and might be missing
> > >> something here, but if this is just meant to be a textual shorthand,
> > >> then why give it a type at all?
> > >
> > > It is meant to be like a user-defined function.
> > >
> > >> Do you think a simpler syntax like this would make sense?
> > >>
> > >>       macro cc-option "$(shell $CC -Werror $(1) -c -x c /dev/null -o /dev/null)"
> > >>
> > >> That's the most general version, where you could use it for other stuff
> > >> besides $(shell ...) as well, just to keep parity.
> > >
> > > This is not extendable.  Let's imagine that you might want to implement
> > > some kind of conditionals some day e.g.:
> > >
> > > macro complex_test
> > >         bool $(shell foo) if LOCKDEP_SUPPORT
> > >         bool y if DEBUG_DRIVER
> > >         bool n
> > 
> > I still don't quite get the semantics here. How would the behavior
> > change if the type was changed to say string or int in some or all of
> > the lines?
> 
> I admit this wouldn't make sense to have multiple different types. In 
> this example, the bool keyword acts as syntactic sugar more than 
> anything else.
> 
> > Since the current model is to evaluate $() while the Kconfig files are
> > being parsed, would this require evaluating Kconfig expressions during
> > parsing? There is a relatively clean and (somewhat) easy to understand
> > parsing/evaluation separation at the moment, which I like.
> 
> Agreed. Let's forget about the conditionals then.
> 
> 
> Nicolas

This is also related to why it feels off to me to (at least for its own
sake) make macro definitions mimic symbol definitions.

To me, parsing being a different domain makes it "okay" to use a
different syntax for macros compared to symbol definitions, especially
if it happens to be handier. It even makes things less confusing,
because there's less risk of mixing up the two domains (it's rare to mix
up the preprocessor with C "proper", since the syntax is so different).

More practically, I'm not sure that

	macro foo "definition"

would be that hard to extend in practice, if you'd ever need to. You could
always add a new keyword:

	fancy-macro/function/whatever foo ...

I admit it'd be a bit ugly if you'd ever end up with something like

	macro foo "definition"
		bit_ugly

It's still not the end of the world though, IMO, and I suspect there'd
be better-looking options if you'd need to extend things on the macro
side.

That macro syntax seems like the simplest possible thing to me, with no
obvious major drawbacks. Keeping parsing and evaluation cleanly
separated is more important than the exact syntax though. It's a bonus
if symbols and macros stand out as coming from different universes to
people reading the Kconfig.

Cheers,
Ulf

Re: [PATCH 07/23] kconfig: add function support and implement 'shell' function

[Ulf Magnusson]

On Sat, Feb 17, 2018 at 03:38:35AM +0900, Masahiro Yamada wrote:
> This commit adds a new concept 'function' to Kconfig.  A function call
> resembles a variable reference with arguments, and looks like this:
> 
>   $(function arg1, arg2, arg3, ...)
> 
> (Actually, this syntax was inspired by Makefile.)
> 
> Real examples will look like this:
> 
>   $(shell true)
>   $(cc-option -fstackprotector)
> 
> This commit adds the basic infrastructure to add, delete, evaluate
> functions.
> 
> Also, add the first built-in function $(shell ...).  This evaluates
> to 'y' if the given command exits with 0, 'n' otherwise.
> 
> I am also planning to support user-defined functions (a.k.a 'macro')
> for cases where hard-coding is not preferred.
> 
> If you want to try this feature, the hello-world code is someting below.
> 
> Example code:
> 
>   config CC_IS_GCC
>           bool
>           default $(shell $CC --version | grep -q gcc)
> 
>   config CC_IS_CLANG
>           bool
>           default $(shell $CC --version | grep -q clang)
> 
>   config CC_HAS_OZ
>           bool
>           default $(shell $CC -Werror -Oz -c -x c /dev/null -o /dev/null)
> 
> Result:
> 
>   $ make -s alldefconfig && tail -n 3 .config
>   CONFIG_CC_IS_GCC=y
>   # CONFIG_CC_IS_CLANG is not set
>   # CONFIG_CC_HAS_OZ is not set
> 
>   $ make CC=clang -s alldefconfig && tail -n 3 .config
>   # CONFIG_CC_IS_GCC is not set
>   CONFIG_CC_IS_CLANG=y
>   CONFIG_CC_HAS_OZ=y
> 
> A function call can appear anywhere a symbol reference can appear.
> So, the following code is possible.
> 
> Example code:
> 
>   config CC_NAME
>           string
>           default "gcc" if $(shell $CC --version | grep -q gcc)
>           default "clang" if $(shell $CC --version | grep -q clang)
>           default "unknown compiler"
> 
> Result:
> 
>   $ make -s alldefconfig && tail -n 1 .config
>   CONFIG_CC_NAME="gcc"
> 
>   $ make CC=clang -s alldefconfig && tail -n 1 .config
>   CONFIG_CC_NAME="clang"
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
> 
> Reminder for myself:
> Update Documentation/kbuild/kconfig-language.txt
> 
> 
>  scripts/kconfig/function.c  | 149 ++++++++++++++++++++++++++++++++++++++++++++
>  scripts/kconfig/lkc_proto.h |   5 ++
>  scripts/kconfig/util.c      |  46 +++++++++++---
>  scripts/kconfig/zconf.l     |  38 ++++++++++-
>  scripts/kconfig/zconf.y     |   9 +++
>  5 files changed, 238 insertions(+), 9 deletions(-)
>  create mode 100644 scripts/kconfig/function.c
> 
> diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
> new file mode 100644
> index 0000000..60e59be
> --- /dev/null
> +++ b/scripts/kconfig/function.c
> @@ -0,0 +1,149 @@
> +// SPDX-License-Identifier: GPL-2.0
> +//
> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +
> +#include "list.h"
> +
> +#define FUNCTION_MAX_ARGS		10
> +
> +static LIST_HEAD(function_list);
> +
> +struct function {
> +	const char *name;
> +	char *(*func)(int argc, char *argv[]);
> +	struct list_head node;
> +};
> +
> +static struct function *func_lookup(const char *name)
> +{
> +	struct function *f;
> +
> +	list_for_each_entry(f, &function_list, node) {
> +		if (!strcmp(name, f->name))
> +			return f;
> +	}
> +
> +	return NULL;
> +}
> +
> +static void func_add(const char *name, char *(*func)(int argc, char *argv[]))
> +{
> +	struct function *f;
> +
> +	f = func_lookup(name);
> +	if (f) {
> +		fprintf(stderr, "%s: function already exists. ignored.\n", name);
> +		return;
> +	}
> +
> +	f = xmalloc(sizeof(*f));
> +	f->name = name;
> +	f->func = func;
> +
> +	list_add_tail(&f->node, &function_list);
> +}
> +
> +static void func_del(struct function *f)
> +{
> +	list_del(&f->node);
> +	free(f);
> +}
> +
> +static char *func_call(int argc, char *argv[])
> +{
> +	struct function *f;
> +
> +	f = func_lookup(argv[0]);
> +	if (!f) {
> +		fprintf(stderr, "%s: function not found\n", argv[0]);
> +		return NULL;
> +	}
> +
> +	return f->func(argc, argv);
> +}
> +
> +static char *func_eval(const char *func)
> +{
> +	char *expanded, *saveptr, *str, *token, *res;
> +	const char *delim;
> +	int argc = 0;
> +	char *argv[FUNCTION_MAX_ARGS];
> +
> +	expanded = expand_string_value(func);
> +
> +	str = expanded;
> +	delim = " ";
> +
> +	while ((token = strtok_r(str, delim, &saveptr))) {
> +		argv[argc++] = token;
> +		str = NULL;
> +		delim = ",";
> +	}
> +
> +	res = func_call(argc, argv);
> +
> +	free(expanded);
> +
> +	return res ?: xstrdup("");
> +}
> +
> +char *func_eval_n(const char *func, size_t n)
> +{
> +	char *tmp, *res;
> +
> +	tmp = xmalloc(n + 1);
> +	memcpy(tmp, func, n);
> +	*(tmp + n) = '\0';
> +
> +	res = func_eval(tmp);
> +
> +	free(tmp);
> +
> +	return res;
> +}
> +
> +/* built-in functions */
> +static char *do_shell(int argc, char *argv[])
> +{
> +	static const char *pre = "(";
> +	static const char *post = ") >/dev/null 2>&1";
> +	char *cmd;
> +	int ret;
> +
> +	if (argc != 2)
> +		return NULL;
> +
> +	/*
> +	 * Surround the command with ( ) in case it is piped commands.
> +	 * Also, redirect stdout and stderr to /dev/null.
> +	 */
> +	cmd = xmalloc(strlen(pre) + strlen(argv[1]) + strlen(post) + 1);
> +	strcpy(cmd, pre);
> +	strcat(cmd, argv[1]);
> +	strcat(cmd, post);
> +
> +	ret = system(cmd);
> +
> +	free(cmd);
> +
> +	return xstrdup(ret == 0 ? "y" : "n");
> +}
> +
> +void func_init(void)
> +{
> +	/* register built-in functions */
> +	func_add("shell", do_shell);
> +}
> +
> +void func_exit(void)
> +{
> +	struct function *f, *tmp;
> +
> +	/* unregister all functions */
> +	list_for_each_entry_safe(f, tmp, &function_list, node)
> +		func_del(f);
> +}
> diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
> index 9884adc..09a4f53 100644
> --- a/scripts/kconfig/lkc_proto.h
> +++ b/scripts/kconfig/lkc_proto.h
> @@ -48,5 +48,10 @@ const char * sym_get_string_value(struct symbol *sym);
>  
>  const char * prop_get_type_name(enum prop_type type);
>  
> +/* function.c */
> +char *func_eval_n(const char *func, size_t n);
> +void func_init(void);
> +void func_exit(void);
> +
>  /* expr.c */
>  void expr_print(struct expr *e, void (*fn)(void *, struct symbol *, const char *), void *data, int prevtoken);
> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
> index dddf85b..ed89fb9 100644
> --- a/scripts/kconfig/util.c
> +++ b/scripts/kconfig/util.c
> @@ -93,9 +93,10 @@ static char *env_expand_n(const char *name, size_t n)
>  }
>  
>  /*
> - * Expand environments embedded in the string given in argument. Environments
> - * to be expanded shall be prefixed by a '$'. Unknown environment expands to
> - * the empty string.
> + * Expand environments and functions embedded in the string given in argument.
> + * Environments to be expanded shall be prefixed by a '$'. Functions to be
> + * evaluated shall be surrounded by $(). Unknown environment/function expands
> + * to the empty string.
>   */
>  char *expand_string_value(const char *in)
>  {
> @@ -113,11 +114,40 @@ char *expand_string_value(const char *in)
>  	while ((p = strchr(in, '$'))) {
>  		char *new;
>  
> -		q = p + 1;
> -		while (isalnum(*q) || *q == '_')
> -			q++;
> -
> -		new = env_expand_n(p + 1, q - p - 1);
> +		/*
> +		 * If the next character is '(', it is a function.
> +		 * Otherwise, environment.
> +		 */
> +		if (*(p + 1) == '(') {
> +			int nest = 0;
> +
> +			q = p + 2;
> +			while (1) {
> +				if (*q == '\0') {
> +					fprintf(stderr,
> +						"unterminated function: %s\n",
> +						p);
> +					new = xstrdup("");
> +					break;
> +				} else if (*q == '(') {
> +					nest++;
> +				} else if (*q == ')') {
> +					if (nest-- == 0) {
> +						new = func_eval_n(p + 2,
> +								  q - p - 2);
> +						q++;
> +						break;
> +					}
> +				}
> +				q++;
> +			}
> +		} else {
> +			q = p + 1;
> +			while (isalnum(*q) || *q == '_')
> +				q++;
> +
> +			new = env_expand_n(p + 1, q - p - 1);
> +		}
>  
>  		reslen = strlen(res) + (p - in) + strlen(new) + 1;
>  		res = xrealloc(res, reslen);
> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
> index 0d89ea6..f433ab0 100644
> --- a/scripts/kconfig/zconf.l
> +++ b/scripts/kconfig/zconf.l
> @@ -1,7 +1,7 @@
>  %option nostdinit noyywrap never-interactive full ecs
>  %option 8bit nodefault perf-report perf-report
>  %option noinput
> -%x COMMAND HELP STRING PARAM
> +%x COMMAND HELP STRING PARAM FUNCTION
>  %{
>  /*
>   * Copyright (C) 2002 Roman Zippel <zippel@linux-m68k.org>
> @@ -25,6 +25,7 @@ static struct {
>  
>  static char *text;
>  static int text_size, text_asize;
> +static int function_nest;
>  
>  struct buffer {
>  	struct buffer *parent;
> @@ -138,6 +139,12 @@ n	[$A-Za-z0-9_-]
>  		new_string();
>  		BEGIN(STRING);
>  	}
> +	"$("	{
> +		new_string();
> +		append_string(yytext, yyleng);
> +		function_nest = 0;
> +		BEGIN(FUNCTION);
> +	}
>  	\n	BEGIN(INITIAL); current_file->lineno++; return T_EOL;
>  	({n}|[/.])+	{
>  		const struct kconf_id *id = kconf_id_lookup(yytext, yyleng);
> @@ -196,6 +203,35 @@ n	[$A-Za-z0-9_-]
>  	}
>  }
>  
> +<FUNCTION>{
> +	[^()\n]* {
> +		append_string(yytext, yyleng);
> +	}
> +	"(" {
> +		append_string(yytext, yyleng);
> +		function_nest++;
> +	}
> +	")" {
> +		append_string(yytext, yyleng);
> +		if (function_nest-- == 0) {
> +			BEGIN(PARAM);
> +			yylval.string = text;
> +			return T_WORD;

T_WORD_QUOTE (which would turn into a constant symbol in most contexts)
would be better here, IMO. That would turn $(foo) into just an alias for
"$(foo)".

See below.

> +		}
> +	}
> +	\n {
> +		fprintf(stderr,
> +			"%s:%d:warning: multi-line function not supported\n",
> +			zconf_curname(), zconf_lineno());
> +		current_file->lineno++;
> +		BEGIN(INITIAL);
> +		return T_EOL;
> +	}
> +	<<EOF>>	{
> +		BEGIN(INITIAL);
> +	}
> +}
> +
>  <HELP>{
>  	[ \t]+	{
>  		ts = 0;
> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
> index 784083d..d9977de 100644
> --- a/scripts/kconfig/zconf.y
> +++ b/scripts/kconfig/zconf.y
> @@ -534,11 +534,19 @@ void conf_parse(const char *name)
>  
>  	zconf_initscan(name);
>  
> +	func_init();
>  	_menu_init();
>  
>  	if (getenv("ZCONF_DEBUG"))
>  		yydebug = 1;
>  	yyparse();
> +
> +	/*
> +	 * Currently, functions are evaluated only when Kconfig files are
> +	 * parsed. We can free functions here.
> +	 */
> +	func_exit();
> +
>  	if (yynerrs)
>  		exit(1);
>  	if (!modules_sym)
> @@ -778,4 +786,5 @@ void zconfdump(FILE *out)
>  #include "confdata.c"
>  #include "expr.c"
>  #include "symbol.c"
> +#include "function.c"
>  #include "menu.c"
> -- 
> 2.7.4
> 

Here's a simplification idea I'm throwing out there:

What about only allowing $ENV and $() within quotes, and just having
them always do simple text substitution (so that they'd indirectly
always generate T_WORD_QUOTE tokens)?

Pros:

 - Zero new syntax outside of strings (until the macro stuff).

 - Makes the behavior and limitations of the syntax obvious: You can't
   have $(foo) expand to a full expression, only to (possibly part of) a
   value. This is a good limitation, IMO, and it's already there.

 - Super simple and straightforward Kconfig implementation. All the new
   magic would happen in expand_string_value().

Neutral:

 - Just as general where it matters. Take something like

     default "$(foo)"

   If $(foo) happens to expand to "y", then that will do its usual thing
   for a bool/tristate symbol. Same for string symbols, etc. It'd just
   be a thin preprocessing step on constant symbol values.

 - The only loss in generality would be that you can no longer have
   a function expand to the name of non-constant symbol. For example,
   take the following:

	default $(foo)

   If $(foo) expands to MY_SYMBOL, then that would work as

	default MY_SYMBOL

  (I.e., it would default to the value of the symbol MY_SYMBOL.)

  With the quotes, it would instead work as

     default "MY_SYMBOL"

  IMO, the second version is less confusing, and deliberately using the
  first version seems like a bad idea (there's likely to be cleaner ways
  to do the indirection in plain Kconfig).

  This is also why I think T_WORD_QUOTE is better in the code above. It
  would make $(foo) work more like a shorthand for "$(foo)".


Cons:

 - Bit more typing to add the quotes

 - Maybe it isn't widely known that "n", "m", "y" work just like n, m, y
   for bool and tristate symbols (the latter automatically get converted
   to the former), though you see them quite a lot in Kconfig files.
   
   ("n", "foo bar", etc., are all just constant symbols. Kconfig keeps
   track of them separately from non-constant symbols. The constant
   symbols "n", "m", and "y" are predefined.)

   If we go with obligatory quotes, I volunteer to explain things in
   kconfig-language.txt at least, to make it clear why you'd see quotes
   in bool/tristate symbols using $(shell). I suspect it wouldn't be
   that tricky to figure out anyway.


What do you think?

Cheers,
Ulf

Re: [PATCH 06/23] kconfig: reference environments directly and remove 'option env=' syntax

[Ulf Magnusson]

On Sat, Feb 17, 2018 at 03:38:34AM +0900, Masahiro Yamada wrote:
> To get an environment value, Kconfig needs to define a symbol using
> "option env=" syntax.  It is tedious to add a config entry for each
> environment given that we need more environments such as 'CC', 'AS',
> 'srctree' etc. to evaluate the compiler capability in Kconfig.
> 
> Adding '$' to symbols is weird.  Kconfig can reference symbols directly
> like this:
> 
>   config FOO
>           string
>           default BAR
> 
> So, I want to use the following syntax to get environment 'BAR' from
> the system:
> 
>   config FOO
>           string
>           default $BAR

I like this idea. These bounce symbols always seemed kinda pointless to
me too (and I've seen some other Kconfig dialects that got rid of them).

Getting rid of those symbols also gets rid of the only case where
Kconfig would previously evaluate Kconfig symbols during parsing (for
'source "$ENV"'), so it makes the parsing/evaluation separation a bit
cleaner.


I wonder if we could just make the quotes mandatory though, and treat
$BAR simply as string interpolation, similar to the idea in
https://lkml.org/lkml/2018/2/17/175.

That would make the behavior clearer I think, and you wouldn't have to
do special parsing for symbols starting with $ (but see the note at the
end).

> 
> Looking at the code, the symbols prefixed with 'S' are expanded by:
>  - conf_expand_value()
>    This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
>  - expand_string_value()
>    This is used to expand strings in 'source' and 'mainmenu'
> 
> All of them are independent of user configuration, i.e. fixed values.
> So, this kind of syntax should be moved to simply take the environment.
> 
> This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
> 'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.
> 
> sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
> should be be given from the environment.
> 
> ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
> by 'default ARCH_DEFCONFIG'.
> 
> An environment can appear anywhere a symbol reference can appear.
> (It is expanded by sym_lookup().)  If an expression (which is derived
> from symbols) is a string, environments in the string are also expanded.
> 
> For example, the following code works.
> 
> Example code:
> 
>   config TOOLCHAIN_LIST
>           string
>           default "My tools: CC=$CC, AS=$AS, CPP=$CPP"
> 
> Result:
> 
>   $ make -s alldefconfig && tail -n 1 .config
>   CONFIG_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
> 
> I tested all 'make *config' for arch architectures.
> I confirmed this commit still produced the same result.
> 
> 
>  Documentation/kbuild/kconfig-language.txt |   8 --
>  Kconfig                                   |   4 -
>  Makefile                                  |   3 +-
>  arch/sh/Kconfig                           |   4 +-
>  arch/sparc/Kconfig                        |   4 +-
>  arch/tile/Kconfig                         |   2 +-
>  arch/um/Kconfig.common                    |   4 -
>  arch/x86/Kconfig                          |   4 +-
>  arch/x86/um/Kconfig                       |   4 +-
>  init/Kconfig                              |  10 +-
>  scripts/kconfig/confdata.c                |  31 +-----
>  scripts/kconfig/kconf_id.c                |   1 -
>  scripts/kconfig/lkc.h                     |   4 -
>  scripts/kconfig/menu.c                    |   3 -
>  scripts/kconfig/symbol.c                  |  84 ++++------------
>  scripts/kconfig/util.c                    | 156 +++++++++++++++++++++---------
>  scripts/kconfig/zconf.l                   |   2 +-
>  scripts/kconfig/zconf.y                   |   1 -
>  18 files changed, 144 insertions(+), 185 deletions(-)
> 
> diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
> index f5b9493..0e966e8 100644
> --- a/Documentation/kbuild/kconfig-language.txt
> +++ b/Documentation/kbuild/kconfig-language.txt
> @@ -198,14 +198,6 @@ applicable everywhere (see syntax).
>      enables the third modular state for all config symbols.
>      At most one symbol may have the "modules" option set.
>  
> -  - "env"=<value>
> -    This imports the environment variable into Kconfig. It behaves like
> -    a default, except that the value comes from the environment, this
> -    also means that the behaviour when mixing it with normal defaults is
> -    undefined at this point. The symbol is currently not exported back
> -    to the build environment (if this is desired, it can be done via
> -    another symbol).
> -
>    - "allnoconfig_y"
>      This declares the symbol as one that should have the value y when
>      using "allnoconfig". Used for symbols that hide other symbols.
> diff --git a/Kconfig b/Kconfig
> index 8c4c1cb..e6ece5b 100644
> --- a/Kconfig
> +++ b/Kconfig
> @@ -5,8 +5,4 @@
>  #
>  mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
>  
> -config SRCARCH
> -	string
> -	option env="SRCARCH"
> -
>  source "arch/$SRCARCH/Kconfig"
> diff --git a/Makefile b/Makefile
> index 94a957e..9a8c689 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -275,7 +275,8 @@ include scripts/Kbuild.include
>  # Read KERNELRELEASE from include/config/kernel.release (if it exists)
>  KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
>  KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
> -export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
> +UNAME_RELEASE := $(shell uname --release)
> +export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
>  
>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>  # first, and if a usermode build is happening, the "ARCH=um" on the command
> diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
> index 97fe293..89fc2f6 100644
> --- a/arch/sh/Kconfig
> +++ b/arch/sh/Kconfig
> @@ -57,7 +57,7 @@ config SUPERH
>  	  <http://www.linux-sh.org/>.
>  
>  config SUPERH32
> -	def_bool ARCH = "sh"
> +	def_bool $ARCH = "sh"

With the idea above, this would just be

	def_bool "$ARCH" = "sh"

instead.

>  	select HAVE_KPROBES
>  	select HAVE_KRETPROBES
>  	select HAVE_IOREMAP_PROT if MMU && !X2TLB
> @@ -76,7 +76,7 @@ config SUPERH32
>  	select HAVE_CC_STACKPROTECTOR
>  
>  config SUPERH64
> -	def_bool ARCH = "sh64"
> +	def_bool $ARCH = "sh64"
>  	select HAVE_EXIT_THREAD
>  	select KALLSYMS
>  
> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
> index 6bf594a..fed3d82 100644
> --- a/arch/sparc/Kconfig
> +++ b/arch/sparc/Kconfig
> @@ -1,6 +1,6 @@
>  config 64BIT
> -	bool "64-bit kernel" if ARCH = "sparc"
> -	default ARCH = "sparc64"
> +	bool "64-bit kernel" if $ARCH = "sparc"
> +	default $ARCH = "sparc64"
>  	help
>  	  SPARC is a family of RISC microprocessors designed and marketed by
>  	  Sun Microsystems, incorporated.  They are very widely found in Sun
> diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
> index ef9d403..fed372e 100644
> --- a/arch/tile/Kconfig
> +++ b/arch/tile/Kconfig
> @@ -119,7 +119,7 @@ config HVC_TILE
>  # Building with ARCH=tilegx (or ARCH=tile) implies using the
>  # 64-bit TILE-Gx toolchain, so force CONFIG_TILEGX on.
>  config TILEGX
> -	def_bool ARCH != "tilepro"
> +	def_bool $ARCH != "tilepro"
>  	select ARCH_SUPPORTS_ATOMIC_RMW
>  	select GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
>  	select HAVE_ARCH_JUMP_LABEL
> diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
> index c68add8..07f84c8 100644
> --- a/arch/um/Kconfig.common
> +++ b/arch/um/Kconfig.common
> @@ -54,10 +54,6 @@ config HZ
>  	int
>  	default 100
>  
> -config SUBARCH
> -	string
> -	option env="SUBARCH"
> -
>  config NR_CPUS
>  	int
>  	range 1 1
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index a528c14..54d943a 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1,8 +1,8 @@
>  # SPDX-License-Identifier: GPL-2.0
>  # Select 32 or 64 bit
>  config 64BIT
> -	bool "64-bit kernel" if ARCH = "x86"
> -	default ARCH != "i386"
> +	bool "64-bit kernel" if $ARCH = "x86"
> +	default $ARCH != "i386"
>  	---help---
>  	  Say yes to build a 64-bit kernel - formerly known as x86_64
>  	  Say no to build a 32-bit kernel - formerly known as i386
> diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
> index 13ed827..d355413 100644
> --- a/arch/x86/um/Kconfig
> +++ b/arch/x86/um/Kconfig
> @@ -16,8 +16,8 @@ config UML_X86
>  	select GENERIC_FIND_FIRST_BIT
>  
>  config 64BIT
> -	bool "64-bit kernel" if SUBARCH = "x86"
> -	default SUBARCH != "i386"
> +	bool "64-bit kernel" if $SUBARCH = "x86"
> +	default $SUBARCH != "i386"
>  
>  config X86_32
>  	def_bool !64BIT
> diff --git a/init/Kconfig b/init/Kconfig
> index df18492..b4814e6 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1,11 +1,3 @@
> -config ARCH
> -	string
> -	option env="ARCH"
> -
> -config KERNELVERSION
> -	string
> -	option env="KERNELVERSION"
> -
>  config DEFCONFIG_LIST
>  	string
>  	depends on !UML
> @@ -13,7 +5,7 @@ config DEFCONFIG_LIST
>  	default "/lib/modules/$UNAME_RELEASE/.config"
>  	default "/etc/kernel-config"
>  	default "/boot/config-$UNAME_RELEASE"
> -	default "$ARCH_DEFCONFIG"
> +	default ARCH_DEFCONFIG
>  	default "arch/$ARCH/defconfig"
>  
>  config CONSTRUCTORS
> diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
> index df26c7b..98c2014 100644
> --- a/scripts/kconfig/confdata.c
> +++ b/scripts/kconfig/confdata.c
> @@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
>  	return name ? name : "include/config/auto.conf";
>  }
>  
> -static char *conf_expand_value(const char *in)
> -{
> -	struct symbol *sym;
> -	const char *src;
> -	static char res_value[SYMBOL_MAXLENGTH];
> -	char *dst, name[SYMBOL_MAXLENGTH];
> -
> -	res_value[0] = 0;
> -	dst = name;
> -	while ((src = strchr(in, '$'))) {
> -		strncat(res_value, in, src - in);
> -		src++;
> -		dst = name;
> -		while (isalnum(*src) || *src == '_')
> -			*dst++ = *src++;
> -		*dst = 0;
> -		sym = sym_lookup(name, 0);
> -		sym_calc_value(sym);
> -		strcat(res_value, sym_get_string_value(sym));
> -		in = src;
> -	}
> -	strcat(res_value, in);
> -
> -	return res_value;
> -}
> -
>  char *conf_get_default_confname(void)
>  {
>  	struct stat buf;
>  	static char fullname[PATH_MAX+1];
>  	char *env, *name;
>  
> -	name = conf_expand_value(conf_defname);
> +	name = expand_string_value(conf_defname);
>  	env = getenv(SRCTREE);
>  	if (env) {
>  		sprintf(fullname, "%s/%s", env, name);
> @@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
>  			if (expr_calc_value(prop->visible.expr) == no ||
>  			    prop->expr->type != E_SYMBOL)
>  				continue;
> -			name = conf_expand_value(prop->expr->left.sym->name);
> +			sym_calc_value(prop->expr->left.sym);
> +			name = sym_get_string_value(prop->expr->left.sym);
>  			in = zconf_fopen(name);
>  			if (in) {
>  				conf_message(_("using defaults found in %s"),
> diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
> index 3ea9c5f..b3e0ea0 100644
> --- a/scripts/kconfig/kconf_id.c
> +++ b/scripts/kconfig/kconf_id.c
> @@ -32,7 +32,6 @@ static struct kconf_id kconf_id_array[] = {
>  	{ "on",			T_ON,			TF_PARAM },
>  	{ "modules",		T_OPT_MODULES,		TF_OPTION },
>  	{ "defconfig_list",	T_OPT_DEFCONFIG_LIST,	TF_OPTION },
> -	{ "env",		T_OPT_ENV,		TF_OPTION },
>  	{ "allnoconfig_y",	T_OPT_ALLNOCONFIG_Y,	TF_OPTION },
>  };
>  
> diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
> index fb2503a..0ff3256 100644
> --- a/scripts/kconfig/lkc.h
> +++ b/scripts/kconfig/lkc.h
> @@ -58,7 +58,6 @@ enum conf_def_mode {
>  
>  #define T_OPT_MODULES		1
>  #define T_OPT_DEFCONFIG_LIST	2
> -#define T_OPT_ENV		3
>  #define T_OPT_ALLNOCONFIG_Y	4

Decrease T_OPT_ALLNOCONFIG_Y to 3 as well? Looks like a bitmask to me
otherwise.

>  
>  struct kconf_id {
> @@ -134,9 +133,6 @@ void str_printf(struct gstr *gs, const char *fmt, ...);
>  const char *str_get(struct gstr *gs);
>  
>  /* symbol.c */
> -extern struct expr *sym_env_list;
> -
> -void sym_init(void);
>  void sym_clear_all_valid(void);
>  struct symbol *sym_choice_default(struct symbol *sym);
>  const char *sym_get_string_default(struct symbol *sym);
> diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
> index 36cd3e1..a9d0ccc 100644
> --- a/scripts/kconfig/menu.c
> +++ b/scripts/kconfig/menu.c
> @@ -214,9 +214,6 @@ void menu_add_option(int token, char *arg)
>  			zconf_error("trying to redefine defconfig symbol");
>  		sym_defconfig_list->flags |= SYMBOL_AUTO;
>  		break;
> -	case T_OPT_ENV:
> -		prop_add_env(arg);
> -		break;
>  	case T_OPT_ALLNOCONFIG_Y:
>  		current_entry->sym->flags |= SYMBOL_ALLNOCONFIG_Y;
>  		break;
> diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
> index e4ccf56..96ea8a9 100644
> --- a/scripts/kconfig/symbol.c
> +++ b/scripts/kconfig/symbol.c
> @@ -33,33 +33,6 @@ struct symbol *sym_defconfig_list;
>  struct symbol *modules_sym;
>  tristate modules_val;
>  
> -struct expr *sym_env_list;
> -
> -static void sym_add_default(struct symbol *sym, const char *def)
> -{
> -	struct property *prop = prop_alloc(P_DEFAULT, sym);
> -
> -	prop->expr = expr_alloc_symbol(sym_lookup(def, SYMBOL_CONST));
> -}
> -
> -void sym_init(void)
> -{
> -	struct symbol *sym;
> -	struct utsname uts;
> -	static bool inited = false;
> -
> -	if (inited)
> -		return;
> -	inited = true;
> -
> -	uname(&uts);
> -
> -	sym = sym_lookup("UNAME_RELEASE", 0);
> -	sym->type = S_STRING;
> -	sym->flags |= SYMBOL_AUTO;
> -	sym_add_default(sym, uts.release);
> -}
> -
>  enum symbol_type sym_get_type(struct symbol *sym)
>  {
>  	enum symbol_type type = sym->type;
> @@ -828,28 +801,38 @@ static unsigned strhash(const char *s)
>  
>  struct symbol *sym_lookup(const char *name, int flags)
>  {
> -	struct symbol *symbol;
> +	struct symbol *symbol = NULL;
>  	char *new_name;
>  	int hash;
>  
>  	if (name) {
> -		if (name[0] && !name[1]) {
> -			switch (name[0]) {
> -			case 'y': return &symbol_yes;
> -			case 'm': return &symbol_mod;
> -			case 'n': return &symbol_no;
> +		new_name = expand_string_value(name);
> +		if (new_name[0] && !new_name[1]) {
> +			switch (new_name[0]) {
> +			case 'y':
> +				symbol = &symbol_yes;
> +				break;
> +			case 'm':
> +				symbol = &symbol_mod;
> +				break;
> +			case 'n':
> +				symbol = &symbol_no;
> +				break;
> +			}
> +			if (symbol) {
> +				free(new_name);
> +				return symbol;
>  			}
>  		}
> -		hash = strhash(name) % SYMBOL_HASHSIZE;
> +		hash = strhash(new_name) % SYMBOL_HASHSIZE;
>  
>  		for (symbol = symbol_hash[hash]; symbol; symbol = symbol->next) {
>  			if (symbol->name &&
> -			    !strcmp(symbol->name, name) &&
> +			    !strcmp(symbol->name, new_name) &&
>  			    (flags ? symbol->flags & flags
>  				   : !(symbol->flags & (SYMBOL_CONST|SYMBOL_CHOICE))))
>  				return symbol;
>  		}
> -		new_name = xstrdup(name);
>  	} else {
>  		new_name = NULL;
>  		hash = 0;
> @@ -1336,32 +1319,3 @@ const char *prop_get_type_name(enum prop_type type)
>  	}
>  	return "unknown";
>  }
> -
> -static void prop_add_env(const char *env)
> -{
> -	struct symbol *sym, *sym2;
> -	struct property *prop;
> -	char *p;
> -
> -	sym = current_entry->sym;
> -	sym->flags |= SYMBOL_AUTO;
> -	for_all_properties(sym, prop, P_ENV) {
> -		sym2 = prop_get_symbol(prop);
> -		if (strcmp(sym2->name, env))
> -			menu_warn(current_entry, "redefining environment symbol from %s",
> -				  sym2->name);
> -		return;
> -	}
> -
> -	prop = prop_alloc(P_ENV, sym);
> -	prop->expr = expr_alloc_symbol(sym_lookup(env, SYMBOL_CONST));
> -
> -	sym_env_list = expr_alloc_one(E_LIST, sym_env_list);
> -	sym_env_list->right.sym = sym;
> -
> -	p = getenv(env);
> -	if (p)
> -		sym_add_default(sym, p);
> -	else
> -		menu_warn(current_entry, "environment variable %s undefined", env);
> -}
> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
> index 22201a4..dddf85b 100644
> --- a/scripts/kconfig/util.c
> +++ b/scripts/kconfig/util.c
> @@ -8,16 +8,98 @@
>  #include <stdarg.h>
>  #include <stdlib.h>
>  #include <string.h>
> +
> +#include "list.h"
>  #include "lkc.h"
>  
> +static LIST_HEAD(env_list);
> +
> +struct env {
> +	char *name;
> +	char *value;
> +	struct list_head node;
> +};
> +
> +static struct env *env_list_lookup(const char *name)
> +{
> +	struct env *e;
> +
> +	list_for_each_entry(e, &env_list, node) {
> +		if (!strcmp(name, e->name))
> +			return e;
> +	}
> +
> +	return NULL;
> +}
> +
> +static void env_list_add(const char *name, const char *value)
> +{
> +	struct env *e;
> +
> +	e = xmalloc(sizeof(*e));
> +	e->name = xstrdup(name);
> +	e->value = xstrdup(value);
> +
> +	list_add_tail(&e->node, &env_list);
> +}
> +
> +static void env_list_del(struct env *e)
> +{
> +	list_del(&e->node);
> +	free(e->name);
> +	free(e->value);
> +	free(e);
> +}
> +
> +/* the returned pointer must be freed when done */
> +static char *env_expand(const char *name)
> +{
> +	struct env *e;
> +	const char *value;
> +
> +	e = env_list_lookup(name);
> +	if (e)
> +		return xstrdup(e->value);
> +
> +	value = getenv(name);
> +	if (!value) {
> +		fprintf(stderr, "environment variable \"%s\" undefined\n", name);
> +		value = "";
> +	}
> +
> +	/*
> +	 * we need to remember all referenced environments.
> +	 * They will be written out to include/config/auto.conf.cmd
> +	 */
> +	env_list_add(name, value);
> +
> +	return xstrdup(value);
> +}
> +
> +/* works like env_expand, but 'name' does not need to be null-terminated */
> +static char *env_expand_n(const char *name, size_t n)
> +{
> +	char *tmp, *res;
> +
> +	tmp = xmalloc(n + 1);
> +	memcpy(tmp, name, n);
> +	*(tmp + n) = '\0';
> +
> +	res = env_expand(tmp);
> +
> +	free(tmp);
> +
> +	return res;
> +}
> +
>  /*
> - * Expand symbol's names embedded in the string given in argument. Symbols'
> - * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
> + * Expand environments embedded in the string given in argument. Environments
> + * to be expanded shall be prefixed by a '$'. Unknown environment expands to
>   * the empty string.
>   */
>  char *expand_string_value(const char *in)
>  {
> -	const char *src;
> +	const char *p, *q;
>  	char *res;
>  	size_t reslen;
>  
> @@ -25,39 +107,28 @@ char *expand_string_value(const char *in)
>  	 * Note: 'in' might come from a token that's about to be
>  	 * freed, so make sure to always allocate a new string
>  	 */
> -	reslen = strlen(in) + 1;
> -	res = xmalloc(reslen);
> -	res[0] = '\0';
> -
> -	while ((src = strchr(in, '$'))) {
> -		char *p, name[SYMBOL_MAXLENGTH];
> -		const char *symval = "";
> -		struct symbol *sym;
> -		size_t newlen;
> -
> -		strncat(res, in, src - in);
> -		src++;
> -
> -		p = name;
> -		while (isalnum(*src) || *src == '_')
> -			*p++ = *src++;
> -		*p = '\0';
> -
> -		sym = sym_find(name);
> -		if (sym != NULL) {
> -			sym_calc_value(sym);
> -			symval = sym_get_string_value(sym);
> -		}
> +	res = xmalloc(1);
> +	*res = '\0';
>  
> -		newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
> -		if (newlen > reslen) {
> -			reslen = newlen;
> -			res = xrealloc(res, reslen);
> -		}
> +	while ((p = strchr(in, '$'))) {
> +		char *new;
> +
> +		q = p + 1;
> +		while (isalnum(*q) || *q == '_')
> +			q++;
>  
> -		strcat(res, symval);
> -		in = src;
> +		new = env_expand_n(p + 1, q - p - 1);
> +
> +		reslen = strlen(res) + (p - in) + strlen(new) + 1;
> +		res = xrealloc(res, reslen);
> +		strncat(res, in, p - in);
> +		strcat(res, new);
> +		free(new);
> +		in = q;
>  	}
> +
> +	reslen = strlen(res) + strlen(in) + 1;
> +	res = xrealloc(res, reslen);
>  	strcat(res, in);
>  
>  	return res;
> @@ -87,8 +158,7 @@ struct file *file_lookup(const char *name)
>  /* write a dependency file as used by kbuild to track dependencies */
>  int file_write_dep(const char *name)
>  {
> -	struct symbol *sym, *env_sym;
> -	struct expr *e;
> +	struct env *env, *tmp;
>  	struct file *file;
>  	FILE *out;
>  
> @@ -107,20 +177,12 @@ int file_write_dep(const char *name)
>  	fprintf(out, "\n%s: \\\n"
>  		     "\t$(deps_config)\n\n", conf_get_autoconfig_name());
>  
> -	expr_list_for_each_sym(sym_env_list, e, sym) {
> -		struct property *prop;
> -		const char *value;
> -
> -		prop = sym_get_env_prop(sym);
> -		env_sym = prop_get_symbol(prop);
> -		if (!env_sym)
> -			continue;
> -		value = getenv(env_sym->name);
> -		if (!value)
> -			value = "";
> -		fprintf(out, "ifneq \"$(%s)\" \"%s\"\n", env_sym->name, value);
> +	list_for_each_entry_safe(env, tmp, &env_list, node) {
> +		fprintf(out, "ifneq \"$(%s)\" \"%s\"\n",
> +			env->name, getenv(env->name) ?: "");
>  		fprintf(out, "%s: FORCE\n", conf_get_autoconfig_name());
>  		fprintf(out, "endif\n");
> +		env_list_del(env);
>  	}
>  
>  	fprintf(out, "\n$(deps_config): ;\n");
> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
> index 02de6fe..0d89ea6 100644
> --- a/scripts/kconfig/zconf.l
> +++ b/scripts/kconfig/zconf.l
> @@ -75,7 +75,7 @@ static void warn_ignored_character(char chr)
>  }
>  %}
>  
> -n	[A-Za-z0-9_-]
> +n	[$A-Za-z0-9_-]
>  
>  %%
>  	int str = 0;
> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
> index 262c464..784083d 100644
> --- a/scripts/kconfig/zconf.y
> +++ b/scripts/kconfig/zconf.y
> @@ -534,7 +534,6 @@ void conf_parse(const char *name)
>  
>  	zconf_initscan(name);
>  
> -	sym_init();
>  	_menu_init();
>  
>  	if (getenv("ZCONF_DEBUG"))
> -- 
> 2.7.4
> 

Another option if we want to allow $ENV outside of quotes (even though I
prefer mandatory quotes) would be to transform it early into "$ENV" by
returning a T_WORD_QUOTE token for it.

I think you could then get expand_string_value() to handle both the
quoted and unquoted cases.

Cheers,
Ulf

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Sam Ravnborg]

Hi Masahiro.

On Sat, Feb 17, 2018 at 03:38:28AM +0900, Masahiro Yamada wrote:
> I brushed up the implementation in this version.
> 
> In the previous RFC, CC_HAS_ was described by using 'option shell=',
> like this:
> 
> config CC_HAS_STACKPROTECTOR
>         bool
>         option shell="$CC -Werror -fstack-protector -c -x c /dev/null"
> 
> After I thought a bit more, the following syntax is more grammatical,
> and flexible.
> 
> config CC_HAS_STACKPROTECTOR
>         bool
>         default $(shell $CC -Werror -fstack-protector -c -x c /dev/null)

Looks good - but maybe we should go one step further.

So we in the syntax explicit handles:
- shell commands
- other commands, defined as strings
- environment variables
- config variables

Each case is explicit - so the reader is not confused what is used when.

$(shell foo)	- output of the shell command foo. Uses $SHELL as the shell.
		  May include optional paramters.
		  foo may be a config variable referenced using ${} or a config variable prefixed with $
		Example:

		config BUILD_DIR
			string
			default $(shell cd ${objtree}; pwd)

$(call bar)	- output of the bar command that may take optional parameters.
		  bar may be a text string, a config variable or an environment variable
		   The definition of bar may reference the parameters using $(1), $(2)
		   In this context a config variable needs to be prefixed with $

		Example:

		config reverse
			string
			default $(2) $(1)

		config NEW_ORDER
			string
			$(call $reverse, A, B)	# Will assign REVERSE the value "B A"


		Example2:

		config CC_OPTION
			string
			default $(shell ${srctree}/scripts/cc-option ${CC} $(1) $(2))

		config CC_OPTIMIZE
			string
			$(call $CC_OPTION, -Oz, -Os)


${FOO}		- environment variable

The above is inspired by how make implement similar functionality.

I'm not happy that we in one context can reference CONFIG variables
directly, but inside the $(call ...) and $(shell ...) needs the $ prefix.
But I could not come up with something un-ambigious where this could be avoided.

The above proposal include the functionality of the macro stuff proposed in this patch-set.
But with a simpler syntax and we keep all the other kconfig logic (depends on etc) - so
users will not be limited in their creativity.

> Current limitations:
> 
> Dependency on outside scripts.
> Inter-option dependency:
> Functions are evaluated statically:

Same limitations exists with the syntax suggested above.

	Sam

Re: [PATCH 11/23] kconfig: add 'shell-stdout' function

[Ulf Magnusson]

On Fri, Feb 16, 2018 at 11:17:52AM -0800, Linus Torvalds wrote:
> On Fri, Feb 16, 2018 at 10:38 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
> > This is the second built-in function, which retrieves the first line
> > of stdout from the given shell command.
> 
> This is the only part I really don't much like in your patch series.
> 
> Most of it is just lovely and looks very nice and powerful, but the
> difference between "$(shell ..." and "$(shell-stdout ..." to me is
> bvery ugly.
> 
> Can we *please* make "shell-stdout" go away, and make this just be "shell"?
> 
> The rule would be very simple:
> 
>  - if the result of the shell command is a failure, the result is 'n'
> 
>  - otherwise, the result is the first line of stdout
> 
>  - if the result is empty, we replace it with 'y'.
> 
> So doing $(shell true) would be 100% equivalent to $(shell echo y),
> and you could still do that
> 
>           default $(shell $CC --version | grep -q gcc)
> 
> because it would just automatically do the right thing.
> 
> Basically, the only difference is how $(shell ) works in the success
> case: the result won't necessarily be 'y', it will be whatever output.
> But if you want to always turn it into 'y' (say, you don't have a "-q"
> flag for the grep equivalent above), you can always do so with
> 
>           default $(shell $CC --version | noqgrep gcc > /dev/null)
> 
> So it seems to me that there is never any fundamental reason why we'd
> want both "shell" and "shell-stdout", since "shell-stdout" is
> fundamentally more powerful than "shell", and can always be used as
> such (and just renamed to "shell").
> 
> Because I really think that it's just much prettier and more intuitive
> to be able to say
> 
>         default "/boot/config-$(shell uname --release)"
> 
> without that "-stdout" thing.
> 
> Hmm?
> 
> But I do want to say how much I liked this series. Just this part
> seemed to result in uglier Kconfig scripts.
> 
>          Linus

Could there be cases where you'd legitimately want to put empty output
from a command in a string (that would be common enough to matter)?
That'd get messier with the above rule, as it never generates an empty
string as output.

For an environment variable, stuff like prefixes come to mind, but I
can't think of anything for a command. I'm more familiar with Kconfig
than the rest of the kernel build system though.


Would you still be as opposed (or more opposed...) to having two
functions if they were called something like 'success' and 'stdout'
instead?

This reads pretty naturally to me:

	config CC_IS_GCC
		def_bool "$(success $CC --version | grep gcc)"

As does this:

	default "/boot/config-$(stdout uname --release)"

The rule for $(success ...) would be that it's textually replaced by "y"
if the exit status of the command is 0, and with "n" in all other cases.

$(stdout ...) would be textually replaced by the first line from stdout.
Maybe it'd be helpful to spit out a warning if the exit status is non-0.

All functions would just do dumb and simple-to-understand text
replacement, and all the interpretation would happen later in the normal
way. Enforcing the quotes would make this behavior obvious. That would
indirectly turn the expanded values into constant symbols internally in
Kconfig.

Thoughts?

Cheers,
Ulf

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Ulf Magnusson]

Hello,

On Sun, Feb 18, 2018 at 11:13 PM, Sam Ravnborg <sam@ravnborg.org> wrote:
> Hi Masahiro.
>
> On Sat, Feb 17, 2018 at 03:38:28AM +0900, Masahiro Yamada wrote:
>> I brushed up the implementation in this version.
>>
>> In the previous RFC, CC_HAS_ was described by using 'option shell=',
>> like this:
>>
>> config CC_HAS_STACKPROTECTOR
>>         bool
>>         option shell="$CC -Werror -fstack-protector -c -x c /dev/null"
>>
>> After I thought a bit more, the following syntax is more grammatical,
>> and flexible.
>>
>> config CC_HAS_STACKPROTECTOR
>>         bool
>>         default $(shell $CC -Werror -fstack-protector -c -x c /dev/null)
>
> Looks good - but maybe we should go one step further.
>
> So we in the syntax explicit handles:
> - shell commands
> - other commands, defined as strings
> - environment variables
> - config variables
>
> Each case is explicit - so the reader is not confused what is used when.
>
> $(shell foo)    - output of the shell command foo. Uses $SHELL as the shell.
>                   May include optional paramters.
>                   foo may be a config variable referenced using ${} or a config variable prefixed with $
>                 Example:
>
>                 config BUILD_DIR
>                         string
>                         default $(shell cd ${objtree}; pwd)
>
> $(call bar)     - output of the bar command that may take optional parameters.
>                   bar may be a text string, a config variable or an environment variable
>                    The definition of bar may reference the parameters using $(1), $(2)
>                    In this context a config variable needs to be prefixed with $
>
>                 Example:
>
>                 config reverse
>                         string
>                         default $(2) $(1)
>
>                 config NEW_ORDER
>                         string
>                         $(call $reverse, A, B)  # Will assign REVERSE the value "B A"
>
>
>                 Example2:
>
>                 config CC_OPTION
>                         string
>                         default $(shell ${srctree}/scripts/cc-option ${CC} $(1) $(2))
>
>                 config CC_OPTIMIZE
>                         string
>                         $(call $CC_OPTION, -Oz, -Os)
>
>
> ${FOO}          - environment variable
>
> The above is inspired by how make implement similar functionality.
>
> I'm not happy that we in one context can reference CONFIG variables
> directly, but inside the $(call ...) and $(shell ...) needs the $ prefix.
> But I could not come up with something un-ambigious where this could be avoided.

I think we should be careful about allowing references to config
symbols. It mixes up the parsing and evaluation phases, since $() is
expanded during parsing (which I consider a feature and think is
needed to retain sanity).

Patch 06/23 removes the last existing instance of symbol references in
strings by getting rid of 'option env'. That's an improvement to me.
We shouldn't add it back.

>
> The above proposal include the functionality of the macro stuff proposed in this patch-set.
> But with a simpler syntax and we keep all the other kconfig logic (depends on etc) - so
> users will not be limited in their creativity.
>
>> Current limitations:
>>
>> Dependency on outside scripts.
>> Inter-option dependency:
>> Functions are evaluated statically:
>
> Same limitations exists with the syntax suggested above.
>
>         Sam

Cheers,
Ulf

Re: [PATCH 07/23] kconfig: add function support and implement 'shell' function

[Masahiro Yamada]

2018-02-18 1:16 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> On Sat, Feb 17, 2018 at 03:38:35AM +0900, Masahiro Yamada wrote:
>> This commit adds a new concept 'function' to Kconfig.  A function call
>> resembles a variable reference with arguments, and looks like this:
>>
>>   $(function arg1, arg2, arg3, ...)
>>
>> (Actually, this syntax was inspired by Makefile.)
>>
>> Real examples will look like this:
>>
>>   $(shell true)
>>   $(cc-option -fstackprotector)
>>
>> This commit adds the basic infrastructure to add, delete, evaluate
>> functions.
>>
>> Also, add the first built-in function $(shell ...).  This evaluates
>> to 'y' if the given command exits with 0, 'n' otherwise.
>>
>> I am also planning to support user-defined functions (a.k.a 'macro')
>> for cases where hard-coding is not preferred.
>>
>> If you want to try this feature, the hello-world code is someting below.
>>
>> Example code:
>>
>>   config CC_IS_GCC
>>           bool
>>           default $(shell $CC --version | grep -q gcc)
>>
>>   config CC_IS_CLANG
>>           bool
>>           default $(shell $CC --version | grep -q clang)
>>
>>   config CC_HAS_OZ
>>           bool
>>           default $(shell $CC -Werror -Oz -c -x c /dev/null -o /dev/null)
>>
>> Result:
>>
>>   $ make -s alldefconfig && tail -n 3 .config
>>   CONFIG_CC_IS_GCC=y
>>   # CONFIG_CC_IS_CLANG is not set
>>   # CONFIG_CC_HAS_OZ is not set
>>
>>   $ make CC=clang -s alldefconfig && tail -n 3 .config
>>   # CONFIG_CC_IS_GCC is not set
>>   CONFIG_CC_IS_CLANG=y
>>   CONFIG_CC_HAS_OZ=y
>>
>> A function call can appear anywhere a symbol reference can appear.
>> So, the following code is possible.
>>
>> Example code:
>>
>>   config CC_NAME
>>           string
>>           default "gcc" if $(shell $CC --version | grep -q gcc)
>>           default "clang" if $(shell $CC --version | grep -q clang)
>>           default "unknown compiler"
>>
>> Result:
>>
>>   $ make -s alldefconfig && tail -n 1 .config
>>   CONFIG_CC_NAME="gcc"
>>
>>   $ make CC=clang -s alldefconfig && tail -n 1 .config
>>   CONFIG_CC_NAME="clang"
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> Reminder for myself:
>> Update Documentation/kbuild/kconfig-language.txt
>>
>>
>>  scripts/kconfig/function.c  | 149 ++++++++++++++++++++++++++++++++++++++++++++
>>  scripts/kconfig/lkc_proto.h |   5 ++
>>  scripts/kconfig/util.c      |  46 +++++++++++---
>>  scripts/kconfig/zconf.l     |  38 ++++++++++-
>>  scripts/kconfig/zconf.y     |   9 +++
>>  5 files changed, 238 insertions(+), 9 deletions(-)
>>  create mode 100644 scripts/kconfig/function.c
>>
>> diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
>> new file mode 100644
>> index 0000000..60e59be
>> --- /dev/null
>> +++ b/scripts/kconfig/function.c
>> @@ -0,0 +1,149 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +//
>> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
>> +
>> +#include <stdio.h>
>> +#include <stdlib.h>
>> +#include <string.h>
>> +
>> +#include "list.h"
>> +
>> +#define FUNCTION_MAX_ARGS            10
>> +
>> +static LIST_HEAD(function_list);
>> +
>> +struct function {
>> +     const char *name;
>> +     char *(*func)(int argc, char *argv[]);
>> +     struct list_head node;
>> +};
>> +
>> +static struct function *func_lookup(const char *name)
>> +{
>> +     struct function *f;
>> +
>> +     list_for_each_entry(f, &function_list, node) {
>> +             if (!strcmp(name, f->name))
>> +                     return f;
>> +     }
>> +
>> +     return NULL;
>> +}
>> +
>> +static void func_add(const char *name, char *(*func)(int argc, char *argv[]))
>> +{
>> +     struct function *f;
>> +
>> +     f = func_lookup(name);
>> +     if (f) {
>> +             fprintf(stderr, "%s: function already exists. ignored.\n", name);
>> +             return;
>> +     }
>> +
>> +     f = xmalloc(sizeof(*f));
>> +     f->name = name;
>> +     f->func = func;
>> +
>> +     list_add_tail(&f->node, &function_list);
>> +}
>> +
>> +static void func_del(struct function *f)
>> +{
>> +     list_del(&f->node);
>> +     free(f);
>> +}
>> +
>> +static char *func_call(int argc, char *argv[])
>> +{
>> +     struct function *f;
>> +
>> +     f = func_lookup(argv[0]);
>> +     if (!f) {
>> +             fprintf(stderr, "%s: function not found\n", argv[0]);
>> +             return NULL;
>> +     }
>> +
>> +     return f->func(argc, argv);
>> +}
>> +
>> +static char *func_eval(const char *func)
>> +{
>> +     char *expanded, *saveptr, *str, *token, *res;
>> +     const char *delim;
>> +     int argc = 0;
>> +     char *argv[FUNCTION_MAX_ARGS];
>> +
>> +     expanded = expand_string_value(func);
>> +
>> +     str = expanded;
>> +     delim = " ";
>> +
>> +     while ((token = strtok_r(str, delim, &saveptr))) {
>> +             argv[argc++] = token;
>> +             str = NULL;
>> +             delim = ",";
>> +     }
>> +
>> +     res = func_call(argc, argv);
>> +
>> +     free(expanded);
>> +
>> +     return res ?: xstrdup("");
>> +}
>> +
>> +char *func_eval_n(const char *func, size_t n)
>> +{
>> +     char *tmp, *res;
>> +
>> +     tmp = xmalloc(n + 1);
>> +     memcpy(tmp, func, n);
>> +     *(tmp + n) = '\0';
>> +
>> +     res = func_eval(tmp);
>> +
>> +     free(tmp);
>> +
>> +     return res;
>> +}
>> +
>> +/* built-in functions */
>> +static char *do_shell(int argc, char *argv[])
>> +{
>> +     static const char *pre = "(";
>> +     static const char *post = ") >/dev/null 2>&1";
>> +     char *cmd;
>> +     int ret;
>> +
>> +     if (argc != 2)
>> +             return NULL;
>> +
>> +     /*
>> +      * Surround the command with ( ) in case it is piped commands.
>> +      * Also, redirect stdout and stderr to /dev/null.
>> +      */
>> +     cmd = xmalloc(strlen(pre) + strlen(argv[1]) + strlen(post) + 1);
>> +     strcpy(cmd, pre);
>> +     strcat(cmd, argv[1]);
>> +     strcat(cmd, post);
>> +
>> +     ret = system(cmd);
>> +
>> +     free(cmd);
>> +
>> +     return xstrdup(ret == 0 ? "y" : "n");
>> +}
>> +
>> +void func_init(void)
>> +{
>> +     /* register built-in functions */
>> +     func_add("shell", do_shell);
>> +}
>> +
>> +void func_exit(void)
>> +{
>> +     struct function *f, *tmp;
>> +
>> +     /* unregister all functions */
>> +     list_for_each_entry_safe(f, tmp, &function_list, node)
>> +             func_del(f);
>> +}
>> diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
>> index 9884adc..09a4f53 100644
>> --- a/scripts/kconfig/lkc_proto.h
>> +++ b/scripts/kconfig/lkc_proto.h
>> @@ -48,5 +48,10 @@ const char * sym_get_string_value(struct symbol *sym);
>>
>>  const char * prop_get_type_name(enum prop_type type);
>>
>> +/* function.c */
>> +char *func_eval_n(const char *func, size_t n);
>> +void func_init(void);
>> +void func_exit(void);
>> +
>>  /* expr.c */
>>  void expr_print(struct expr *e, void (*fn)(void *, struct symbol *, const char *), void *data, int prevtoken);
>> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
>> index dddf85b..ed89fb9 100644
>> --- a/scripts/kconfig/util.c
>> +++ b/scripts/kconfig/util.c
>> @@ -93,9 +93,10 @@ static char *env_expand_n(const char *name, size_t n)
>>  }
>>
>>  /*
>> - * Expand environments embedded in the string given in argument. Environments
>> - * to be expanded shall be prefixed by a '$'. Unknown environment expands to
>> - * the empty string.
>> + * Expand environments and functions embedded in the string given in argument.
>> + * Environments to be expanded shall be prefixed by a '$'. Functions to be
>> + * evaluated shall be surrounded by $(). Unknown environment/function expands
>> + * to the empty string.
>>   */
>>  char *expand_string_value(const char *in)
>>  {
>> @@ -113,11 +114,40 @@ char *expand_string_value(const char *in)
>>       while ((p = strchr(in, '$'))) {
>>               char *new;
>>
>> -             q = p + 1;
>> -             while (isalnum(*q) || *q == '_')
>> -                     q++;
>> -
>> -             new = env_expand_n(p + 1, q - p - 1);
>> +             /*
>> +              * If the next character is '(', it is a function.
>> +              * Otherwise, environment.
>> +              */
>> +             if (*(p + 1) == '(') {
>> +                     int nest = 0;
>> +
>> +                     q = p + 2;
>> +                     while (1) {
>> +                             if (*q == '\0') {
>> +                                     fprintf(stderr,
>> +                                             "unterminated function: %s\n",
>> +                                             p);
>> +                                     new = xstrdup("");
>> +                                     break;
>> +                             } else if (*q == '(') {
>> +                                     nest++;
>> +                             } else if (*q == ')') {
>> +                                     if (nest-- == 0) {
>> +                                             new = func_eval_n(p + 2,
>> +                                                               q - p - 2);
>> +                                             q++;
>> +                                             break;
>> +                                     }
>> +                             }
>> +                             q++;
>> +                     }
>> +             } else {
>> +                     q = p + 1;
>> +                     while (isalnum(*q) || *q == '_')
>> +                             q++;
>> +
>> +                     new = env_expand_n(p + 1, q - p - 1);
>> +             }
>>
>>               reslen = strlen(res) + (p - in) + strlen(new) + 1;
>>               res = xrealloc(res, reslen);
>> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
>> index 0d89ea6..f433ab0 100644
>> --- a/scripts/kconfig/zconf.l
>> +++ b/scripts/kconfig/zconf.l
>> @@ -1,7 +1,7 @@
>>  %option nostdinit noyywrap never-interactive full ecs
>>  %option 8bit nodefault perf-report perf-report
>>  %option noinput
>> -%x COMMAND HELP STRING PARAM
>> +%x COMMAND HELP STRING PARAM FUNCTION
>>  %{
>>  /*
>>   * Copyright (C) 2002 Roman Zippel <zippel@linux-m68k.org>
>> @@ -25,6 +25,7 @@ static struct {
>>
>>  static char *text;
>>  static int text_size, text_asize;
>> +static int function_nest;
>>
>>  struct buffer {
>>       struct buffer *parent;
>> @@ -138,6 +139,12 @@ n        [$A-Za-z0-9_-]
>>               new_string();
>>               BEGIN(STRING);
>>       }
>> +     "$("    {
>> +             new_string();
>> +             append_string(yytext, yyleng);
>> +             function_nest = 0;
>> +             BEGIN(FUNCTION);
>> +     }
>>       \n      BEGIN(INITIAL); current_file->lineno++; return T_EOL;
>>       ({n}|[/.])+     {
>>               const struct kconf_id *id = kconf_id_lookup(yytext, yyleng);
>> @@ -196,6 +203,35 @@ n        [$A-Za-z0-9_-]
>>       }
>>  }
>>
>> +<FUNCTION>{
>> +     [^()\n]* {
>> +             append_string(yytext, yyleng);
>> +     }
>> +     "(" {
>> +             append_string(yytext, yyleng);
>> +             function_nest++;
>> +     }
>> +     ")" {
>> +             append_string(yytext, yyleng);
>> +             if (function_nest-- == 0) {
>> +                     BEGIN(PARAM);
>> +                     yylval.string = text;
>> +                     return T_WORD;
>
> T_WORD_QUOTE (which would turn into a constant symbol in most contexts)
> would be better here, IMO. That would turn $(foo) into just an alias for
> "$(foo)".
>
> See below.
>
>> +             }
>> +     }
>> +     \n {
>> +             fprintf(stderr,
>> +                     "%s:%d:warning: multi-line function not supported\n",
>> +                     zconf_curname(), zconf_lineno());
>> +             current_file->lineno++;
>> +             BEGIN(INITIAL);
>> +             return T_EOL;
>> +     }
>> +     <<EOF>> {
>> +             BEGIN(INITIAL);
>> +     }
>> +}
>> +
>>  <HELP>{
>>       [ \t]+  {
>>               ts = 0;
>> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
>> index 784083d..d9977de 100644
>> --- a/scripts/kconfig/zconf.y
>> +++ b/scripts/kconfig/zconf.y
>> @@ -534,11 +534,19 @@ void conf_parse(const char *name)
>>
>>       zconf_initscan(name);
>>
>> +     func_init();
>>       _menu_init();
>>
>>       if (getenv("ZCONF_DEBUG"))
>>               yydebug = 1;
>>       yyparse();
>> +
>> +     /*
>> +      * Currently, functions are evaluated only when Kconfig files are
>> +      * parsed. We can free functions here.
>> +      */
>> +     func_exit();
>> +
>>       if (yynerrs)
>>               exit(1);
>>       if (!modules_sym)
>> @@ -778,4 +786,5 @@ void zconfdump(FILE *out)
>>  #include "confdata.c"
>>  #include "expr.c"
>>  #include "symbol.c"
>> +#include "function.c"
>>  #include "menu.c"
>> --
>> 2.7.4
>>
>
> Here's a simplification idea I'm throwing out there:
>
> What about only allowing $ENV and $() within quotes, and just having
> them always do simple text substitution (so that they'd indirectly
> always generate T_WORD_QUOTE tokens)?


I ended up with a new state <FUNCTION>,
but the real problem is the lexer is too ugly.

So, maybe, the solution is not to make it into a string,
but to re-write the lexer.


<STRING> and <FUNCTION> are almost the same in the sense
that whitespaces do not split tokens.

The difference is the characters to start/end with.

  ' ... '
  " ... "
  ( ... )

If we encounter with the 3 special characters,  ' , '', (,
then we enter into <STRING> state,
then it ends with ', ", ), respectively.
(but we need to take care of nesting, escaping)

Double-surrounding like  "$(cc-option -Oz)"
looks ugly to me.





> Pros:
>
>  - Zero new syntax outside of strings (until the macro stuff).
>
>  - Makes the behavior and limitations of the syntax obvious: You can't
>    have $(foo) expand to a full expression, only to (possibly part of) a
>    value. This is a good limitation, IMO, and it's already there.
>
>  - Super simple and straightforward Kconfig implementation. All the new
>    magic would happen in expand_string_value().
>
> Neutral:
>
>  - Just as general where it matters. Take something like
>
>      default "$(foo)"
>
>    If $(foo) happens to expand to "y", then that will do its usual thing
>    for a bool/tristate symbol. Same for string symbols, etc. It'd just
>    be a thin preprocessing step on constant symbol values.
>
>  - The only loss in generality would be that you can no longer have
>    a function expand to the name of non-constant symbol. For example,
>    take the following:
>
>         default $(foo)
>
>    If $(foo) expands to MY_SYMBOL, then that would work as
>
>         default MY_SYMBOL


Probably, this is a "do not do this".


If the symbol is 'int' type,
and  $MY_NUMBER is expanded to 1,
it is our intention.





>   (I.e., it would default to the value of the symbol MY_SYMBOL.)
>
>   With the quotes, it would instead work as
>
>      default "MY_SYMBOL"
>
>   IMO, the second version is less confusing, and deliberately using the
>   first version seems like a bad idea (there's likely to be cleaner ways
>   to do the indirection in plain Kconfig).
>
>   This is also why I think T_WORD_QUOTE is better in the code above. It
>   would make $(foo) work more like a shorthand for "$(foo)".
>
>
> Cons:
>
>  - Bit more typing to add the quotes
>
>  - Maybe it isn't widely known that "n", "m", "y" work just like n, m, y
>    for bool and tristate symbols (the latter automatically get converted
>    to the former), though you see them quite a lot in Kconfig files.

Not only bool/tristate, but also int/hex.
"1" is equivalent to 1.


Kconfig is screwed up in handling of literals.


>    ("n", "foo bar", etc., are all just constant symbols. Kconfig keeps
>    track of them separately from non-constant symbols. The constant
>    symbols "n", "m", and "y" are predefined.)
>
>    If we go with obligatory quotes, I volunteer to explain things in
>    kconfig-language.txt at least, to make it clear why you'd see quotes
>    in bool/tristate symbols using $(shell). I suspect it wouldn't be
>    that tricky to figure out anyway.


I think the right thing to do is
to fix the code instead of advertising it.






> What do you think?
>
> Cheers,
> Ulf
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

Re: [PATCH 11/23] kconfig: add 'shell-stdout' function

[Linus Torvalds]

On Sun, Feb 18, 2018 at 8:48 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
> On Fri, Feb 16, 2018 at 11:17:52AM -0800, Linus Torvalds wrote:
>>
>> Can we *please* make "shell-stdout" go away, and make this just be "shell"?
>>
>> The rule would be very simple:
>>
>>  - if the result of the shell command is a failure, the result is 'n'
>>
>>  - otherwise, the result is the first line of stdout
>>
>>  - if the result is empty, we replace it with 'y'.
>
> Could there be cases where you'd legitimately want to put empty output
> from a command in a string (that would be common enough to matter)?
> That'd get messier with the above rule, as it never generates an empty
> string as output.

Hmm. Maybe. Something like "LOCALVERSION_AUTO" might want that where
you add a version string if something is true, and maybe you'd use a
shell script for it and generate it at Kconfig time.

I'm not seeing anything like that right now, but I could imagine it in
theory, so your worry is valid.

> Would you still be as opposed (or more opposed...) to having two
> functions if they were called something like 'success' and 'stdout'
> instead?

Maybe the naming is indeed what annoyed me the most.

I do like your "success"/"stdout" more than "shell"/"shell-stdout",
because with that naming I don't get the feeling that one should
subsume the other.

          Linus

Re: [PATCH 07/23] kconfig: add function support and implement 'shell' function

[Ulf Magnusson]

On Tue, Feb 20, 2018 at 12:57:13AM +0900, Masahiro Yamada wrote:
> 2018-02-18 1:16 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> > On Sat, Feb 17, 2018 at 03:38:35AM +0900, Masahiro Yamada wrote:
> >> This commit adds a new concept 'function' to Kconfig.  A function call
> >> resembles a variable reference with arguments, and looks like this:
> >>
> >>   $(function arg1, arg2, arg3, ...)
> >>
> >> (Actually, this syntax was inspired by Makefile.)
> >>
> >> Real examples will look like this:
> >>
> >>   $(shell true)
> >>   $(cc-option -fstackprotector)
> >>
> >> This commit adds the basic infrastructure to add, delete, evaluate
> >> functions.
> >>
> >> Also, add the first built-in function $(shell ...).  This evaluates
> >> to 'y' if the given command exits with 0, 'n' otherwise.
> >>
> >> I am also planning to support user-defined functions (a.k.a 'macro')
> >> for cases where hard-coding is not preferred.
> >>
> >> If you want to try this feature, the hello-world code is someting below.
> >>
> >> Example code:
> >>
> >>   config CC_IS_GCC
> >>           bool
> >>           default $(shell $CC --version | grep -q gcc)
> >>
> >>   config CC_IS_CLANG
> >>           bool
> >>           default $(shell $CC --version | grep -q clang)
> >>
> >>   config CC_HAS_OZ
> >>           bool
> >>           default $(shell $CC -Werror -Oz -c -x c /dev/null -o /dev/null)
> >>
> >> Result:
> >>
> >>   $ make -s alldefconfig && tail -n 3 .config
> >>   CONFIG_CC_IS_GCC=y
> >>   # CONFIG_CC_IS_CLANG is not set
> >>   # CONFIG_CC_HAS_OZ is not set
> >>
> >>   $ make CC=clang -s alldefconfig && tail -n 3 .config
> >>   # CONFIG_CC_IS_GCC is not set
> >>   CONFIG_CC_IS_CLANG=y
> >>   CONFIG_CC_HAS_OZ=y
> >>
> >> A function call can appear anywhere a symbol reference can appear.
> >> So, the following code is possible.
> >>
> >> Example code:
> >>
> >>   config CC_NAME
> >>           string
> >>           default "gcc" if $(shell $CC --version | grep -q gcc)
> >>           default "clang" if $(shell $CC --version | grep -q clang)
> >>           default "unknown compiler"
> >>
> >> Result:
> >>
> >>   $ make -s alldefconfig && tail -n 1 .config
> >>   CONFIG_CC_NAME="gcc"
> >>
> >>   $ make CC=clang -s alldefconfig && tail -n 1 .config
> >>   CONFIG_CC_NAME="clang"
> >>
> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> >> ---
> >>
> >> Reminder for myself:
> >> Update Documentation/kbuild/kconfig-language.txt
> >>
> >>
> >>  scripts/kconfig/function.c  | 149 ++++++++++++++++++++++++++++++++++++++++++++
> >>  scripts/kconfig/lkc_proto.h |   5 ++
> >>  scripts/kconfig/util.c      |  46 +++++++++++---
> >>  scripts/kconfig/zconf.l     |  38 ++++++++++-
> >>  scripts/kconfig/zconf.y     |   9 +++
> >>  5 files changed, 238 insertions(+), 9 deletions(-)
> >>  create mode 100644 scripts/kconfig/function.c
> >>
> >> diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
> >> new file mode 100644
> >> index 0000000..60e59be
> >> --- /dev/null
> >> +++ b/scripts/kconfig/function.c
> >> @@ -0,0 +1,149 @@
> >> +// SPDX-License-Identifier: GPL-2.0
> >> +//
> >> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
> >> +
> >> +#include <stdio.h>
> >> +#include <stdlib.h>
> >> +#include <string.h>
> >> +
> >> +#include "list.h"
> >> +
> >> +#define FUNCTION_MAX_ARGS            10
> >> +
> >> +static LIST_HEAD(function_list);
> >> +
> >> +struct function {
> >> +     const char *name;
> >> +     char *(*func)(int argc, char *argv[]);
> >> +     struct list_head node;
> >> +};
> >> +
> >> +static struct function *func_lookup(const char *name)
> >> +{
> >> +     struct function *f;
> >> +
> >> +     list_for_each_entry(f, &function_list, node) {
> >> +             if (!strcmp(name, f->name))
> >> +                     return f;
> >> +     }
> >> +
> >> +     return NULL;
> >> +}
> >> +
> >> +static void func_add(const char *name, char *(*func)(int argc, char *argv[]))
> >> +{
> >> +     struct function *f;
> >> +
> >> +     f = func_lookup(name);
> >> +     if (f) {
> >> +             fprintf(stderr, "%s: function already exists. ignored.\n", name);
> >> +             return;
> >> +     }
> >> +
> >> +     f = xmalloc(sizeof(*f));
> >> +     f->name = name;
> >> +     f->func = func;
> >> +
> >> +     list_add_tail(&f->node, &function_list);
> >> +}
> >> +
> >> +static void func_del(struct function *f)
> >> +{
> >> +     list_del(&f->node);
> >> +     free(f);
> >> +}
> >> +
> >> +static char *func_call(int argc, char *argv[])
> >> +{
> >> +     struct function *f;
> >> +
> >> +     f = func_lookup(argv[0]);
> >> +     if (!f) {
> >> +             fprintf(stderr, "%s: function not found\n", argv[0]);
> >> +             return NULL;
> >> +     }
> >> +
> >> +     return f->func(argc, argv);
> >> +}
> >> +
> >> +static char *func_eval(const char *func)
> >> +{
> >> +     char *expanded, *saveptr, *str, *token, *res;
> >> +     const char *delim;
> >> +     int argc = 0;
> >> +     char *argv[FUNCTION_MAX_ARGS];
> >> +
> >> +     expanded = expand_string_value(func);
> >> +
> >> +     str = expanded;
> >> +     delim = " ";
> >> +
> >> +     while ((token = strtok_r(str, delim, &saveptr))) {
> >> +             argv[argc++] = token;
> >> +             str = NULL;
> >> +             delim = ",";
> >> +     }
> >> +
> >> +     res = func_call(argc, argv);
> >> +
> >> +     free(expanded);
> >> +
> >> +     return res ?: xstrdup("");
> >> +}
> >> +
> >> +char *func_eval_n(const char *func, size_t n)
> >> +{
> >> +     char *tmp, *res;
> >> +
> >> +     tmp = xmalloc(n + 1);
> >> +     memcpy(tmp, func, n);
> >> +     *(tmp + n) = '\0';
> >> +
> >> +     res = func_eval(tmp);
> >> +
> >> +     free(tmp);
> >> +
> >> +     return res;
> >> +}
> >> +
> >> +/* built-in functions */
> >> +static char *do_shell(int argc, char *argv[])
> >> +{
> >> +     static const char *pre = "(";
> >> +     static const char *post = ") >/dev/null 2>&1";
> >> +     char *cmd;
> >> +     int ret;
> >> +
> >> +     if (argc != 2)
> >> +             return NULL;
> >> +
> >> +     /*
> >> +      * Surround the command with ( ) in case it is piped commands.
> >> +      * Also, redirect stdout and stderr to /dev/null.
> >> +      */
> >> +     cmd = xmalloc(strlen(pre) + strlen(argv[1]) + strlen(post) + 1);
> >> +     strcpy(cmd, pre);
> >> +     strcat(cmd, argv[1]);
> >> +     strcat(cmd, post);
> >> +
> >> +     ret = system(cmd);
> >> +
> >> +     free(cmd);
> >> +
> >> +     return xstrdup(ret == 0 ? "y" : "n");
> >> +}
> >> +
> >> +void func_init(void)
> >> +{
> >> +     /* register built-in functions */
> >> +     func_add("shell", do_shell);
> >> +}
> >> +
> >> +void func_exit(void)
> >> +{
> >> +     struct function *f, *tmp;
> >> +
> >> +     /* unregister all functions */
> >> +     list_for_each_entry_safe(f, tmp, &function_list, node)
> >> +             func_del(f);
> >> +}
> >> diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
> >> index 9884adc..09a4f53 100644
> >> --- a/scripts/kconfig/lkc_proto.h
> >> +++ b/scripts/kconfig/lkc_proto.h
> >> @@ -48,5 +48,10 @@ const char * sym_get_string_value(struct symbol *sym);
> >>
> >>  const char * prop_get_type_name(enum prop_type type);
> >>
> >> +/* function.c */
> >> +char *func_eval_n(const char *func, size_t n);
> >> +void func_init(void);
> >> +void func_exit(void);
> >> +
> >>  /* expr.c */
> >>  void expr_print(struct expr *e, void (*fn)(void *, struct symbol *, const char *), void *data, int prevtoken);
> >> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
> >> index dddf85b..ed89fb9 100644
> >> --- a/scripts/kconfig/util.c
> >> +++ b/scripts/kconfig/util.c
> >> @@ -93,9 +93,10 @@ static char *env_expand_n(const char *name, size_t n)
> >>  }
> >>
> >>  /*
> >> - * Expand environments embedded in the string given in argument. Environments
> >> - * to be expanded shall be prefixed by a '$'. Unknown environment expands to
> >> - * the empty string.
> >> + * Expand environments and functions embedded in the string given in argument.
> >> + * Environments to be expanded shall be prefixed by a '$'. Functions to be
> >> + * evaluated shall be surrounded by $(). Unknown environment/function expands
> >> + * to the empty string.
> >>   */
> >>  char *expand_string_value(const char *in)
> >>  {
> >> @@ -113,11 +114,40 @@ char *expand_string_value(const char *in)
> >>       while ((p = strchr(in, '$'))) {
> >>               char *new;
> >>
> >> -             q = p + 1;
> >> -             while (isalnum(*q) || *q == '_')
> >> -                     q++;
> >> -
> >> -             new = env_expand_n(p + 1, q - p - 1);
> >> +             /*
> >> +              * If the next character is '(', it is a function.
> >> +              * Otherwise, environment.
> >> +              */
> >> +             if (*(p + 1) == '(') {
> >> +                     int nest = 0;
> >> +
> >> +                     q = p + 2;
> >> +                     while (1) {
> >> +                             if (*q == '\0') {
> >> +                                     fprintf(stderr,
> >> +                                             "unterminated function: %s\n",
> >> +                                             p);
> >> +                                     new = xstrdup("");
> >> +                                     break;
> >> +                             } else if (*q == '(') {
> >> +                                     nest++;
> >> +                             } else if (*q == ')') {
> >> +                                     if (nest-- == 0) {
> >> +                                             new = func_eval_n(p + 2,
> >> +                                                               q - p - 2);
> >> +                                             q++;
> >> +                                             break;
> >> +                                     }
> >> +                             }
> >> +                             q++;
> >> +                     }
> >> +             } else {
> >> +                     q = p + 1;
> >> +                     while (isalnum(*q) || *q == '_')
> >> +                             q++;
> >> +
> >> +                     new = env_expand_n(p + 1, q - p - 1);
> >> +             }
> >>
> >>               reslen = strlen(res) + (p - in) + strlen(new) + 1;
> >>               res = xrealloc(res, reslen);
> >> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
> >> index 0d89ea6..f433ab0 100644
> >> --- a/scripts/kconfig/zconf.l
> >> +++ b/scripts/kconfig/zconf.l
> >> @@ -1,7 +1,7 @@
> >>  %option nostdinit noyywrap never-interactive full ecs
> >>  %option 8bit nodefault perf-report perf-report
> >>  %option noinput
> >> -%x COMMAND HELP STRING PARAM
> >> +%x COMMAND HELP STRING PARAM FUNCTION
> >>  %{
> >>  /*
> >>   * Copyright (C) 2002 Roman Zippel <zippel@linux-m68k.org>
> >> @@ -25,6 +25,7 @@ static struct {
> >>
> >>  static char *text;
> >>  static int text_size, text_asize;
> >> +static int function_nest;
> >>
> >>  struct buffer {
> >>       struct buffer *parent;
> >> @@ -138,6 +139,12 @@ n        [$A-Za-z0-9_-]
> >>               new_string();
> >>               BEGIN(STRING);
> >>       }
> >> +     "$("    {
> >> +             new_string();
> >> +             append_string(yytext, yyleng);
> >> +             function_nest = 0;
> >> +             BEGIN(FUNCTION);
> >> +     }
> >>       \n      BEGIN(INITIAL); current_file->lineno++; return T_EOL;
> >>       ({n}|[/.])+     {
> >>               const struct kconf_id *id = kconf_id_lookup(yytext, yyleng);
> >> @@ -196,6 +203,35 @@ n        [$A-Za-z0-9_-]
> >>       }
> >>  }
> >>
> >> +<FUNCTION>{
> >> +     [^()\n]* {
> >> +             append_string(yytext, yyleng);
> >> +     }
> >> +     "(" {
> >> +             append_string(yytext, yyleng);
> >> +             function_nest++;
> >> +     }
> >> +     ")" {
> >> +             append_string(yytext, yyleng);
> >> +             if (function_nest-- == 0) {
> >> +                     BEGIN(PARAM);
> >> +                     yylval.string = text;
> >> +                     return T_WORD;
> >
> > T_WORD_QUOTE (which would turn into a constant symbol in most contexts)
> > would be better here, IMO. That would turn $(foo) into just an alias for
> > "$(foo)".
> >
> > See below.
> >
> >> +             }
> >> +     }
> >> +     \n {
> >> +             fprintf(stderr,
> >> +                     "%s:%d:warning: multi-line function not supported\n",
> >> +                     zconf_curname(), zconf_lineno());
> >> +             current_file->lineno++;
> >> +             BEGIN(INITIAL);
> >> +             return T_EOL;
> >> +     }
> >> +     <<EOF>> {
> >> +             BEGIN(INITIAL);
> >> +     }
> >> +}
> >> +
> >>  <HELP>{
> >>       [ \t]+  {
> >>               ts = 0;
> >> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
> >> index 784083d..d9977de 100644
> >> --- a/scripts/kconfig/zconf.y
> >> +++ b/scripts/kconfig/zconf.y
> >> @@ -534,11 +534,19 @@ void conf_parse(const char *name)
> >>
> >>       zconf_initscan(name);
> >>
> >> +     func_init();
> >>       _menu_init();
> >>
> >>       if (getenv("ZCONF_DEBUG"))
> >>               yydebug = 1;
> >>       yyparse();
> >> +
> >> +     /*
> >> +      * Currently, functions are evaluated only when Kconfig files are
> >> +      * parsed. We can free functions here.
> >> +      */
> >> +     func_exit();
> >> +
> >>       if (yynerrs)
> >>               exit(1);
> >>       if (!modules_sym)
> >> @@ -778,4 +786,5 @@ void zconfdump(FILE *out)
> >>  #include "confdata.c"
> >>  #include "expr.c"
> >>  #include "symbol.c"
> >> +#include "function.c"
> >>  #include "menu.c"
> >> --
> >> 2.7.4
> >>
> >
> > Here's a simplification idea I'm throwing out there:
> >
> > What about only allowing $ENV and $() within quotes, and just having
> > them always do simple text substitution (so that they'd indirectly
> > always generate T_WORD_QUOTE tokens)?
> 
> 
> I ended up with a new state <FUNCTION>,
> but the real problem is the lexer is too ugly.
> 
> So, maybe, the solution is not to make it into a string,
> but to re-write the lexer.
> 
> 
> <STRING> and <FUNCTION> are almost the same in the sense
> that whitespaces do not split tokens.
> 
> The difference is the characters to start/end with.
> 
>   ' ... '
>   " ... "
>   ( ... )
> 
> If we encounter with the 3 special characters,  ' , '', (,
> then we enter into <STRING> state,
> then it ends with ', ", ), respectively.
> (but we need to take care of nesting, escaping)
> 
> Double-surrounding like  "$(cc-option -Oz)"
> looks ugly to me.

It makes it clear that all we're doing is string interpolation, which is
the important thing.

The simplest and to me sanest way to implement $(cc-option -Oz) is as
"$(cc-option -Oz)" (a SYMBOL_CONST symbol, which lives in a separate
namespace -- see below), and at that point $(cc-option -Oz) would just
be syntactic sugar for "$(cc-option -Oz)".

IMO, that syntactic sugar just hides how simple the behavior really is
and makes Kconfig more confusing. Plus it complicates the
implementation.

> 
> 
> 
> 
> 
> > Pros:
> >
> >  - Zero new syntax outside of strings (until the macro stuff).
> >
> >  - Makes the behavior and limitations of the syntax obvious: You can't
> >    have $(foo) expand to a full expression, only to (possibly part of) a
> >    value. This is a good limitation, IMO, and it's already there.
> >
> >  - Super simple and straightforward Kconfig implementation. All the new
> >    magic would happen in expand_string_value().
> >
> > Neutral:
> >
> >  - Just as general where it matters. Take something like
> >
> >      default "$(foo)"
> >
> >    If $(foo) happens to expand to "y", then that will do its usual thing
> >    for a bool/tristate symbol. Same for string symbols, etc. It'd just
> >    be a thin preprocessing step on constant symbol values.
> >
> >  - The only loss in generality would be that you can no longer have
> >    a function expand to the name of non-constant symbol. For example,
> >    take the following:
> >
> >         default $(foo)
> >
> >    If $(foo) expands to MY_SYMBOL, then that would work as
> >
> >         default MY_SYMBOL
> 
> 
> Probably, this is a "do not do this".
> 
> 
> If the symbol is 'int' type,
> and  $MY_NUMBER is expanded to 1,
> it is our intention.

Just to summarize the tradeoffs in each approach:

Approach 1:
Let $(fn ...) expand to FOO and reference the symbol FOO if it exists

Approach 2:
Let $(fn ...) expand to "FOO", which could never reference existing
symbols


Pros of approach 1:

  - If someone legitimately wants to do indirection by generating
    Kconfig symbol names, they can.

Cons of approach 1:

  - If $(fn ...) happens to expand to the name of an existing symbol, it
    will reference it, even if it wasn't intended. This might get really
    tricky to debug, as it probably won't be obvious that this is how it
    works.


Pros of approach 2:

  - You can never accidentally reference an existing symbol when it
    wasn't intended. You always generate a "value". Constant symbols
    live in a separate namespace.

  - The behavior is simpler to understand, IMO. $(fn ...) and $FOO
    consistently deal with values and just do string interpolation. You
    don't get "either a symbol or a value" depending on what's already
    defined.

Cons of approach 2:

  - You can't do indirection by generating Kconfig symbol names. I
    suspect there'd always be cleaner ways to do that in practice, but
    I'm showing my bias here.


Approach 2 seems much simpler and less magical to me.

> 
> 
> 
> 
> 
> >   (I.e., it would default to the value of the symbol MY_SYMBOL.)
> >
> >   With the quotes, it would instead work as
> >
> >      default "MY_SYMBOL"
> >
> >   IMO, the second version is less confusing, and deliberately using the
> >   first version seems like a bad idea (there's likely to be cleaner ways
> >   to do the indirection in plain Kconfig).
> >
> >   This is also why I think T_WORD_QUOTE is better in the code above. It
> >   would make $(foo) work more like a shorthand for "$(foo)".
> >
> >
> > Cons:
> >
> >  - Bit more typing to add the quotes
> >
> >  - Maybe it isn't widely known that "n", "m", "y" work just like n, m, y
> >    for bool and tristate symbols (the latter automatically get converted
> >    to the former), though you see them quite a lot in Kconfig files.
> 
> Not only bool/tristate, but also int/hex.
> "1" is equivalent to 1.
> 
> 
> Kconfig is screwed up in handling of literals.

I don't actually think the design is horrible here (though it has
flaws). The biggest problem is that it's poorly documented and
understood.

Expressions are composed of symbols and operators. You have ordinary
symbols and constant symbols (quoted stuff, "values" if you will). All
symbols have a tristate value and a string value. Which value gets used,
and how, depends on the operator.

Note that constant symbols are actually stored separately from
nonconstant symbols internally (see SYMBOL_CONST). 1 and "1" are not the
same symbol (this is easy to miss unless you've spent too much
deciphering Kconfig's code). The reason a plain 1 works is that
undefined symbols share many behaviors with constant symbols.

The biggest mess-up in Kconfig I think is not making the constant vs.
undefined distinction clear. It would have been neater if references to
undefined symbols simply threw an error, though that gets a bit icky
with shared Kconfig files, like in the kernel (would need to make sure
that referenced symbols are always defined or predeclare them as
undefined).

Some people might object to "1" as well, but it would've been a lesser
evil I think, once the symbol/value distinction would be clear.

The other option would be to tack some kind of complicated type system
on top of Kconfig's evaluation semantics. That's overkilling it, IMO.

> 
> 
> >    ("n", "foo bar", etc., are all just constant symbols. Kconfig keeps
> >    track of them separately from non-constant symbols. The constant
> >    symbols "n", "m", and "y" are predefined.)
> >
> >    If we go with obligatory quotes, I volunteer to explain things in
> >    kconfig-language.txt at least, to make it clear why you'd see quotes
> >    in bool/tristate symbols using $(shell). I suspect it wouldn't be
> >    that tricky to figure out anyway.
> 
> 
> I think the right thing to do is
> to fix the code instead of advertising it.

I don't think the behavior is all bad, just obscure. It could easily be
explained.

> 
> 
> 
> 
> 
> 
> > What do you think?
> >
> > Cheers,
> > Ulf
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> -- 
> Best Regards
> Masahiro Yamada

Cheers,
Ulf

Re: [PATCH 11/23] kconfig: add 'shell-stdout' function

[Linus Torvalds]

On Mon, Feb 19, 2018 at 9:44 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> I do like your "success"/"stdout" more than "shell"/"shell-stdout",
> because with that naming I don't get the feeling that one should
> subsume the other.

Hmm. Thinking about it some more, I really would prefer just "$(shell
...)" everywhere.

But it would be nice if perhaps the error handling would match the
context somehow.

I'm wondering if this might tie into the whole quoting discussion in
the other thread.

Because the rule could be:

(a) unquoted $(shell ) is a bool, and failing is ok (and turns into
y/n depending on whether successful or failing)

So

  config CC_IS_GCC
          bool
          default $(shell $CC --version | grep -q gcc)

works automatically.

(b) but with quoting, $(shell ) is a string, and failing is an error

So

  config GCC_VERSION
          int
          default "$(shell-stdout $srctree/scripts/gcc-version.sh $CC
| sed 's/^0*//')" if CC_IS_GCC
          default 0

would need those quotes, and if the shell-script returns a failure,
we'd _abort_.

Which is actually what we want there.

Hmm? Is that too nasty?

                Linus

Re: [PATCH 11/23] kconfig: add 'shell-stdout' function

[Ulf Magnusson]

On Mon, Feb 19, 2018 at 10:01:49AM -0800, Linus Torvalds wrote:
> On Mon, Feb 19, 2018 at 9:44 AM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > I do like your "success"/"stdout" more than "shell"/"shell-stdout",
> > because with that naming I don't get the feeling that one should
> > subsume the other.
> 
> Hmm. Thinking about it some more, I really would prefer just "$(shell
> ...)" everywhere.
> 
> But it would be nice if perhaps the error handling would match the
> context somehow.
> 
> I'm wondering if this might tie into the whole quoting discussion in
> the other thread.
> 
> Because the rule could be:
> 
> (a) unquoted $(shell ) is a bool, and failing is ok (and turns into
> y/n depending on whether successful or failing)
> 
> So
> 
>   config CC_IS_GCC
>           bool
>           default $(shell $CC --version | grep -q gcc)
> 
> works automatically.
> 
> (b) but with quoting, $(shell ) is a string, and failing is an error
> 
> So
> 
>   config GCC_VERSION
>           int
>           default "$(shell-stdout $srctree/scripts/gcc-version.sh $CC
> | sed 's/^0*//')" if CC_IS_GCC
>           default 0
> 
> would need those quotes, and if the shell-script returns a failure,
> we'd _abort_.
> 
> Which is actually what we want there.
> 
> Hmm? Is that too nasty?
> 
>                 Linus

One minor drawback would be slight kludginess if you want "n"/"y" put
into a string depending on the success of a command:

	default	"foo-$(shell cmd && echo y || echo n)"

As opposed to:

	default "foo-$(success cmd)"

I don't know if that's significant enough to matter in practice.

Keeping it objective, I can't see any major downsides, though I'd really
prefer to just have $() do string interpolation within "". That keeps
the implementation trivial and makes the behavior and limitations
obvious once you know that n/m/y is just shorthand for "n"/"m"/"y".

Cheers,
Ulf

Re: [PATCH 07/23] kconfig: add function support and implement 'shell' function

[Ulf Magnusson]

On Mon, Feb 19, 2018 at 6:50 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
> On Tue, Feb 20, 2018 at 12:57:13AM +0900, Masahiro Yamada wrote:
>> 2018-02-18 1:16 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>> > On Sat, Feb 17, 2018 at 03:38:35AM +0900, Masahiro Yamada wrote:
>> >> This commit adds a new concept 'function' to Kconfig.  A function call
>> >> resembles a variable reference with arguments, and looks like this:
>> >>
>> >>   $(function arg1, arg2, arg3, ...)
>> >>
>> >> (Actually, this syntax was inspired by Makefile.)
>> >>
>> >> Real examples will look like this:
>> >>
>> >>   $(shell true)
>> >>   $(cc-option -fstackprotector)
>> >>
>> >> This commit adds the basic infrastructure to add, delete, evaluate
>> >> functions.
>> >>
>> >> Also, add the first built-in function $(shell ...).  This evaluates
>> >> to 'y' if the given command exits with 0, 'n' otherwise.
>> >>
>> >> I am also planning to support user-defined functions (a.k.a 'macro')
>> >> for cases where hard-coding is not preferred.
>> >>
>> >> If you want to try this feature, the hello-world code is someting below.
>> >>
>> >> Example code:
>> >>
>> >>   config CC_IS_GCC
>> >>           bool
>> >>           default $(shell $CC --version | grep -q gcc)
>> >>
>> >>   config CC_IS_CLANG
>> >>           bool
>> >>           default $(shell $CC --version | grep -q clang)
>> >>
>> >>   config CC_HAS_OZ
>> >>           bool
>> >>           default $(shell $CC -Werror -Oz -c -x c /dev/null -o /dev/null)
>> >>
>> >> Result:
>> >>
>> >>   $ make -s alldefconfig && tail -n 3 .config
>> >>   CONFIG_CC_IS_GCC=y
>> >>   # CONFIG_CC_IS_CLANG is not set
>> >>   # CONFIG_CC_HAS_OZ is not set
>> >>
>> >>   $ make CC=clang -s alldefconfig && tail -n 3 .config
>> >>   # CONFIG_CC_IS_GCC is not set
>> >>   CONFIG_CC_IS_CLANG=y
>> >>   CONFIG_CC_HAS_OZ=y
>> >>
>> >> A function call can appear anywhere a symbol reference can appear.
>> >> So, the following code is possible.
>> >>
>> >> Example code:
>> >>
>> >>   config CC_NAME
>> >>           string
>> >>           default "gcc" if $(shell $CC --version | grep -q gcc)
>> >>           default "clang" if $(shell $CC --version | grep -q clang)
>> >>           default "unknown compiler"
>> >>
>> >> Result:
>> >>
>> >>   $ make -s alldefconfig && tail -n 1 .config
>> >>   CONFIG_CC_NAME="gcc"
>> >>
>> >>   $ make CC=clang -s alldefconfig && tail -n 1 .config
>> >>   CONFIG_CC_NAME="clang"
>> >>
>> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> >> ---
>> >>
>> >> Reminder for myself:
>> >> Update Documentation/kbuild/kconfig-language.txt
>> >>
>> >>
>> >>  scripts/kconfig/function.c  | 149 ++++++++++++++++++++++++++++++++++++++++++++
>> >>  scripts/kconfig/lkc_proto.h |   5 ++
>> >>  scripts/kconfig/util.c      |  46 +++++++++++---
>> >>  scripts/kconfig/zconf.l     |  38 ++++++++++-
>> >>  scripts/kconfig/zconf.y     |   9 +++
>> >>  5 files changed, 238 insertions(+), 9 deletions(-)
>> >>  create mode 100644 scripts/kconfig/function.c
>> >>
>> >> diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
>> >> new file mode 100644
>> >> index 0000000..60e59be
>> >> --- /dev/null
>> >> +++ b/scripts/kconfig/function.c
>> >> @@ -0,0 +1,149 @@
>> >> +// SPDX-License-Identifier: GPL-2.0
>> >> +//
>> >> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
>> >> +
>> >> +#include <stdio.h>
>> >> +#include <stdlib.h>
>> >> +#include <string.h>
>> >> +
>> >> +#include "list.h"
>> >> +
>> >> +#define FUNCTION_MAX_ARGS            10
>> >> +
>> >> +static LIST_HEAD(function_list);
>> >> +
>> >> +struct function {
>> >> +     const char *name;
>> >> +     char *(*func)(int argc, char *argv[]);
>> >> +     struct list_head node;
>> >> +};
>> >> +
>> >> +static struct function *func_lookup(const char *name)
>> >> +{
>> >> +     struct function *f;
>> >> +
>> >> +     list_for_each_entry(f, &function_list, node) {
>> >> +             if (!strcmp(name, f->name))
>> >> +                     return f;
>> >> +     }
>> >> +
>> >> +     return NULL;
>> >> +}
>> >> +
>> >> +static void func_add(const char *name, char *(*func)(int argc, char *argv[]))
>> >> +{
>> >> +     struct function *f;
>> >> +
>> >> +     f = func_lookup(name);
>> >> +     if (f) {
>> >> +             fprintf(stderr, "%s: function already exists. ignored.\n", name);
>> >> +             return;
>> >> +     }
>> >> +
>> >> +     f = xmalloc(sizeof(*f));
>> >> +     f->name = name;
>> >> +     f->func = func;
>> >> +
>> >> +     list_add_tail(&f->node, &function_list);
>> >> +}
>> >> +
>> >> +static void func_del(struct function *f)
>> >> +{
>> >> +     list_del(&f->node);
>> >> +     free(f);
>> >> +}
>> >> +
>> >> +static char *func_call(int argc, char *argv[])
>> >> +{
>> >> +     struct function *f;
>> >> +
>> >> +     f = func_lookup(argv[0]);
>> >> +     if (!f) {
>> >> +             fprintf(stderr, "%s: function not found\n", argv[0]);
>> >> +             return NULL;
>> >> +     }
>> >> +
>> >> +     return f->func(argc, argv);
>> >> +}
>> >> +
>> >> +static char *func_eval(const char *func)
>> >> +{
>> >> +     char *expanded, *saveptr, *str, *token, *res;
>> >> +     const char *delim;
>> >> +     int argc = 0;
>> >> +     char *argv[FUNCTION_MAX_ARGS];
>> >> +
>> >> +     expanded = expand_string_value(func);
>> >> +
>> >> +     str = expanded;
>> >> +     delim = " ";
>> >> +
>> >> +     while ((token = strtok_r(str, delim, &saveptr))) {
>> >> +             argv[argc++] = token;
>> >> +             str = NULL;
>> >> +             delim = ",";
>> >> +     }
>> >> +
>> >> +     res = func_call(argc, argv);
>> >> +
>> >> +     free(expanded);
>> >> +
>> >> +     return res ?: xstrdup("");
>> >> +}
>> >> +
>> >> +char *func_eval_n(const char *func, size_t n)
>> >> +{
>> >> +     char *tmp, *res;
>> >> +
>> >> +     tmp = xmalloc(n + 1);
>> >> +     memcpy(tmp, func, n);
>> >> +     *(tmp + n) = '\0';
>> >> +
>> >> +     res = func_eval(tmp);
>> >> +
>> >> +     free(tmp);
>> >> +
>> >> +     return res;
>> >> +}
>> >> +
>> >> +/* built-in functions */
>> >> +static char *do_shell(int argc, char *argv[])
>> >> +{
>> >> +     static const char *pre = "(";
>> >> +     static const char *post = ") >/dev/null 2>&1";
>> >> +     char *cmd;
>> >> +     int ret;
>> >> +
>> >> +     if (argc != 2)
>> >> +             return NULL;
>> >> +
>> >> +     /*
>> >> +      * Surround the command with ( ) in case it is piped commands.
>> >> +      * Also, redirect stdout and stderr to /dev/null.
>> >> +      */
>> >> +     cmd = xmalloc(strlen(pre) + strlen(argv[1]) + strlen(post) + 1);
>> >> +     strcpy(cmd, pre);
>> >> +     strcat(cmd, argv[1]);
>> >> +     strcat(cmd, post);
>> >> +
>> >> +     ret = system(cmd);
>> >> +
>> >> +     free(cmd);
>> >> +
>> >> +     return xstrdup(ret == 0 ? "y" : "n");
>> >> +}
>> >> +
>> >> +void func_init(void)
>> >> +{
>> >> +     /* register built-in functions */
>> >> +     func_add("shell", do_shell);
>> >> +}
>> >> +
>> >> +void func_exit(void)
>> >> +{
>> >> +     struct function *f, *tmp;
>> >> +
>> >> +     /* unregister all functions */
>> >> +     list_for_each_entry_safe(f, tmp, &function_list, node)
>> >> +             func_del(f);
>> >> +}
>> >> diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
>> >> index 9884adc..09a4f53 100644
>> >> --- a/scripts/kconfig/lkc_proto.h
>> >> +++ b/scripts/kconfig/lkc_proto.h
>> >> @@ -48,5 +48,10 @@ const char * sym_get_string_value(struct symbol *sym);
>> >>
>> >>  const char * prop_get_type_name(enum prop_type type);
>> >>
>> >> +/* function.c */
>> >> +char *func_eval_n(const char *func, size_t n);
>> >> +void func_init(void);
>> >> +void func_exit(void);
>> >> +
>> >>  /* expr.c */
>> >>  void expr_print(struct expr *e, void (*fn)(void *, struct symbol *, const char *), void *data, int prevtoken);
>> >> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
>> >> index dddf85b..ed89fb9 100644
>> >> --- a/scripts/kconfig/util.c
>> >> +++ b/scripts/kconfig/util.c
>> >> @@ -93,9 +93,10 @@ static char *env_expand_n(const char *name, size_t n)
>> >>  }
>> >>
>> >>  /*
>> >> - * Expand environments embedded in the string given in argument. Environments
>> >> - * to be expanded shall be prefixed by a '$'. Unknown environment expands to
>> >> - * the empty string.
>> >> + * Expand environments and functions embedded in the string given in argument.
>> >> + * Environments to be expanded shall be prefixed by a '$'. Functions to be
>> >> + * evaluated shall be surrounded by $(). Unknown environment/function expands
>> >> + * to the empty string.
>> >>   */
>> >>  char *expand_string_value(const char *in)
>> >>  {
>> >> @@ -113,11 +114,40 @@ char *expand_string_value(const char *in)
>> >>       while ((p = strchr(in, '$'))) {
>> >>               char *new;
>> >>
>> >> -             q = p + 1;
>> >> -             while (isalnum(*q) || *q == '_')
>> >> -                     q++;
>> >> -
>> >> -             new = env_expand_n(p + 1, q - p - 1);
>> >> +             /*
>> >> +              * If the next character is '(', it is a function.
>> >> +              * Otherwise, environment.
>> >> +              */
>> >> +             if (*(p + 1) == '(') {
>> >> +                     int nest = 0;
>> >> +
>> >> +                     q = p + 2;
>> >> +                     while (1) {
>> >> +                             if (*q == '\0') {
>> >> +                                     fprintf(stderr,
>> >> +                                             "unterminated function: %s\n",
>> >> +                                             p);
>> >> +                                     new = xstrdup("");
>> >> +                                     break;
>> >> +                             } else if (*q == '(') {
>> >> +                                     nest++;
>> >> +                             } else if (*q == ')') {
>> >> +                                     if (nest-- == 0) {
>> >> +                                             new = func_eval_n(p + 2,
>> >> +                                                               q - p - 2);
>> >> +                                             q++;
>> >> +                                             break;
>> >> +                                     }
>> >> +                             }
>> >> +                             q++;
>> >> +                     }
>> >> +             } else {
>> >> +                     q = p + 1;
>> >> +                     while (isalnum(*q) || *q == '_')
>> >> +                             q++;
>> >> +
>> >> +                     new = env_expand_n(p + 1, q - p - 1);
>> >> +             }
>> >>
>> >>               reslen = strlen(res) + (p - in) + strlen(new) + 1;
>> >>               res = xrealloc(res, reslen);
>> >> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
>> >> index 0d89ea6..f433ab0 100644
>> >> --- a/scripts/kconfig/zconf.l
>> >> +++ b/scripts/kconfig/zconf.l
>> >> @@ -1,7 +1,7 @@
>> >>  %option nostdinit noyywrap never-interactive full ecs
>> >>  %option 8bit nodefault perf-report perf-report
>> >>  %option noinput
>> >> -%x COMMAND HELP STRING PARAM
>> >> +%x COMMAND HELP STRING PARAM FUNCTION
>> >>  %{
>> >>  /*
>> >>   * Copyright (C) 2002 Roman Zippel <zippel@linux-m68k.org>
>> >> @@ -25,6 +25,7 @@ static struct {
>> >>
>> >>  static char *text;
>> >>  static int text_size, text_asize;
>> >> +static int function_nest;
>> >>
>> >>  struct buffer {
>> >>       struct buffer *parent;
>> >> @@ -138,6 +139,12 @@ n        [$A-Za-z0-9_-]
>> >>               new_string();
>> >>               BEGIN(STRING);
>> >>       }
>> >> +     "$("    {
>> >> +             new_string();
>> >> +             append_string(yytext, yyleng);
>> >> +             function_nest = 0;
>> >> +             BEGIN(FUNCTION);
>> >> +     }
>> >>       \n      BEGIN(INITIAL); current_file->lineno++; return T_EOL;
>> >>       ({n}|[/.])+     {
>> >>               const struct kconf_id *id = kconf_id_lookup(yytext, yyleng);
>> >> @@ -196,6 +203,35 @@ n        [$A-Za-z0-9_-]
>> >>       }
>> >>  }
>> >>
>> >> +<FUNCTION>{
>> >> +     [^()\n]* {
>> >> +             append_string(yytext, yyleng);
>> >> +     }
>> >> +     "(" {
>> >> +             append_string(yytext, yyleng);
>> >> +             function_nest++;
>> >> +     }
>> >> +     ")" {
>> >> +             append_string(yytext, yyleng);
>> >> +             if (function_nest-- == 0) {
>> >> +                     BEGIN(PARAM);
>> >> +                     yylval.string = text;
>> >> +                     return T_WORD;
>> >
>> > T_WORD_QUOTE (which would turn into a constant symbol in most contexts)
>> > would be better here, IMO. That would turn $(foo) into just an alias for
>> > "$(foo)".
>> >
>> > See below.
>> >
>> >> +             }
>> >> +     }
>> >> +     \n {
>> >> +             fprintf(stderr,
>> >> +                     "%s:%d:warning: multi-line function not supported\n",
>> >> +                     zconf_curname(), zconf_lineno());
>> >> +             current_file->lineno++;
>> >> +             BEGIN(INITIAL);
>> >> +             return T_EOL;
>> >> +     }
>> >> +     <<EOF>> {
>> >> +             BEGIN(INITIAL);
>> >> +     }
>> >> +}
>> >> +
>> >>  <HELP>{
>> >>       [ \t]+  {
>> >>               ts = 0;
>> >> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
>> >> index 784083d..d9977de 100644
>> >> --- a/scripts/kconfig/zconf.y
>> >> +++ b/scripts/kconfig/zconf.y
>> >> @@ -534,11 +534,19 @@ void conf_parse(const char *name)
>> >>
>> >>       zconf_initscan(name);
>> >>
>> >> +     func_init();
>> >>       _menu_init();
>> >>
>> >>       if (getenv("ZCONF_DEBUG"))
>> >>               yydebug = 1;
>> >>       yyparse();
>> >> +
>> >> +     /*
>> >> +      * Currently, functions are evaluated only when Kconfig files are
>> >> +      * parsed. We can free functions here.
>> >> +      */
>> >> +     func_exit();
>> >> +
>> >>       if (yynerrs)
>> >>               exit(1);
>> >>       if (!modules_sym)
>> >> @@ -778,4 +786,5 @@ void zconfdump(FILE *out)
>> >>  #include "confdata.c"
>> >>  #include "expr.c"
>> >>  #include "symbol.c"
>> >> +#include "function.c"
>> >>  #include "menu.c"
>> >> --
>> >> 2.7.4
>> >>
>> >
>> > Here's a simplification idea I'm throwing out there:
>> >
>> > What about only allowing $ENV and $() within quotes, and just having
>> > them always do simple text substitution (so that they'd indirectly
>> > always generate T_WORD_QUOTE tokens)?
>>
>>
>> I ended up with a new state <FUNCTION>,
>> but the real problem is the lexer is too ugly.
>>
>> So, maybe, the solution is not to make it into a string,
>> but to re-write the lexer.
>>
>>
>> <STRING> and <FUNCTION> are almost the same in the sense
>> that whitespaces do not split tokens.
>>
>> The difference is the characters to start/end with.
>>
>>   ' ... '
>>   " ... "
>>   ( ... )
>>
>> If we encounter with the 3 special characters,  ' , '', (,
>> then we enter into <STRING> state,
>> then it ends with ', ", ), respectively.
>> (but we need to take care of nesting, escaping)
>>
>> Double-surrounding like  "$(cc-option -Oz)"
>> looks ugly to me.
>
> It makes it clear that all we're doing is string interpolation, which is
> the important thing.
>
> The simplest and to me sanest way to implement $(cc-option -Oz) is as
> "$(cc-option -Oz)" (a SYMBOL_CONST symbol, which lives in a separate
> namespace -- see below), and at that point $(cc-option -Oz) would just
> be syntactic sugar for "$(cc-option -Oz)".
>
> IMO, that syntactic sugar just hides how simple the behavior really is
> and makes Kconfig more confusing. Plus it complicates the
> implementation.
>
>>
>>
>>
>>
>>
>> > Pros:
>> >
>> >  - Zero new syntax outside of strings (until the macro stuff).
>> >
>> >  - Makes the behavior and limitations of the syntax obvious: You can't
>> >    have $(foo) expand to a full expression, only to (possibly part of) a
>> >    value. This is a good limitation, IMO, and it's already there.
>> >
>> >  - Super simple and straightforward Kconfig implementation. All the new
>> >    magic would happen in expand_string_value().
>> >
>> > Neutral:
>> >
>> >  - Just as general where it matters. Take something like
>> >
>> >      default "$(foo)"
>> >
>> >    If $(foo) happens to expand to "y", then that will do its usual thing
>> >    for a bool/tristate symbol. Same for string symbols, etc. It'd just
>> >    be a thin preprocessing step on constant symbol values.
>> >
>> >  - The only loss in generality would be that you can no longer have
>> >    a function expand to the name of non-constant symbol. For example,
>> >    take the following:
>> >
>> >         default $(foo)
>> >
>> >    If $(foo) expands to MY_SYMBOL, then that would work as
>> >
>> >         default MY_SYMBOL
>>
>>
>> Probably, this is a "do not do this".
>>
>>
>> If the symbol is 'int' type,
>> and  $MY_NUMBER is expanded to 1,
>> it is our intention.
>
> Just to summarize the tradeoffs in each approach:
>
> Approach 1:
> Let $(fn ...) expand to FOO and reference the symbol FOO if it exists
>
> Approach 2:
> Let $(fn ...) expand to "FOO", which could never reference existing
> symbols
>
>
> Pros of approach 1:
>
>   - If someone legitimately wants to do indirection by generating
>     Kconfig symbol names, they can.
>
> Cons of approach 1:
>
>   - If $(fn ...) happens to expand to the name of an existing symbol, it
>     will reference it, even if it wasn't intended. This might get really
>     tricky to debug, as it probably won't be obvious that this is how it
>     works.
>
>
> Pros of approach 2:
>
>   - You can never accidentally reference an existing symbol when it
>     wasn't intended. You always generate a "value". Constant symbols
>     live in a separate namespace.
>
>   - The behavior is simpler to understand, IMO. $(fn ...) and $FOO
>     consistently deal with values and just do string interpolation. You
>     don't get "either a symbol or a value" depending on what's already
>     defined.
>
> Cons of approach 2:
>
>   - You can't do indirection by generating Kconfig symbol names. I
>     suspect there'd always be cleaner ways to do that in practice, but
>     I'm showing my bias here.
>
>
> Approach 2 seems much simpler and less magical to me.
>
>>
>>
>>
>>
>>
>> >   (I.e., it would default to the value of the symbol MY_SYMBOL.)
>> >
>> >   With the quotes, it would instead work as
>> >
>> >      default "MY_SYMBOL"
>> >
>> >   IMO, the second version is less confusing, and deliberately using the
>> >   first version seems like a bad idea (there's likely to be cleaner ways
>> >   to do the indirection in plain Kconfig).
>> >
>> >   This is also why I think T_WORD_QUOTE is better in the code above. It
>> >   would make $(foo) work more like a shorthand for "$(foo)".
>> >
>> >
>> > Cons:
>> >
>> >  - Bit more typing to add the quotes
>> >
>> >  - Maybe it isn't widely known that "n", "m", "y" work just like n, m, y
>> >    for bool and tristate symbols (the latter automatically get converted
>> >    to the former), though you see them quite a lot in Kconfig files.
>>
>> Not only bool/tristate, but also int/hex.
>> "1" is equivalent to 1.
>>
>>
>> Kconfig is screwed up in handling of literals.
>
> I don't actually think the design is horrible here (though it has
> flaws). The biggest problem is that it's poorly documented and
> understood.
>
> Expressions are composed of symbols and operators. You have ordinary
> symbols and constant symbols (quoted stuff, "values" if you will). All
> symbols have a tristate value and a string value. Which value gets used,
> and how, depends on the operator.
>
> Note that constant symbols are actually stored separately from
> nonconstant symbols internally (see SYMBOL_CONST). 1 and "1" are not the
> same symbol (this is easy to miss unless you've spent too much
> deciphering Kconfig's code). The reason a plain 1 works is that
> undefined symbols share many behaviors with constant symbols.
>
> The biggest mess-up in Kconfig I think is not making the constant vs.
> undefined distinction clear. It would have been neater if references to
> undefined symbols simply threw an error, though that gets a bit icky
> with shared Kconfig files, like in the kernel (would need to make sure
> that referenced symbols are always defined or predeclare them as
> undefined).
>
> Some people might object to "1" as well, but it would've been a lesser
> evil I think, once the symbol/value distinction would be clear.
>
> The other option would be to tack some kind of complicated type system
> on top of Kconfig's evaluation semantics. That's overkilling it, IMO.
>
>>
>>
>> >    ("n", "foo bar", etc., are all just constant symbols. Kconfig keeps
>> >    track of them separately from non-constant symbols. The constant
>> >    symbols "n", "m", and "y" are predefined.)
>> >
>> >    If we go with obligatory quotes, I volunteer to explain things in
>> >    kconfig-language.txt at least, to make it clear why you'd see quotes
>> >    in bool/tristate symbols using $(shell). I suspect it wouldn't be
>> >    that tricky to figure out anyway.
>>
>>
>> I think the right thing to do is
>> to fix the code instead of advertising it.
>
> I don't think the behavior is all bad, just obscure. It could easily be
> explained.

Some unrelated rambling on how Kconfig could've done it differently:

Currently, "n"/"m"/"y" are the canonical predefined tristate symbols.
n/m/y just get transformed into those whenever they're looked up, as a
syntactic shorthand (which has probably confused a lot of people).

Another option would've been to have three predefined ordinary symbols
n/m/y, and get rid of "n"/"m"/"y". Then none of the constant (quoted)
symbols would be "magic".

In practice, I think that'd just lead to breaking a different set of
invariants though. You'd get three magic ordinary symbols instead of
three magic constant symbols, and you'd end up with slightly iffy
stuff like defined symbols that have no define location (and probably
other stuff I haven't thought of). Conceptually, it makes sense to
group n/m/y among the constant symbols.

So, I'm not sure changing things there would be an improvement.
Strange as Kconfig is, some parts do actually kinda make sense at some
level.

And if you're aware of how quotes and "n"/"m"/"y" work, and think that
$() stuff ought to produce constant values (like I do... I'll stop
nagging about it soon), then it makes to limit it to within "".

Cheers,
Ulf

Re: [PATCH 07/23] kconfig: add function support and implement 'shell' function

[Ulf Magnusson]

On Mon, Feb 19, 2018 at 09:06:55PM +0100, Ulf Magnusson wrote:
> On Mon, Feb 19, 2018 at 6:50 PM, Ulf Magnusson <ulfalizer@gmail.com> wrote:
> > On Tue, Feb 20, 2018 at 12:57:13AM +0900, Masahiro Yamada wrote:
> >> 2018-02-18 1:16 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> >> > On Sat, Feb 17, 2018 at 03:38:35AM +0900, Masahiro Yamada wrote:
> >> >> This commit adds a new concept 'function' to Kconfig.  A function call
> >> >> resembles a variable reference with arguments, and looks like this:
> >> >>
> >> >>   $(function arg1, arg2, arg3, ...)
> >> >>
> >> >> (Actually, this syntax was inspired by Makefile.)
> >> >>
> >> >> Real examples will look like this:
> >> >>
> >> >>   $(shell true)
> >> >>   $(cc-option -fstackprotector)
> >> >>
> >> >> This commit adds the basic infrastructure to add, delete, evaluate
> >> >> functions.
> >> >>
> >> >> Also, add the first built-in function $(shell ...).  This evaluates
> >> >> to 'y' if the given command exits with 0, 'n' otherwise.
> >> >>
> >> >> I am also planning to support user-defined functions (a.k.a 'macro')
> >> >> for cases where hard-coding is not preferred.
> >> >>
> >> >> If you want to try this feature, the hello-world code is someting below.
> >> >>
> >> >> Example code:
> >> >>
> >> >>   config CC_IS_GCC
> >> >>           bool
> >> >>           default $(shell $CC --version | grep -q gcc)
> >> >>
> >> >>   config CC_IS_CLANG
> >> >>           bool
> >> >>           default $(shell $CC --version | grep -q clang)
> >> >>
> >> >>   config CC_HAS_OZ
> >> >>           bool
> >> >>           default $(shell $CC -Werror -Oz -c -x c /dev/null -o /dev/null)
> >> >>
> >> >> Result:
> >> >>
> >> >>   $ make -s alldefconfig && tail -n 3 .config
> >> >>   CONFIG_CC_IS_GCC=y
> >> >>   # CONFIG_CC_IS_CLANG is not set
> >> >>   # CONFIG_CC_HAS_OZ is not set
> >> >>
> >> >>   $ make CC=clang -s alldefconfig && tail -n 3 .config
> >> >>   # CONFIG_CC_IS_GCC is not set
> >> >>   CONFIG_CC_IS_CLANG=y
> >> >>   CONFIG_CC_HAS_OZ=y
> >> >>
> >> >> A function call can appear anywhere a symbol reference can appear.
> >> >> So, the following code is possible.
> >> >>
> >> >> Example code:
> >> >>
> >> >>   config CC_NAME
> >> >>           string
> >> >>           default "gcc" if $(shell $CC --version | grep -q gcc)
> >> >>           default "clang" if $(shell $CC --version | grep -q clang)
> >> >>           default "unknown compiler"
> >> >>
> >> >> Result:
> >> >>
> >> >>   $ make -s alldefconfig && tail -n 1 .config
> >> >>   CONFIG_CC_NAME="gcc"
> >> >>
> >> >>   $ make CC=clang -s alldefconfig && tail -n 1 .config
> >> >>   CONFIG_CC_NAME="clang"
> >> >>
> >> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> >> >> ---
> >> >>
> >> >> Reminder for myself:
> >> >> Update Documentation/kbuild/kconfig-language.txt
> >> >>
> >> >>
> >> >>  scripts/kconfig/function.c  | 149 ++++++++++++++++++++++++++++++++++++++++++++
> >> >>  scripts/kconfig/lkc_proto.h |   5 ++
> >> >>  scripts/kconfig/util.c      |  46 +++++++++++---
> >> >>  scripts/kconfig/zconf.l     |  38 ++++++++++-
> >> >>  scripts/kconfig/zconf.y     |   9 +++
> >> >>  5 files changed, 238 insertions(+), 9 deletions(-)
> >> >>  create mode 100644 scripts/kconfig/function.c
> >> >>
> >> >> diff --git a/scripts/kconfig/function.c b/scripts/kconfig/function.c
> >> >> new file mode 100644
> >> >> index 0000000..60e59be
> >> >> --- /dev/null
> >> >> +++ b/scripts/kconfig/function.c
> >> >> @@ -0,0 +1,149 @@
> >> >> +// SPDX-License-Identifier: GPL-2.0
> >> >> +//
> >> >> +// Copyright (C) 2018 Masahiro Yamada <yamada.masahiro@socionext.com>
> >> >> +
> >> >> +#include <stdio.h>
> >> >> +#include <stdlib.h>
> >> >> +#include <string.h>
> >> >> +
> >> >> +#include "list.h"
> >> >> +
> >> >> +#define FUNCTION_MAX_ARGS            10
> >> >> +
> >> >> +static LIST_HEAD(function_list);
> >> >> +
> >> >> +struct function {
> >> >> +     const char *name;
> >> >> +     char *(*func)(int argc, char *argv[]);
> >> >> +     struct list_head node;
> >> >> +};
> >> >> +
> >> >> +static struct function *func_lookup(const char *name)
> >> >> +{
> >> >> +     struct function *f;
> >> >> +
> >> >> +     list_for_each_entry(f, &function_list, node) {
> >> >> +             if (!strcmp(name, f->name))
> >> >> +                     return f;
> >> >> +     }
> >> >> +
> >> >> +     return NULL;
> >> >> +}
> >> >> +
> >> >> +static void func_add(const char *name, char *(*func)(int argc, char *argv[]))
> >> >> +{
> >> >> +     struct function *f;
> >> >> +
> >> >> +     f = func_lookup(name);
> >> >> +     if (f) {
> >> >> +             fprintf(stderr, "%s: function already exists. ignored.\n", name);
> >> >> +             return;
> >> >> +     }
> >> >> +
> >> >> +     f = xmalloc(sizeof(*f));
> >> >> +     f->name = name;
> >> >> +     f->func = func;
> >> >> +
> >> >> +     list_add_tail(&f->node, &function_list);
> >> >> +}
> >> >> +
> >> >> +static void func_del(struct function *f)
> >> >> +{
> >> >> +     list_del(&f->node);
> >> >> +     free(f);
> >> >> +}
> >> >> +
> >> >> +static char *func_call(int argc, char *argv[])
> >> >> +{
> >> >> +     struct function *f;
> >> >> +
> >> >> +     f = func_lookup(argv[0]);
> >> >> +     if (!f) {
> >> >> +             fprintf(stderr, "%s: function not found\n", argv[0]);
> >> >> +             return NULL;
> >> >> +     }
> >> >> +
> >> >> +     return f->func(argc, argv);
> >> >> +}
> >> >> +
> >> >> +static char *func_eval(const char *func)
> >> >> +{
> >> >> +     char *expanded, *saveptr, *str, *token, *res;
> >> >> +     const char *delim;
> >> >> +     int argc = 0;
> >> >> +     char *argv[FUNCTION_MAX_ARGS];
> >> >> +
> >> >> +     expanded = expand_string_value(func);
> >> >> +
> >> >> +     str = expanded;
> >> >> +     delim = " ";
> >> >> +
> >> >> +     while ((token = strtok_r(str, delim, &saveptr))) {
> >> >> +             argv[argc++] = token;
> >> >> +             str = NULL;
> >> >> +             delim = ",";
> >> >> +     }
> >> >> +
> >> >> +     res = func_call(argc, argv);
> >> >> +
> >> >> +     free(expanded);
> >> >> +
> >> >> +     return res ?: xstrdup("");
> >> >> +}
> >> >> +
> >> >> +char *func_eval_n(const char *func, size_t n)
> >> >> +{
> >> >> +     char *tmp, *res;
> >> >> +
> >> >> +     tmp = xmalloc(n + 1);
> >> >> +     memcpy(tmp, func, n);
> >> >> +     *(tmp + n) = '\0';
> >> >> +
> >> >> +     res = func_eval(tmp);
> >> >> +
> >> >> +     free(tmp);
> >> >> +
> >> >> +     return res;
> >> >> +}
> >> >> +
> >> >> +/* built-in functions */
> >> >> +static char *do_shell(int argc, char *argv[])
> >> >> +{
> >> >> +     static const char *pre = "(";
> >> >> +     static const char *post = ") >/dev/null 2>&1";
> >> >> +     char *cmd;
> >> >> +     int ret;
> >> >> +
> >> >> +     if (argc != 2)
> >> >> +             return NULL;
> >> >> +
> >> >> +     /*
> >> >> +      * Surround the command with ( ) in case it is piped commands.
> >> >> +      * Also, redirect stdout and stderr to /dev/null.
> >> >> +      */
> >> >> +     cmd = xmalloc(strlen(pre) + strlen(argv[1]) + strlen(post) + 1);
> >> >> +     strcpy(cmd, pre);
> >> >> +     strcat(cmd, argv[1]);
> >> >> +     strcat(cmd, post);
> >> >> +
> >> >> +     ret = system(cmd);
> >> >> +
> >> >> +     free(cmd);
> >> >> +
> >> >> +     return xstrdup(ret == 0 ? "y" : "n");
> >> >> +}
> >> >> +
> >> >> +void func_init(void)
> >> >> +{
> >> >> +     /* register built-in functions */
> >> >> +     func_add("shell", do_shell);
> >> >> +}
> >> >> +
> >> >> +void func_exit(void)
> >> >> +{
> >> >> +     struct function *f, *tmp;
> >> >> +
> >> >> +     /* unregister all functions */
> >> >> +     list_for_each_entry_safe(f, tmp, &function_list, node)
> >> >> +             func_del(f);
> >> >> +}
> >> >> diff --git a/scripts/kconfig/lkc_proto.h b/scripts/kconfig/lkc_proto.h
> >> >> index 9884adc..09a4f53 100644
> >> >> --- a/scripts/kconfig/lkc_proto.h
> >> >> +++ b/scripts/kconfig/lkc_proto.h
> >> >> @@ -48,5 +48,10 @@ const char * sym_get_string_value(struct symbol *sym);
> >> >>
> >> >>  const char * prop_get_type_name(enum prop_type type);
> >> >>
> >> >> +/* function.c */
> >> >> +char *func_eval_n(const char *func, size_t n);
> >> >> +void func_init(void);
> >> >> +void func_exit(void);
> >> >> +
> >> >>  /* expr.c */
> >> >>  void expr_print(struct expr *e, void (*fn)(void *, struct symbol *, const char *), void *data, int prevtoken);
> >> >> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
> >> >> index dddf85b..ed89fb9 100644
> >> >> --- a/scripts/kconfig/util.c
> >> >> +++ b/scripts/kconfig/util.c
> >> >> @@ -93,9 +93,10 @@ static char *env_expand_n(const char *name, size_t n)
> >> >>  }
> >> >>
> >> >>  /*
> >> >> - * Expand environments embedded in the string given in argument. Environments
> >> >> - * to be expanded shall be prefixed by a '$'. Unknown environment expands to
> >> >> - * the empty string.
> >> >> + * Expand environments and functions embedded in the string given in argument.
> >> >> + * Environments to be expanded shall be prefixed by a '$'. Functions to be
> >> >> + * evaluated shall be surrounded by $(). Unknown environment/function expands
> >> >> + * to the empty string.
> >> >>   */
> >> >>  char *expand_string_value(const char *in)
> >> >>  {
> >> >> @@ -113,11 +114,40 @@ char *expand_string_value(const char *in)
> >> >>       while ((p = strchr(in, '$'))) {
> >> >>               char *new;
> >> >>
> >> >> -             q = p + 1;
> >> >> -             while (isalnum(*q) || *q == '_')
> >> >> -                     q++;
> >> >> -
> >> >> -             new = env_expand_n(p + 1, q - p - 1);
> >> >> +             /*
> >> >> +              * If the next character is '(', it is a function.
> >> >> +              * Otherwise, environment.
> >> >> +              */
> >> >> +             if (*(p + 1) == '(') {
> >> >> +                     int nest = 0;
> >> >> +
> >> >> +                     q = p + 2;
> >> >> +                     while (1) {
> >> >> +                             if (*q == '\0') {
> >> >> +                                     fprintf(stderr,
> >> >> +                                             "unterminated function: %s\n",
> >> >> +                                             p);
> >> >> +                                     new = xstrdup("");
> >> >> +                                     break;
> >> >> +                             } else if (*q == '(') {
> >> >> +                                     nest++;
> >> >> +                             } else if (*q == ')') {
> >> >> +                                     if (nest-- == 0) {
> >> >> +                                             new = func_eval_n(p + 2,
> >> >> +                                                               q - p - 2);
> >> >> +                                             q++;
> >> >> +                                             break;
> >> >> +                                     }
> >> >> +                             }
> >> >> +                             q++;
> >> >> +                     }
> >> >> +             } else {
> >> >> +                     q = p + 1;
> >> >> +                     while (isalnum(*q) || *q == '_')
> >> >> +                             q++;
> >> >> +
> >> >> +                     new = env_expand_n(p + 1, q - p - 1);
> >> >> +             }
> >> >>
> >> >>               reslen = strlen(res) + (p - in) + strlen(new) + 1;
> >> >>               res = xrealloc(res, reslen);
> >> >> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
> >> >> index 0d89ea6..f433ab0 100644
> >> >> --- a/scripts/kconfig/zconf.l
> >> >> +++ b/scripts/kconfig/zconf.l
> >> >> @@ -1,7 +1,7 @@
> >> >>  %option nostdinit noyywrap never-interactive full ecs
> >> >>  %option 8bit nodefault perf-report perf-report
> >> >>  %option noinput
> >> >> -%x COMMAND HELP STRING PARAM
> >> >> +%x COMMAND HELP STRING PARAM FUNCTION
> >> >>  %{
> >> >>  /*
> >> >>   * Copyright (C) 2002 Roman Zippel <zippel@linux-m68k.org>
> >> >> @@ -25,6 +25,7 @@ static struct {
> >> >>
> >> >>  static char *text;
> >> >>  static int text_size, text_asize;
> >> >> +static int function_nest;
> >> >>
> >> >>  struct buffer {
> >> >>       struct buffer *parent;
> >> >> @@ -138,6 +139,12 @@ n        [$A-Za-z0-9_-]
> >> >>               new_string();
> >> >>               BEGIN(STRING);
> >> >>       }
> >> >> +     "$("    {
> >> >> +             new_string();
> >> >> +             append_string(yytext, yyleng);
> >> >> +             function_nest = 0;
> >> >> +             BEGIN(FUNCTION);
> >> >> +     }
> >> >>       \n      BEGIN(INITIAL); current_file->lineno++; return T_EOL;
> >> >>       ({n}|[/.])+     {
> >> >>               const struct kconf_id *id = kconf_id_lookup(yytext, yyleng);
> >> >> @@ -196,6 +203,35 @@ n        [$A-Za-z0-9_-]
> >> >>       }
> >> >>  }
> >> >>
> >> >> +<FUNCTION>{
> >> >> +     [^()\n]* {
> >> >> +             append_string(yytext, yyleng);
> >> >> +     }
> >> >> +     "(" {
> >> >> +             append_string(yytext, yyleng);
> >> >> +             function_nest++;
> >> >> +     }
> >> >> +     ")" {
> >> >> +             append_string(yytext, yyleng);
> >> >> +             if (function_nest-- == 0) {
> >> >> +                     BEGIN(PARAM);
> >> >> +                     yylval.string = text;
> >> >> +                     return T_WORD;
> >> >
> >> > T_WORD_QUOTE (which would turn into a constant symbol in most contexts)
> >> > would be better here, IMO. That would turn $(foo) into just an alias for
> >> > "$(foo)".
> >> >
> >> > See below.
> >> >
> >> >> +             }
> >> >> +     }
> >> >> +     \n {
> >> >> +             fprintf(stderr,
> >> >> +                     "%s:%d:warning: multi-line function not supported\n",
> >> >> +                     zconf_curname(), zconf_lineno());
> >> >> +             current_file->lineno++;
> >> >> +             BEGIN(INITIAL);
> >> >> +             return T_EOL;
> >> >> +     }
> >> >> +     <<EOF>> {
> >> >> +             BEGIN(INITIAL);
> >> >> +     }
> >> >> +}
> >> >> +
> >> >>  <HELP>{
> >> >>       [ \t]+  {
> >> >>               ts = 0;
> >> >> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
> >> >> index 784083d..d9977de 100644
> >> >> --- a/scripts/kconfig/zconf.y
> >> >> +++ b/scripts/kconfig/zconf.y
> >> >> @@ -534,11 +534,19 @@ void conf_parse(const char *name)
> >> >>
> >> >>       zconf_initscan(name);
> >> >>
> >> >> +     func_init();
> >> >>       _menu_init();
> >> >>
> >> >>       if (getenv("ZCONF_DEBUG"))
> >> >>               yydebug = 1;
> >> >>       yyparse();
> >> >> +
> >> >> +     /*
> >> >> +      * Currently, functions are evaluated only when Kconfig files are
> >> >> +      * parsed. We can free functions here.
> >> >> +      */
> >> >> +     func_exit();
> >> >> +
> >> >>       if (yynerrs)
> >> >>               exit(1);
> >> >>       if (!modules_sym)
> >> >> @@ -778,4 +786,5 @@ void zconfdump(FILE *out)
> >> >>  #include "confdata.c"
> >> >>  #include "expr.c"
> >> >>  #include "symbol.c"
> >> >> +#include "function.c"
> >> >>  #include "menu.c"
> >> >> --
> >> >> 2.7.4
> >> >>
> >> >
> >> > Here's a simplification idea I'm throwing out there:
> >> >
> >> > What about only allowing $ENV and $() within quotes, and just having
> >> > them always do simple text substitution (so that they'd indirectly
> >> > always generate T_WORD_QUOTE tokens)?
> >>
> >>
> >> I ended up with a new state <FUNCTION>,
> >> but the real problem is the lexer is too ugly.
> >>
> >> So, maybe, the solution is not to make it into a string,
> >> but to re-write the lexer.
> >>
> >>
> >> <STRING> and <FUNCTION> are almost the same in the sense
> >> that whitespaces do not split tokens.
> >>
> >> The difference is the characters to start/end with.
> >>
> >>   ' ... '
> >>   " ... "
> >>   ( ... )
> >>
> >> If we encounter with the 3 special characters,  ' , '', (,
> >> then we enter into <STRING> state,
> >> then it ends with ', ", ), respectively.
> >> (but we need to take care of nesting, escaping)
> >>
> >> Double-surrounding like  "$(cc-option -Oz)"
> >> looks ugly to me.
> >
> > It makes it clear that all we're doing is string interpolation, which is
> > the important thing.
> >
> > The simplest and to me sanest way to implement $(cc-option -Oz) is as
> > "$(cc-option -Oz)" (a SYMBOL_CONST symbol, which lives in a separate
> > namespace -- see below), and at that point $(cc-option -Oz) would just
> > be syntactic sugar for "$(cc-option -Oz)".
> >
> > IMO, that syntactic sugar just hides how simple the behavior really is
> > and makes Kconfig more confusing. Plus it complicates the
> > implementation.
> >
> >>
> >>
> >>
> >>
> >>
> >> > Pros:
> >> >
> >> >  - Zero new syntax outside of strings (until the macro stuff).
> >> >
> >> >  - Makes the behavior and limitations of the syntax obvious: You can't
> >> >    have $(foo) expand to a full expression, only to (possibly part of) a
> >> >    value. This is a good limitation, IMO, and it's already there.
> >> >
> >> >  - Super simple and straightforward Kconfig implementation. All the new
> >> >    magic would happen in expand_string_value().
> >> >
> >> > Neutral:
> >> >
> >> >  - Just as general where it matters. Take something like
> >> >
> >> >      default "$(foo)"
> >> >
> >> >    If $(foo) happens to expand to "y", then that will do its usual thing
> >> >    for a bool/tristate symbol. Same for string symbols, etc. It'd just
> >> >    be a thin preprocessing step on constant symbol values.
> >> >
> >> >  - The only loss in generality would be that you can no longer have
> >> >    a function expand to the name of non-constant symbol. For example,
> >> >    take the following:
> >> >
> >> >         default $(foo)
> >> >
> >> >    If $(foo) expands to MY_SYMBOL, then that would work as
> >> >
> >> >         default MY_SYMBOL
> >>
> >>
> >> Probably, this is a "do not do this".
> >>
> >>
> >> If the symbol is 'int' type,
> >> and  $MY_NUMBER is expanded to 1,
> >> it is our intention.
> >
> > Just to summarize the tradeoffs in each approach:
> >
> > Approach 1:
> > Let $(fn ...) expand to FOO and reference the symbol FOO if it exists
> >
> > Approach 2:
> > Let $(fn ...) expand to "FOO", which could never reference existing
> > symbols
> >
> >
> > Pros of approach 1:
> >
> >   - If someone legitimately wants to do indirection by generating
> >     Kconfig symbol names, they can.
> >
> > Cons of approach 1:
> >
> >   - If $(fn ...) happens to expand to the name of an existing symbol, it
> >     will reference it, even if it wasn't intended. This might get really
> >     tricky to debug, as it probably won't be obvious that this is how it
> >     works.
> >
> >
> > Pros of approach 2:
> >
> >   - You can never accidentally reference an existing symbol when it
> >     wasn't intended. You always generate a "value". Constant symbols
> >     live in a separate namespace.
> >
> >   - The behavior is simpler to understand, IMO. $(fn ...) and $FOO
> >     consistently deal with values and just do string interpolation. You
> >     don't get "either a symbol or a value" depending on what's already
> >     defined.
> >
> > Cons of approach 2:
> >
> >   - You can't do indirection by generating Kconfig symbol names. I
> >     suspect there'd always be cleaner ways to do that in practice, but
> >     I'm showing my bias here.
> >
> >
> > Approach 2 seems much simpler and less magical to me.
> >
> >>
> >>
> >>
> >>
> >>
> >> >   (I.e., it would default to the value of the symbol MY_SYMBOL.)
> >> >
> >> >   With the quotes, it would instead work as
> >> >
> >> >      default "MY_SYMBOL"
> >> >
> >> >   IMO, the second version is less confusing, and deliberately using the
> >> >   first version seems like a bad idea (there's likely to be cleaner ways
> >> >   to do the indirection in plain Kconfig).
> >> >
> >> >   This is also why I think T_WORD_QUOTE is better in the code above. It
> >> >   would make $(foo) work more like a shorthand for "$(foo)".
> >> >
> >> >
> >> > Cons:
> >> >
> >> >  - Bit more typing to add the quotes
> >> >
> >> >  - Maybe it isn't widely known that "n", "m", "y" work just like n, m, y
> >> >    for bool and tristate symbols (the latter automatically get converted
> >> >    to the former), though you see them quite a lot in Kconfig files.
> >>
> >> Not only bool/tristate, but also int/hex.
> >> "1" is equivalent to 1.
> >>
> >>
> >> Kconfig is screwed up in handling of literals.
> >
> > I don't actually think the design is horrible here (though it has
> > flaws). The biggest problem is that it's poorly documented and
> > understood.
> >
> > Expressions are composed of symbols and operators. You have ordinary
> > symbols and constant symbols (quoted stuff, "values" if you will). All
> > symbols have a tristate value and a string value. Which value gets used,
> > and how, depends on the operator.
> >
> > Note that constant symbols are actually stored separately from
> > nonconstant symbols internally (see SYMBOL_CONST). 1 and "1" are not the
> > same symbol (this is easy to miss unless you've spent too much
> > deciphering Kconfig's code). The reason a plain 1 works is that
> > undefined symbols share many behaviors with constant symbols.
> >
> > The biggest mess-up in Kconfig I think is not making the constant vs.
> > undefined distinction clear. It would have been neater if references to
> > undefined symbols simply threw an error, though that gets a bit icky
> > with shared Kconfig files, like in the kernel (would need to make sure
> > that referenced symbols are always defined or predeclare them as
> > undefined).
> >
> > Some people might object to "1" as well, but it would've been a lesser
> > evil I think, once the symbol/value distinction would be clear.
> >
> > The other option would be to tack some kind of complicated type system
> > on top of Kconfig's evaluation semantics. That's overkilling it, IMO.
> >
> >>
> >>
> >> >    ("n", "foo bar", etc., are all just constant symbols. Kconfig keeps
> >> >    track of them separately from non-constant symbols. The constant
> >> >    symbols "n", "m", and "y" are predefined.)
> >> >
> >> >    If we go with obligatory quotes, I volunteer to explain things in
> >> >    kconfig-language.txt at least, to make it clear why you'd see quotes
> >> >    in bool/tristate symbols using $(shell). I suspect it wouldn't be
> >> >    that tricky to figure out anyway.
> >>
> >>
> >> I think the right thing to do is
> >> to fix the code instead of advertising it.
> >
> > I don't think the behavior is all bad, just obscure. It could easily be
> > explained.
> 
> Some unrelated rambling on how Kconfig could've done it differently:
> 
> Currently, "n"/"m"/"y" are the canonical predefined tristate symbols.
> n/m/y just get transformed into those whenever they're looked up, as a
> syntactic shorthand (which has probably confused a lot of people).
> 
> Another option would've been to have three predefined ordinary symbols
> n/m/y, and get rid of "n"/"m"/"y". Then none of the constant (quoted)
> symbols would be "magic".
> 
> In practice, I think that'd just lead to breaking a different set of
> invariants though. You'd get three magic ordinary symbols instead of
> three magic constant symbols, and you'd end up with slightly iffy
> stuff like defined symbols that have no define location (and probably
> other stuff I haven't thought of). Conceptually, it makes sense to
> group n/m/y among the constant symbols.
> 
> So, I'm not sure changing things there would be an improvement.
> Strange as Kconfig is, some parts do actually kinda make sense at some
> level.
> 
> And if you're aware of how quotes and "n"/"m"/"y" work, and think that
> $() stuff ought to produce constant values (like I do... I'll stop
> nagging about it soon), then it makes to limit it to within "".
> 
> Cheers,
> Ulf

Here's how the description in kconfig-language.txt might look by the
way. It might clarify why I think it would be really simple:

	Within quotes, some simple forms of string interpolation are
	supported:
	
	  - $ENV is replaced by the value of the environment variable
	    ENV.

	  - $(success cmd) is replaced by "y" if the exit status of cmd
	    is 0, and with "n" otherwise. cmd is passed as-is to the
	    shell and may contain e.g. spaces or pipes.

	  - $(stdout cmd) is replaced by the first line cmd writes to
	    stdout. Like for $(success), cmd is passed as-is to the
	    shell.

	  - To avoid repetition, a custom macro can be defined as
	    follows:

	        macro my-macro "$(success cmd $(1) --flag $(2))"

	    $(my-macro foo bar) is then be replaced by
	    $(success cmd foo --flag bar), which is in turn replaced
	    by either "n" or "y".

	For bool and tristate symbols, note that "n", "m", and "y" are
	synonymous with n, m, and y (the latter are a syntactic
	shorthand for the former). This allows tests to be implemented
	as follows:

	    config CC_IS_GCC
	        def_bool "$(success $CC --version | grep -q gcc)"

	Assuming that CC is GCC, this would turn into

	    config CC_IS_GCC
		def_bool "y"

	Here's an example that makes use of $(stdout):

	    config DEFCONFIG_LIST
	        ...
		default "/boot/config-$(stdout uname --release)"
		...

	After expansion, this might turn into

	    config DEFCONFIG_LIST
	        ...
		default "/boot/config-4.13.0-32-generic"
	
	Note that this syntax also works for other contexts where
	quoted strings appear, e.g.

	    source "arch/$SRCARCH/Kconfig"


	String expansion is done during parsing, meaning you can't look
	up values of config symbols inside strings. Macros must be
	defined before the point where they are used.


It seems simple and gotcha-free to me. All the expansion got could be
done during tokenization.

Cheers,
Ulf

Re: [PATCH 10/23] stack-protector: test compiler capability in Kconfig and drop AUTO mode

[Masahiro Yamada]

2018-02-17 3:38 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:
> Add CC_HAS_STACKPROTECTOR(_STRONG) to test if the compiler supports
> -fstack-protector(-strong) option.
>
> X86 has additional shell scripts in case the compiler supports the
> option, but generates broken code.  I added CC_HAS_SANE_STACKPROTECTOR
> to test this.  I had to add -m32 to gcc-x86_32-has-stack-protector.sh
> to make it work correctly.
>
> If the compiler does not support the option, the menu is automatically
> hidden.  If _STRONG is not supported, it will fall back to _REGULAR.
> This means, _AUTO is implicitly supported in the dependency solver of
> Kconfig, hence removed.
>
> I also turned the 'choice' into only two boolean symbols.  The use of
> 'choice' is not a good idea here, because all of all{yes,mod,no}config
> would choose the first visible value, while we want allnoconfig to
> disable as many features as possible.
>
> I did not add CC_HAS_STACKPROTECTOR_NONE in the hope that GCC versions
> we support will recognize -fno-stack-protector.
>
> If this turns out to be a problem, it will be possible to do this:
>
> stackp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector
> stackp-flags-$(CONFIG_CC_STACKPROTECTOR)          := -fstack-protector
> stackp-flags-$(CONFIG_CC_STACKPROTECTOR_STRONG)   := -fstack-protector-strong
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
>  Makefile                                  | 93 ++-----------------------------
>  arch/Kconfig                              | 37 ++++++------
>  arch/x86/Kconfig                          |  8 ++-
>  scripts/gcc-x86_32-has-stack-protector.sh |  7 +--
>  scripts/gcc-x86_64-has-stack-protector.sh |  5 --
>  5 files changed, 30 insertions(+), 120 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 9a8c689..e9fc7c9 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -675,55 +675,11 @@ ifneq ($(CONFIG_FRAME_WARN),0)
>  KBUILD_CFLAGS += $(call cc-option,-Wframe-larger-than=${CONFIG_FRAME_WARN})
>  endif
>
> -# This selects the stack protector compiler flag. Testing it is delayed
> -# until after .config has been reprocessed, in the prepare-compiler-check
> -# target.
> -ifdef CONFIG_CC_STACKPROTECTOR_AUTO
> -  stackp-flag := $(call cc-option,-fstack-protector-strong,$(call cc-option,-fstack-protector))
> -  stackp-name := AUTO
> -else
> -ifdef CONFIG_CC_STACKPROTECTOR_REGULAR
> -  stackp-flag := -fstack-protector
> -  stackp-name := REGULAR
> -else
> -ifdef CONFIG_CC_STACKPROTECTOR_STRONG
> -  stackp-flag := -fstack-protector-strong
> -  stackp-name := STRONG
> -else
> -  # If either there is no stack protector for this architecture or
> -  # CONFIG_CC_STACKPROTECTOR_NONE is selected, we're done, and $(stackp-name)
> -  # is empty, skipping all remaining stack protector tests.
> -  #
> -  # Force off for distro compilers that enable stack protector by default.
> -  KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
> -endif
> -endif
> -endif
> -# Find arch-specific stack protector compiler sanity-checking script.
> -ifdef stackp-name
> -ifneq ($(stackp-flag),)
> -  stackp-path := $(srctree)/scripts/gcc-$(SRCARCH)_$(BITS)-has-stack-protector.sh
> -  stackp-check := $(wildcard $(stackp-path))
> -  # If the wildcard test matches a test script, run it to check functionality.
> -  ifdef stackp-check
> -    ifneq ($(shell $(CONFIG_SHELL) $(stackp-check) $(CC) $(KBUILD_CPPFLAGS) $(biarch)),y)
> -      stackp-broken := y
> -    endif
> -  endif
> -  ifndef stackp-broken
> -    # If the stack protector is functional, enable code that depends on it.
> -    KBUILD_CPPFLAGS += -DCONFIG_CC_STACKPROTECTOR
> -    # Either we've already detected the flag (for AUTO) or we'll fail the
> -    # build in the prepare-compiler-check rule (for specific flag).
> -    KBUILD_CFLAGS += $(stackp-flag)
> -  else
> -    # We have to make sure stack protector is unconditionally disabled if
> -    # the compiler is broken (in case we're going to continue the build in
> -    # AUTO mode).
> -    KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
> -  endif
> -endif
> -endif
> +stackp-flags-y                                 := -fno-stack-protector
> +stackp-flags-$(CONFIG_CC_STACKPROTECTOR)       := -fstack-protector
> +stackp-flags-$(CONFIG_CC_STACKPROTECTOR_STRONG)        := -fstack-protector-strong
> +
> +KBUILD_CFLAGS += $(stackp-flags-y)
>
>  ifeq ($(cc-name),clang)
>  KBUILD_CPPFLAGS += $(call cc-option,-Qunused-arguments,)
> @@ -1079,7 +1035,7 @@ endif
>  # prepare2 creates a makefile if using a separate output directory.
>  # From this point forward, .config has been reprocessed, so any rules
>  # that need to depend on updated CONFIG_* values can be checked here.
> -prepare2: prepare3 prepare-compiler-check outputmakefile asm-generic
> +prepare2: prepare3 outputmakefile asm-generic
>
>  prepare1: prepare2 $(version_h) include/generated/utsrelease.h \
>                     include/config/auto.conf
> @@ -1105,43 +1061,6 @@ uapi-asm-generic:
>  PHONY += prepare-objtool
>  prepare-objtool: $(objtool_target)
>
> -# Check for CONFIG flags that require compiler support. Abort the build
> -# after .config has been processed, but before the kernel build starts.
> -#
> -# For security-sensitive CONFIG options, we don't want to fallback and/or
> -# silently change which compiler flags will be used, since that leads to
> -# producing kernels with different security feature characteristics
> -# depending on the compiler used. (For example, "But I selected
> -# CC_STACKPROTECTOR_STRONG! Why did it build with _REGULAR?!")
> -PHONY += prepare-compiler-check
> -prepare-compiler-check: FORCE
> -# Make sure compiler supports requested stack protector flag.
> -ifdef stackp-name
> -  # Warn about CONFIG_CC_STACKPROTECTOR_AUTO having found no option.
> -  ifeq ($(stackp-flag),)
> -       @echo CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
> -                 Compiler does not support any known stack-protector >&2
> -  else
> -  # Fail if specifically requested stack protector is missing.
> -  ifeq ($(call cc-option, $(stackp-flag)),)
> -       @echo Cannot use CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
> -                 $(stackp-flag) not supported by compiler >&2 && exit 1
> -  endif
> -  endif
> -endif
> -# Make sure compiler does not have buggy stack-protector support. If a
> -# specific stack-protector was requested, fail the build, otherwise warn.
> -ifdef stackp-broken
> -  ifeq ($(stackp-name),AUTO)
> -       @echo CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
> -                  $(stackp-flag) available but compiler is broken: disabling >&2
> -  else
> -       @echo Cannot use CONFIG_CC_STACKPROTECTOR_$(stackp-name): \
> -                  $(stackp-flag) available but compiler is broken >&2 && exit 1
> -  endif
> -endif
> -       @:
> -
>  # Generate some files
>  # ---------------------------------------------------------------------------
>
> diff --git a/arch/Kconfig b/arch/Kconfig
> index 76c0b54..9b7a628 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -535,13 +535,21 @@ config HAVE_CC_STACKPROTECTOR
>         bool
>         help
>           An arch should select this symbol if:
> -         - its compiler supports the -fstack-protector option
>           - it has implemented a stack canary (e.g. __stack_chk_guard)
>
> -choice
> -       prompt "Stack Protector buffer overflow detection"
> +config CC_HAS_STACKPROTECTOR
> +       bool
> +       default $(cc-option -fstack-protector)
> +
> +config CC_HAS_STACKPROTECTOR_STRONG
> +       bool
> +       default $(cc-option -fstack-protector-strong)
> +
> +config CC_STACKPROTECTOR
> +       bool "Stack Protector buffer overflow detection"
>         depends on HAVE_CC_STACKPROTECTOR
> -       default CC_STACKPROTECTOR_AUTO
> +       depends on CC_HAS_STACKPROTECTOR
> +       default y
>         help



CC_HAS_STACKPROTECTOR is not mandatory in this case
because we can directly describe $(cc-option ...)
in 'depends on' context, like this:


config CC_STACKPROTECTOR
       bool "Stack Protector buffer overflow detection"
       depends on HAVE_CC_STACKPROTECTOR
       depends on $(cc-option -fstack-protector-strong)
       default y



The difference is CONFIG_CC_HAS_STACKPROTECTOR
will not be written into the .config file.


Maybe, is this useful information?

You can check .config in case
"BTW, can my compiler support this flag?"


I will keep CC_HAS_ symbols,
but I am open to this item.


-- 
Best Regards
Masahiro Yamada

Re: [PATCH 11/23] kconfig: add 'shell-stdout' function

[Masahiro Yamada]

2018-02-20 3:01 GMT+09:00 Linus Torvalds <torvalds@linux-foundation.org>:
> On Mon, Feb 19, 2018 at 9:44 AM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
>>
>> I do like your "success"/"stdout" more than "shell"/"shell-stdout",
>> because with that naming I don't get the feeling that one should
>> subsume the other.
>
> Hmm. Thinking about it some more, I really would prefer just "$(shell
> ...)" everywhere.
>
> But it would be nice if perhaps the error handling would match the
> context somehow.
>
> I'm wondering if this might tie into the whole quoting discussion in
> the other thread.
>
> Because the rule could be:
>
> (a) unquoted $(shell ) is a bool, and failing is ok (and turns into
> y/n depending on whether successful or failing)
>
> So
>
>   config CC_IS_GCC
>           bool
>           default $(shell $CC --version | grep -q gcc)
>
> works automatically.
>
> (b) but with quoting, $(shell ) is a string, and failing is an error
>
> So
>
>   config GCC_VERSION
>           int
>           default "$(shell-stdout $srctree/scripts/gcc-version.sh $CC
> | sed 's/^0*//')" if CC_IS_GCC
>           default 0
>
> would need those quotes, and if the shell-script returns a failure,
> we'd _abort_.


GCC_VERSION is int type.

Setting aside the Kconfig internal, I prefer 50700 to "50700"

According to my common sense, I do not want to quote integers.




IMO, I prefer to use different names for different purpose.
So, 'stdout' and 'success' look good to me.



BTW, I noticed just one built-in function is enough
because 'success' can be derived from 'stdout'.


So, my plan is, implement $(shell ...) as a built-in function.
This returns the stdout from the command.


Then, implement 'success' as a textual shorthand
by using macro, like this:

macro success $(shell ($(1) && echo y) || echo n)


macro can be expanded recursively, so cc-option
can be implemented based on 'success' macro.





-- 
Best Regards
Masahiro Yamada

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:

>>
>> I'm not happy that we in one context can reference CONFIG variables
>> directly, but inside the $(call ...) and $(shell ...) needs the $ prefix.
>> But I could not come up with something un-ambigious where this could be avoided.
>
> I think we should be careful about allowing references to config
> symbols. It mixes up the parsing and evaluation phases, since $() is
> expanded during parsing (which I consider a feature and think is
> needed to retain sanity).
>
> Patch 06/23 removes the last existing instance of symbol references in
> strings by getting rid of 'option env'. That's an improvement to me.
> We shouldn't add it back.


This is really important design decision,
so I'd like to hear a little more from experts.


For example, x86 allows users to choose sub-arch, either 'i386' or 'x86_64'.

https://github.com/torvalds/linux/blob/v4.16-rc2/arch/x86/Kconfig#L4



If the user toggles CONFIG_64BIT,
the bi-arch compiler will work in a slightly different mode
(at least, back-end parts)

So, my question is, is there a case,

$(cc-option, -m32 -foo) is y, but
$(cc-option, -m64 -foo) is n  ?
(or vice versa)


If the answer is yes, $(cc-option -foo) would have to be re-calculated
every time CONFIG_64BIT is toggled.

This is what I'd like to avoid, though.



-- 
Best Regards
Masahiro Yamada

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Arnd Bergmann]

On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>
>>>
>>> I'm not happy that we in one context can reference CONFIG variables
>>> directly, but inside the $(call ...) and $(shell ...) needs the $ prefix.
>>> But I could not come up with something un-ambigious where this could be avoided.
>>
>> I think we should be careful about allowing references to config
>> symbols. It mixes up the parsing and evaluation phases, since $() is
>> expanded during parsing (which I consider a feature and think is
>> needed to retain sanity).
>>
>> Patch 06/23 removes the last existing instance of symbol references in
>> strings by getting rid of 'option env'. That's an improvement to me.
>> We shouldn't add it back.
>
>
> This is really important design decision,
> so I'd like to hear a little more from experts.
>
>
> For example, x86 allows users to choose sub-arch, either 'i386' or 'x86_64'.
>
> https://github.com/torvalds/linux/blob/v4.16-rc2/arch/x86/Kconfig#L4
>
>
>
> If the user toggles CONFIG_64BIT,
> the bi-arch compiler will work in a slightly different mode
> (at least, back-end parts)
>
> So, my question is, is there a case,
>
> $(cc-option, -m32 -foo) is y, but
> $(cc-option, -m64 -foo) is n  ?
> (or vice versa)
>
>
> If the answer is yes, $(cc-option -foo) would have to be re-calculated
> every time CONFIG_64BIT is toggled.
>
> This is what I'd like to avoid, though.

The -m32/-m64 trick (and -mbig-endian/-mlittle-endian on other architectures
as well as a couple of other flags) only works if the compiler is configured to
support it. In other cases (e.g. big-endian xtensa), the kernel always
detects what the compiler does and silently configures itself to match
using Makefile logic.

On x86, compilers are usually built as bi-arch, but you can build one that
only allows one of them.

I can see two reasonable ways out:

- we don't use  $(cc-option -foo) in a case like this, and instead require the
  user to have a matching toolchain.
- we could make the 32/64 selection on x86 a 'choice' statement where
  each option depends on both the ARCH= variable and the
  $(cc-option, -m32)/ $(cc-option, -m64) output.

       Arnd

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>>
>>>>
>>>> I'm not happy that we in one context can reference CONFIG variables
>>>> directly, but inside the $(call ...) and $(shell ...) needs the $ prefix.
>>>> But I could not come up with something un-ambigious where this could be avoided.
>>>
>>> I think we should be careful about allowing references to config
>>> symbols. It mixes up the parsing and evaluation phases, since $() is
>>> expanded during parsing (which I consider a feature and think is
>>> needed to retain sanity).
>>>
>>> Patch 06/23 removes the last existing instance of symbol references in
>>> strings by getting rid of 'option env'. That's an improvement to me.
>>> We shouldn't add it back.
>>
>>
>> This is really important design decision,
>> so I'd like to hear a little more from experts.
>>
>>
>> For example, x86 allows users to choose sub-arch, either 'i386' or 'x86_64'.
>>
>> https://github.com/torvalds/linux/blob/v4.16-rc2/arch/x86/Kconfig#L4
>>
>>
>>
>> If the user toggles CONFIG_64BIT,
>> the bi-arch compiler will work in a slightly different mode
>> (at least, back-end parts)
>>
>> So, my question is, is there a case,
>>
>> $(cc-option, -m32 -foo) is y, but
>> $(cc-option, -m64 -foo) is n  ?
>> (or vice versa)
>>
>>
>> If the answer is yes, $(cc-option -foo) would have to be re-calculated
>> every time CONFIG_64BIT is toggled.
>>
>> This is what I'd like to avoid, though.
>
> The -m32/-m64 trick (and -mbig-endian/-mlittle-endian on other architectures
> as well as a couple of other flags) only works if the compiler is configured to
> support it. In other cases (e.g. big-endian xtensa), the kernel always
> detects what the compiler does and silently configures itself to match
> using Makefile logic.
>
> On x86, compilers are usually built as bi-arch, but you can build one that
> only allows one of them.
>
> I can see two reasonable ways out:
>
> - we don't use  $(cc-option -foo) in a case like this, and instead require the
>   user to have a matching toolchain.
> - we could make the 32/64 selection on x86 a 'choice' statement where
>   each option depends on both the ARCH= variable and the
>   $(cc-option, -m32)/ $(cc-option, -m64) output.
>
>        Arnd



Let me clarify my concern.

When we test the compiler flag, is there a case
where a particular flag depends on -m{32,64} ?

For example, is there a compiler that supports -fstack-protector
for 64bit mode, but unsupports it for 32bit mode?

  $(cc-option -m32)                     ->  y
  $(cc-option -m64)                     ->  y
  $(cc-option -fstack-protector)        ->  y
  $(cc-option -m32 -fstack-protector)   ->  n
  $(cc-option -m64 -fstack-protector)   ->  y

I guess this is unlikely to happen,
but I am not whether it is zero possibility.

If this could happen,
$(cc-option ) must be evaluated together with
correct bi-arch option (either -m32 or -m64).


Currently, -m32/-m64 is specified in Makefile,
but we are moving compiler tests to Kconfig
and, CONFIG_64BIT can be dynamically toggled in Kconfig.




-- 
Best Regards
Masahiro Yamada

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Arnd Bergmann]

On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
>> <yamada.masahiro@socionext.com> wrote:
>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>
> Let me clarify my concern.
>
> When we test the compiler flag, is there a case
> where a particular flag depends on -m{32,64} ?
>
> For example, is there a compiler that supports -fstack-protector
> for 64bit mode, but unsupports it for 32bit mode?
>
>   $(cc-option -m32)                     ->  y
>   $(cc-option -m64)                     ->  y
>   $(cc-option -fstack-protector)        ->  y
>   $(cc-option -m32 -fstack-protector)   ->  n
>   $(cc-option -m64 -fstack-protector)   ->  y
>
> I guess this is unlikely to happen,
> but I am not whether it is zero possibility.
>
> If this could happen,
> $(cc-option ) must be evaluated together with
> correct bi-arch option (either -m32 or -m64).
>
>
> Currently, -m32/-m64 is specified in Makefile,
> but we are moving compiler tests to Kconfig
> and, CONFIG_64BIT can be dynamically toggled in Kconfig.

I don't think it can happen for this particular combination (stack protector
and word size), but I'm sure we'll eventually run into options that
need to be tested in combination. For the current CFLAGS_KERNEL
setting, we definitely have the case of needing the variables to be
evaluated in a specific order.

      Arnd

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
>>> <yamada.masahiro@socionext.com> wrote:
>>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>>
>> Let me clarify my concern.
>>
>> When we test the compiler flag, is there a case
>> where a particular flag depends on -m{32,64} ?
>>
>> For example, is there a compiler that supports -fstack-protector
>> for 64bit mode, but unsupports it for 32bit mode?
>>
>>   $(cc-option -m32)                     ->  y
>>   $(cc-option -m64)                     ->  y
>>   $(cc-option -fstack-protector)        ->  y
>>   $(cc-option -m32 -fstack-protector)   ->  n
>>   $(cc-option -m64 -fstack-protector)   ->  y
>>
>> I guess this is unlikely to happen,
>> but I am not whether it is zero possibility.
>>
>> If this could happen,
>> $(cc-option ) must be evaluated together with
>> correct bi-arch option (either -m32 or -m64).
>>
>>
>> Currently, -m32/-m64 is specified in Makefile,
>> but we are moving compiler tests to Kconfig
>> and, CONFIG_64BIT can be dynamically toggled in Kconfig.
>
> I don't think it can happen for this particular combination (stack protector
> and word size), but I'm sure we'll eventually run into options that
> need to be tested in combination. For the current CFLAGS_KERNEL
> setting, we definitely have the case of needing the variables to be
> evaluated in a specific order.
>




I was thinking of how we can handle complex cases
in the current approach.



(Case 1)

Compiler flag -foo and -bar interacts, so
we also need to check the combination of the two.


config CC_HAS_FOO
        def_bool $(cc-option -foo)

config CC_HAS_BAR
        def_bool $(cc-option -bar)

config CC_HAS_FOO_WITH_BAR
        def_bool $(cc-option -foo -bar)



(Case 2)
Compiler flag -foo is sensitive to word-size.
So, we need to test this option together with -m32/-m64.
User can toggle CONFIG_64BIT, like i386/x86_64.


config CC_NEEDS_M64
          def_bool $(cc-option -m64) && 64BIT

config CC_NEEDS_M32
          def_bool $(cc-option -m32) && !64BIT

config CC_HAS_FOO
         bool
         default $(cc-option -m64 -foo) if CC_NEEDS_M64
         default $(cc-option -m32 -foo) if CC_NEEDS_M32
         default $(cc-option -foo)



(Case 3)
Compiler flag -foo is sensitive to endian-ness.


config CC_NEEDS_BIG_ENDIAN
          def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN

config CC_NEEDS_LITTLE_ENDIAN
          def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN

config CC_HAS_FOO
         bool
         default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
         default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
         default $(cc-option -foo)




Hmm, I think I can implement those somehow.
But, I hope we do not have many instances like this...


If you know more naive cases, please share your knowledge.

Thanks!


-- 
Best Regards
Masahiro Yamada

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Arnd Bergmann]

On Wed, Feb 21, 2018 at 1:57 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>> On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
>> <yamada.masahiro@socionext.com> wrote:
>>> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>>>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
>>>> <yamada.masahiro@socionext.com> wrote:
>>>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>
> Hmm, I think I can implement those somehow.
> But, I hope we do not have many instances like this...
>
>
> If you know more naive cases, please share your knowledge.
>

One case that comes to mind would be architecture level selection on 32-bit
ARM, which is roughly this (I probably have some details wrong, but you
get the idea):

- older compilers don't support the latest architecture setting (-march=armv8
  or -march=armv7ve)
- newer compilers no longer support really old architectures (-march=armv4)
- setting -mthumb requires setting one of  -march=armv7-a, armv7ve, armv7-m or
  armv8 if the compiler doesn't default to those
- on a compiler that defaults to -marm, setting -march=armv7-m requires
  setting -mthumb (IIRC)
- really old compilers only support OABI, but not EABI
- newer compilers no longer support OABI
- mthumb requires EABI
- armv6 and higher are subtly broken with OABI, but only when using
  certain inline assembly with 64-bit arguments in register pairs.

I think we just shouldn't try to capture all of the above correctly in Kconfig
conditionals.

       Arnd

Re: [PATCH 11/23] kconfig: add 'shell-stdout' function

[Ulf Magnusson]

On Wed, Feb 21, 2018 at 01:59:59PM +0900, Masahiro Yamada wrote:
> 2018-02-20 3:01 GMT+09:00 Linus Torvalds <torvalds@linux-foundation.org>:
> > On Mon, Feb 19, 2018 at 9:44 AM, Linus Torvalds
> > <torvalds@linux-foundation.org> wrote:
> >>
> >> I do like your "success"/"stdout" more than "shell"/"shell-stdout",
> >> because with that naming I don't get the feeling that one should
> >> subsume the other.
> >
> > Hmm. Thinking about it some more, I really would prefer just "$(shell
> > ...)" everywhere.
> >
> > But it would be nice if perhaps the error handling would match the
> > context somehow.
> >
> > I'm wondering if this might tie into the whole quoting discussion in
> > the other thread.
> >
> > Because the rule could be:
> >
> > (a) unquoted $(shell ) is a bool, and failing is ok (and turns into
> > y/n depending on whether successful or failing)
> >
> > So
> >
> >   config CC_IS_GCC
> >           bool
> >           default $(shell $CC --version | grep -q gcc)
> >
> > works automatically.
> >
> > (b) but with quoting, $(shell ) is a string, and failing is an error
> >
> > So
> >
> >   config GCC_VERSION
> >           int
> >           default "$(shell-stdout $srctree/scripts/gcc-version.sh $CC
> > | sed 's/^0*//')" if CC_IS_GCC
> >           default 0
> >
> > would need those quotes, and if the shell-script returns a failure,
> > we'd _abort_.
> 
> 
> GCC_VERSION is int type.
> 
> Setting aside the Kconfig internal, I prefer 50700 to "50700"
> 
> According to my common sense, I do not want to quote integers.

Yeah, definitely looks a bit weird coming from other languages. "" just
means constant (a literal value, implemented as a constant symbol, as
opposed to a symbol reference).


If I were to redesign things, something like this would be closer to
what people intuitively expect:

  - n, m, y produces three predefined tristate symbols (they already do,
    through the n/m/y -> "n"/"m"/"y" shorthand)

  - "foo" produces a symbol of type string. Get rid of "n", "m", "y"
    (would require cleanup in Kconfig files).

  - 123 produces a symbol of type int

  - 0x123 produces a symbol of type hex

  - Everything else is a symbol reference

A nice side effect would be making all undefined symbol references just
that. I think that might come in handy. Some simple type checking could
be added to expressions as well.

You could also get rid of the obscure "undefined symbols produce their
name as their string value" magic at that point if you wanted to. That's
what makes 123 work currently.

One thing that gets a bit icky is that Kconfig currently stores constant
symbols in the symbol table, meaning "123" as a string might clash with
123 as an int, if you use the string value as the key (this would be a
problem if you want to give them the proper type). I wonder if there's
even any point to storing constant symbols in the symbol table though,
as opposed to just having them appear in expressions.

> IMO, I prefer to use different names for different purpose.
> So, 'stdout' and 'success' look good to me.
> 
> 
> 
> BTW, I noticed just one built-in function is enough
> because 'success' can be derived from 'stdout'.
> 
> 
> So, my plan is, implement $(shell ...) as a built-in function.
> This returns the stdout from the command.
> 
> 
> Then, implement 'success' as a textual shorthand
> by using macro, like this:
> 
> macro success $(shell ($(1) && echo y) || echo n)
> 
> 
> macro can be expanded recursively, so cc-option
> can be implemented based on 'success' macro.

Was worried about the error handling for a second there, but this looks
like it might work out pretty nicely. $(success) would never have non-0
exit status, so you could still warn for that in $(shell).

There's also the question of shared Windows/Linux support in other
projects that use Kconfig, though my feeling is kernel devs don't care.
Might be pretty easy to work around anyway, by source'ing a different
Kconfig file with macro definitions depending on the OS. Would need that
for more complicated stuff anyway.

Cheers,
Ulf

Re: [PATCH 11/23] kconfig: add 'shell-stdout' function

[Linus Torvalds]

On Tue, Feb 20, 2018 at 8:59 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
>
> IMO, I prefer to use different names for different purpose.
> So, 'stdout' and 'success' look good to me.
>
> BTW, I noticed just one built-in function is enough
> because 'success' can be derived from 'stdout'.
>
> So, my plan is, implement $(shell ...) as a built-in function.
> This returns the stdout from the command.
>
> Then, implement 'success' as a textual shorthand
> by using macro, like this:
>
> macro success $(shell ($(1) && echo y) || echo n)

I like it. This is nice and clean, and seems to be very generic. I see
no issues with this approach.

                Linus

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Ulf Magnusson]

On Wed, Feb 21, 2018 at 09:57:03PM +0900, Masahiro Yamada wrote:
> 2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> > On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
> > <yamada.masahiro@socionext.com> wrote:
> >> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> >>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
> >>> <yamada.masahiro@socionext.com> wrote:
> >>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> >>
> >> Let me clarify my concern.
> >>
> >> When we test the compiler flag, is there a case
> >> where a particular flag depends on -m{32,64} ?
> >>
> >> For example, is there a compiler that supports -fstack-protector
> >> for 64bit mode, but unsupports it for 32bit mode?
> >>
> >>   $(cc-option -m32)                     ->  y
> >>   $(cc-option -m64)                     ->  y
> >>   $(cc-option -fstack-protector)        ->  y
> >>   $(cc-option -m32 -fstack-protector)   ->  n
> >>   $(cc-option -m64 -fstack-protector)   ->  y
> >>
> >> I guess this is unlikely to happen,
> >> but I am not whether it is zero possibility.
> >>
> >> If this could happen,
> >> $(cc-option ) must be evaluated together with
> >> correct bi-arch option (either -m32 or -m64).
> >>
> >>
> >> Currently, -m32/-m64 is specified in Makefile,
> >> but we are moving compiler tests to Kconfig
> >> and, CONFIG_64BIT can be dynamically toggled in Kconfig.
> >
> > I don't think it can happen for this particular combination (stack protector
> > and word size), but I'm sure we'll eventually run into options that
> > need to be tested in combination. For the current CFLAGS_KERNEL
> > setting, we definitely have the case of needing the variables to be
> > evaluated in a specific order.
> >
> 
> 
> 
> 
> I was thinking of how we can handle complex cases
> in the current approach.
> 
> 
> 
> (Case 1)
> 
> Compiler flag -foo and -bar interacts, so
> we also need to check the combination of the two.
> 
> 
> config CC_HAS_FOO
>         def_bool $(cc-option -foo)
> 
> config CC_HAS_BAR
>         def_bool $(cc-option -bar)
> 
> config CC_HAS_FOO_WITH_BAR
>         def_bool $(cc-option -foo -bar)
> 
> 
> 
> (Case 2)
> Compiler flag -foo is sensitive to word-size.
> So, we need to test this option together with -m32/-m64.
> User can toggle CONFIG_64BIT, like i386/x86_64.
> 
> 
> config CC_NEEDS_M64
>           def_bool $(cc-option -m64) && 64BIT
> 
> config CC_NEEDS_M32
>           def_bool $(cc-option -m32) && !64BIT
> 
> config CC_HAS_FOO
>          bool
>          default $(cc-option -m64 -foo) if CC_NEEDS_M64
>          default $(cc-option -m32 -foo) if CC_NEEDS_M32
>          default $(cc-option -foo)
> 
> 
> 
> (Case 3)
> Compiler flag -foo is sensitive to endian-ness.
> 
> 
> config CC_NEEDS_BIG_ENDIAN
>           def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN
> 
> config CC_NEEDS_LITTLE_ENDIAN
>           def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN
> 
> config CC_HAS_FOO
>          bool
>          default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
>          default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
>          default $(cc-option -foo)
> 
> 
> 
> 
> Hmm, I think I can implement those somehow.
> But, I hope we do not have many instances like this...
> 
> 
> If you know more naive cases, please share your knowledge.
> 
> Thanks!
> 
> 
> -- 
> Best Regards
> Masahiro Yamada

Would get pretty bad if a test needs to consider multiple symbols.
Exponential explosion there...


I thought some more about the implementation of dynamic (post-parsing)
functions to see how bad it would get with the current implementation.

Some background on how things work now:

  1. All expression operands in Kconfig are symbols.

  2. Returning '$ENV' or '$(fn foo)' as a T_WORD during parsing gets
     you symbols with those strings as names and S_UNKNOWN type (because
     they act like references to undefined symbols).

  3. For "foo-$(fn foo)", you also get a symbol with that string as its
     name and S_UNKNOWN type (stored among the SYMBOL_CONST symbols)

  4. Symbols with S_UNKNOWN type get their name as their string value,
     and the tristate value n.

So, if you do string expansion on the names of symbols with S_UNKNOWN
type in sym_calc_value(), you're almost there with the current
implementation, except for the tristate case.

Maybe you could set the tristate value of S_UNKNOWN symbols depending on
the string value you end up with. Things are getting pretty confusing at
that point.

Could have something like S_DYNAMIC as well. More Kconfig complexity...

Then there's other complications:

  1. SYMBOL_CONST is no longer constant.

  2. Dependency loop detection needs to consider symbol references
     within strings.

  3. Dependency loop detection relies on static knowledge of what
     symbols a symbol depends on. That might get messy for certain
     expansions, though it might be things you wouldn't do in practice.

  4. Symbols still need to be properly invalidated. It looks like at
     least menuconfig just does a dumb invalidate-everything whenever
     the value of a symbol is changed though, so it might not require
     extra work. (Bit messier in Kconfiglib, which does minimal
     invalidation to keep scripts fast, but just need to extract a few
     extra deps there.)


It looks like dynamic functions could get quite messy, but might be
doable if absolutely required. There's probably more devils in the
details though.

I don't think the static function model precludes switching models later
btw, when people have more experience.

Cheers,
Ulf

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Michael Ellerman]

Masahiro Yamada <yamada.masahiro@socionext.com> writes:
>
<snip>
>
> (Case 3)
> Compiler flag -foo is sensitive to endian-ness.
>
>
> config CC_NEEDS_BIG_ENDIAN
>           def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN
>
> config CC_NEEDS_LITTLE_ENDIAN
>           def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN
>
> config CC_HAS_FOO
>          bool
>          default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
>          default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
>          default $(cc-option -foo)

We may do something like this on powerpc, where we have 32/64-bit and
big/little endian (on 64-bit) and then some ABI options that we
set/unset depending on endian.

The above looks like it could work though.

cheers

Re: [PATCH 21/23] gcc-plugins: move GCC version check for PowerPC to Kconfig

[Andrew Donnellan]

On 17/02/18 05:38, Masahiro Yamada wrote:
> For PowerPC, GCC 5.2 is the requirement for GCC plugins.  Move the
> version check to Kconfig, and remove the ugly checker.
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

The old checker was as non-ugly as I could make it. :)

Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>

-- 
Andrew Donnellan              OzLabs, ADL Canberra
andrew.donnellan@au1.ibm.com  IBM Australia Limited

Re: [PATCH 20/23] gcc-plugins: always build plugins with C++

[Emese Revfy]

On Sat, 17 Feb 2018 03:38:48 +0900
Masahiro Yamada <yamada.masahiro@socionext.com> wrote:

> If the target compiler is GCC 4.8 or newer, plugins are compiled with
> HOSTCXX.  Otherwise, gcc-plugin.sh will select HOSTCC or HOSTCXX.
> 
> To simpily things, let's decide GCC 4.8 is the requirement for GCC
> pulgins.  With this, plugins are always built with HOSTCXX.  This is a
> feature of advanced users, so this requirement whould not be not a big
> issue.

I disagree with such a decision. I think the gcc plugins should compile
with older version because the kernel can be compiled with them too.
The distros use these older versions as well and I can very well imagine
that new plugins will extend these with backported features such as
my sancov plugin.

-- 
Emese

Re: [PATCH 20/23] gcc-plugins: always build plugins with C++

[Masahiro Yamada]

2018-02-23 3:45 GMT+09:00 Emese Revfy <re.emese@gmail.com>:
> On Sat, 17 Feb 2018 03:38:48 +0900
> Masahiro Yamada <yamada.masahiro@socionext.com> wrote:
>
>> If the target compiler is GCC 4.8 or newer, plugins are compiled with
>> HOSTCXX.  Otherwise, gcc-plugin.sh will select HOSTCC or HOSTCXX.
>>
>> To simpily things, let's decide GCC 4.8 is the requirement for GCC
>> pulgins.  With this, plugins are always built with HOSTCXX.  This is a
>> feature of advanced users, so this requirement whould not be not a big
>> issue.
>
> I disagree with such a decision. I think the gcc plugins should compile
> with older version because the kernel can be compiled with them too.
> The distros use these older versions as well and I can very well imagine
> that new plugins will extend these with backported features such as
> my sancov plugin.
>

Let me clarify this.

I removed the version check in the following patch:
https://patchwork.kernel.org/patch/10225379/

So, you can enable CONFIG_GCC_PLAUGINS
as long as scripts/gcc-plugin.sh passes
(i.e. C++ can compile plugins).

Even if it is unlikely happen,
GCC 4.5 -> 4.8 is not a big jump.

People were recently talking about the minimum compiler version
https://patchwork.kernel.org/patch/10207385/

The thread is super long, but my understanding was
4.5 or 4.6 would be reasonable.

So, we will be there sooner or later.


-- 
Best Regards
Masahiro Yamada

Re: [PATCH 03/23] kconfig: add xstrdup() helper

[Masahiro Yamada]

2018-02-17 3:38 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:
> We already have xmalloc(), xcalloc(), and xrealloc(().  Add xstrdup()
> as well to save tedious error handling.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---


This is a trivial patch,
so I will pick it up now.




-- 
Best Regards
Masahiro Yamada

Re: [PATCH 04/23] kconfig: set SYMBOL_AUTO to the symbol marked with defconfig_list

[Masahiro Yamada]

2018-02-17 3:38 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:
> The 'defconfig_list' is a weird attribute.  If the '.config' is
> missing, conf_read_simple() iterates over all visible defaults,
> then it uses the first one for which fopen() succeeds.
>
> config DEFCONFIG_LIST
>         string
>         depends on !UML
>         option defconfig_list
>         default "/lib/modules/$UNAME_RELEASE/.config"
>         default "/etc/kernel-config"
>         default "/boot/config-$UNAME_RELEASE"
>         default "$ARCH_DEFCONFIG"
>         default "arch/$ARCH/defconfig"
>
> However, like other symbols, the first visible default is always
> written out to the .config file.  This might be different from what
> has been actually used.
>
> For example, on my machine, the third one "/boot/config-$UNAME_RELEASE"
> is opened, like follows:
>
>   $ rm .config
>   $ make oldconfig 2>/dev/null
>   scripts/kconfig/conf  --oldconfig Kconfig
>   #
>   # using defaults found in /boot/config-4.4.0-112-generic
>   #
>   *
>   * Restart config...
>   *
>   *
>   * IRQ subsystem
>   *
>   Expose irq internals in debugfs (GENERIC_IRQ_DEBUGFS) [N/y/?] (NEW)
>
> However, the resulted .config file contains the first one since it is
> visible:
>
>   $ grep CONFIG_DEFCONFIG_LIST .config
>   CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
>
> In order to stop confusing people, prevent this CONFIG option from
> being written to the .config file.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---


This one is almost trivial, so I will pick it up now.

Ulf, do you have any comment about this one?




> I'd like to fix the root case of this weirdness later.
> (and other 'option' attributes as well)
>
> But, this series is focusing a more important work in a bigger picture.
>
> For now, I decided to just hide CONFIG_DEFCONFIG_LIST
> from the .config file.
>
>
>  scripts/kconfig/menu.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
> index 9922285..36cd3e1 100644
> --- a/scripts/kconfig/menu.c
> +++ b/scripts/kconfig/menu.c
> @@ -212,6 +212,7 @@ void menu_add_option(int token, char *arg)
>                         sym_defconfig_list = current_entry->sym;
>                 else if (sym_defconfig_list != current_entry->sym)
>                         zconf_error("trying to redefine defconfig symbol");
> +               sym_defconfig_list->flags |= SYMBOL_AUTO;
>                 break;
>         case T_OPT_ENV:
>                 prop_add_env(arg);
> --
> 2.7.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

Re: [PATCH 04/23] kconfig: set SYMBOL_AUTO to the symbol marked with defconfig_list

[Ulf Magnusson]

On Thu, Mar 1, 2018 at 4:05 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2018-02-17 3:38 GMT+09:00 Masahiro Yamada <yamada.masahiro@socionext.com>:
>> The 'defconfig_list' is a weird attribute.  If the '.config' is
>> missing, conf_read_simple() iterates over all visible defaults,
>> then it uses the first one for which fopen() succeeds.
>>
>> config DEFCONFIG_LIST
>>         string
>>         depends on !UML
>>         option defconfig_list
>>         default "/lib/modules/$UNAME_RELEASE/.config"
>>         default "/etc/kernel-config"
>>         default "/boot/config-$UNAME_RELEASE"
>>         default "$ARCH_DEFCONFIG"
>>         default "arch/$ARCH/defconfig"
>>
>> However, like other symbols, the first visible default is always
>> written out to the .config file.  This might be different from what
>> has been actually used.
>>
>> For example, on my machine, the third one "/boot/config-$UNAME_RELEASE"
>> is opened, like follows:
>>
>>   $ rm .config
>>   $ make oldconfig 2>/dev/null
>>   scripts/kconfig/conf  --oldconfig Kconfig
>>   #
>>   # using defaults found in /boot/config-4.4.0-112-generic
>>   #
>>   *
>>   * Restart config...
>>   *
>>   *
>>   * IRQ subsystem
>>   *
>>   Expose irq internals in debugfs (GENERIC_IRQ_DEBUGFS) [N/y/?] (NEW)
>>
>> However, the resulted .config file contains the first one since it is
>> visible:
>>
>>   $ grep CONFIG_DEFCONFIG_LIST .config
>>   CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
>>
>> In order to stop confusing people, prevent this CONFIG option from
>> being written to the .config file.
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>
>
> This one is almost trivial, so I will pick it up now.
>
> Ulf, do you have any comment about this one?
>
>
>
>
>> I'd like to fix the root case of this weirdness later.
>> (and other 'option' attributes as well)
>>
>> But, this series is focusing a more important work in a bigger picture.
>>
>> For now, I decided to just hide CONFIG_DEFCONFIG_LIST
>> from the .config file.
>>
>>
>>  scripts/kconfig/menu.c | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
>> index 9922285..36cd3e1 100644
>> --- a/scripts/kconfig/menu.c
>> +++ b/scripts/kconfig/menu.c
>> @@ -212,6 +212,7 @@ void menu_add_option(int token, char *arg)
>>                         sym_defconfig_list = current_entry->sym;
>>                 else if (sym_defconfig_list != current_entry->sym)
>>                         zconf_error("trying to redefine defconfig symbol");
>> +               sym_defconfig_list->flags |= SYMBOL_AUTO;
>>                 break;
>>         case T_OPT_ENV:
>>                 prop_add_env(arg);
>> --
>> 2.7.4
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Best Regards
> Masahiro Yamada

Seems reasonable to me.

Reviewed-by: Ulf Magnusson <ulfalizer@gmail.com>

Cheers,
Ulf

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

2018-02-22 6:39 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> On Wed, Feb 21, 2018 at 09:57:03PM +0900, Masahiro Yamada wrote:
>> 2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>> > On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
>> > <yamada.masahiro@socionext.com> wrote:
>> >> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>> >>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
>> >>> <yamada.masahiro@socionext.com> wrote:
>> >>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>> >>
>> >> Let me clarify my concern.
>> >>
>> >> When we test the compiler flag, is there a case
>> >> where a particular flag depends on -m{32,64} ?
>> >>
>> >> For example, is there a compiler that supports -fstack-protector
>> >> for 64bit mode, but unsupports it for 32bit mode?
>> >>
>> >>   $(cc-option -m32)                     ->  y
>> >>   $(cc-option -m64)                     ->  y
>> >>   $(cc-option -fstack-protector)        ->  y
>> >>   $(cc-option -m32 -fstack-protector)   ->  n
>> >>   $(cc-option -m64 -fstack-protector)   ->  y
>> >>
>> >> I guess this is unlikely to happen,
>> >> but I am not whether it is zero possibility.
>> >>
>> >> If this could happen,
>> >> $(cc-option ) must be evaluated together with
>> >> correct bi-arch option (either -m32 or -m64).
>> >>
>> >>
>> >> Currently, -m32/-m64 is specified in Makefile,
>> >> but we are moving compiler tests to Kconfig
>> >> and, CONFIG_64BIT can be dynamically toggled in Kconfig.
>> >
>> > I don't think it can happen for this particular combination (stack protector
>> > and word size), but I'm sure we'll eventually run into options that
>> > need to be tested in combination. For the current CFLAGS_KERNEL
>> > setting, we definitely have the case of needing the variables to be
>> > evaluated in a specific order.
>> >
>>
>>
>>
>>
>> I was thinking of how we can handle complex cases
>> in the current approach.
>>
>>
>>
>> (Case 1)
>>
>> Compiler flag -foo and -bar interacts, so
>> we also need to check the combination of the two.
>>
>>
>> config CC_HAS_FOO
>>         def_bool $(cc-option -foo)
>>
>> config CC_HAS_BAR
>>         def_bool $(cc-option -bar)
>>
>> config CC_HAS_FOO_WITH_BAR
>>         def_bool $(cc-option -foo -bar)
>>
>>
>>
>> (Case 2)
>> Compiler flag -foo is sensitive to word-size.
>> So, we need to test this option together with -m32/-m64.
>> User can toggle CONFIG_64BIT, like i386/x86_64.
>>
>>
>> config CC_NEEDS_M64
>>           def_bool $(cc-option -m64) && 64BIT
>>
>> config CC_NEEDS_M32
>>           def_bool $(cc-option -m32) && !64BIT
>>
>> config CC_HAS_FOO
>>          bool
>>          default $(cc-option -m64 -foo) if CC_NEEDS_M64
>>          default $(cc-option -m32 -foo) if CC_NEEDS_M32
>>          default $(cc-option -foo)
>>
>>
>>
>> (Case 3)
>> Compiler flag -foo is sensitive to endian-ness.
>>
>>
>> config CC_NEEDS_BIG_ENDIAN
>>           def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN
>>
>> config CC_NEEDS_LITTLE_ENDIAN
>>           def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN
>>
>> config CC_HAS_FOO
>>          bool
>>          default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
>>          default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
>>          default $(cc-option -foo)
>>
>>
>>
>>
>> Hmm, I think I can implement those somehow.
>> But, I hope we do not have many instances like this...
>>
>>
>> If you know more naive cases, please share your knowledge.
>>
>> Thanks!
>>
>>
>> --
>> Best Regards
>> Masahiro Yamada
>
> Would get pretty bad if a test needs to consider multiple symbols.
> Exponential explosion there...
>
>
> I thought some more about the implementation of dynamic (post-parsing)
> functions to see how bad it would get with the current implementation.
>
> Some background on how things work now:
>
>   1. All expression operands in Kconfig are symbols.
>
>   2. Returning '$ENV' or '$(fn foo)' as a T_WORD during parsing gets
>      you symbols with those strings as names and S_UNKNOWN type (because
>      they act like references to undefined symbols).
>
>   3. For "foo-$(fn foo)", you also get a symbol with that string as its
>      name and S_UNKNOWN type (stored among the SYMBOL_CONST symbols)
>
>   4. Symbols with S_UNKNOWN type get their name as their string value,
>      and the tristate value n.
>
> So, if you do string expansion on the names of symbols with S_UNKNOWN
> type in sym_calc_value(), you're almost there with the current
> implementation, except for the tristate case.
>
> Maybe you could set the tristate value of S_UNKNOWN symbols depending on
> the string value you end up with. Things are getting pretty confusing at
> that point.
>
> Could have something like S_DYNAMIC as well. More Kconfig complexity...
>
> Then there's other complications:
>
>   1. SYMBOL_CONST is no longer constant.
>
>   2. Dependency loop detection needs to consider symbol references
>      within strings.
>
>   3. Dependency loop detection relies on static knowledge of what
>      symbols a symbol depends on. That might get messy for certain
>      expansions, though it might be things you wouldn't do in practice.
>
>   4. Symbols still need to be properly invalidated. It looks like at
>      least menuconfig just does a dumb invalidate-everything whenever
>      the value of a symbol is changed though, so it might not require
>      extra work. (Bit messier in Kconfiglib, which does minimal
>      invalidation to keep scripts fast, but just need to extract a few
>      extra deps there.)
>
>
> It looks like dynamic functions could get quite messy, but might be
> doable if absolutely required. There's probably more devils in the
> details though.
>
> I don't think the static function model precludes switching models later
> btw, when people have more experience.



I really want to start with the static function model
and see if we need the dynamic function implementation.

Here is an idea for the migration path in case
we need to do that in the future.



Currently, every time user input is given,
sym_clear_all_valid() is called.

It is not efficient to blindly re-evaluate expensive $(shell ...)


So, have a list of symbols the function depends on
in its arguments.

For example,

config CC_HAS_SANE_STACKPROTECTOR
        def_bool $(shell $srctree/scripts/gcc-has-stack-protector.sh
$CC $(1), CFLAGS_BASE)


Here the first argument
  $srctree/scripts/gcc-x86-has-stack-protector.sh $CC $(1)

is the shell command.
$(1), $(2), ... will be replaced with the values of symbols (or expressions)
that follow when running the shell command.


The second argument
CFLAGS_BASE
is the dependency symbol (or expression).


CFLAGS_BASE can be dynamically changed like

config CFLAGS_BASE
        string
        default "-m64" if 64BIT
        default "-m32"


When and only when CFLAGS_BASE is updated, the function should be re-calculated.
(This will require efforts to minimize the amount of re-evaluation.)




cc-option will be implemented like follows:

macro cc-option $(shell $CC -Werror $$(1) $(1) -c -x c /dev/null -o
/dev/null, CFLAGS_BASE)



Please notice the difference between $$(1) and $(1).

$(1) is immediately expanded by cc-option macro.

$$(1) is escaped since we want to expand it by $(shell ...), not by
$(cc-option ...)



For example,

  $(cc-option -fstack-protector)

will be expanded to

  $(shell gcc -Werror $(1) -fstack-protector -c -x c /dev/null -o
/dev/null, CFLAGS_BASE)

Since macros are just textual shorthand, so this expansion happens
during the parse phase.



Then, the evaluation phase does the following every time CFLAGS_BASE is updated.

gcc -Werror [value of CFLAGS_BASE] -fstack-protector -c -x c /dev/null
-o /dev/null


This is a new form of expression, so it will be managed in AST tree
with a flag E_SHELL (or E_FUNC) etc.


Not implemented at all.  Just a rough sketch.


-- 
Best Regards
Masahiro Yamada

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Ulf Magnusson]

On Fri, Mar 02, 2018 at 02:50:39PM +0900, Masahiro Yamada wrote:
> 2018-02-22 6:39 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> > On Wed, Feb 21, 2018 at 09:57:03PM +0900, Masahiro Yamada wrote:
> >> 2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> >> > On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
> >> > <yamada.masahiro@socionext.com> wrote:
> >> >> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> >> >>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
> >> >>> <yamada.masahiro@socionext.com> wrote:
> >> >>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> >> >>
> >> >> Let me clarify my concern.
> >> >>
> >> >> When we test the compiler flag, is there a case
> >> >> where a particular flag depends on -m{32,64} ?
> >> >>
> >> >> For example, is there a compiler that supports -fstack-protector
> >> >> for 64bit mode, but unsupports it for 32bit mode?
> >> >>
> >> >>   $(cc-option -m32)                     ->  y
> >> >>   $(cc-option -m64)                     ->  y
> >> >>   $(cc-option -fstack-protector)        ->  y
> >> >>   $(cc-option -m32 -fstack-protector)   ->  n
> >> >>   $(cc-option -m64 -fstack-protector)   ->  y
> >> >>
> >> >> I guess this is unlikely to happen,
> >> >> but I am not whether it is zero possibility.
> >> >>
> >> >> If this could happen,
> >> >> $(cc-option ) must be evaluated together with
> >> >> correct bi-arch option (either -m32 or -m64).
> >> >>
> >> >>
> >> >> Currently, -m32/-m64 is specified in Makefile,
> >> >> but we are moving compiler tests to Kconfig
> >> >> and, CONFIG_64BIT can be dynamically toggled in Kconfig.
> >> >
> >> > I don't think it can happen for this particular combination (stack protector
> >> > and word size), but I'm sure we'll eventually run into options that
> >> > need to be tested in combination. For the current CFLAGS_KERNEL
> >> > setting, we definitely have the case of needing the variables to be
> >> > evaluated in a specific order.
> >> >
> >>
> >>
> >>
> >>
> >> I was thinking of how we can handle complex cases
> >> in the current approach.
> >>
> >>
> >>
> >> (Case 1)
> >>
> >> Compiler flag -foo and -bar interacts, so
> >> we also need to check the combination of the two.
> >>
> >>
> >> config CC_HAS_FOO
> >>         def_bool $(cc-option -foo)
> >>
> >> config CC_HAS_BAR
> >>         def_bool $(cc-option -bar)
> >>
> >> config CC_HAS_FOO_WITH_BAR
> >>         def_bool $(cc-option -foo -bar)
> >>
> >>
> >>
> >> (Case 2)
> >> Compiler flag -foo is sensitive to word-size.
> >> So, we need to test this option together with -m32/-m64.
> >> User can toggle CONFIG_64BIT, like i386/x86_64.
> >>
> >>
> >> config CC_NEEDS_M64
> >>           def_bool $(cc-option -m64) && 64BIT
> >>
> >> config CC_NEEDS_M32
> >>           def_bool $(cc-option -m32) && !64BIT
> >>
> >> config CC_HAS_FOO
> >>          bool
> >>          default $(cc-option -m64 -foo) if CC_NEEDS_M64
> >>          default $(cc-option -m32 -foo) if CC_NEEDS_M32
> >>          default $(cc-option -foo)
> >>
> >>
> >>
> >> (Case 3)
> >> Compiler flag -foo is sensitive to endian-ness.
> >>
> >>
> >> config CC_NEEDS_BIG_ENDIAN
> >>           def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN
> >>
> >> config CC_NEEDS_LITTLE_ENDIAN
> >>           def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN
> >>
> >> config CC_HAS_FOO
> >>          bool
> >>          default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
> >>          default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
> >>          default $(cc-option -foo)
> >>
> >>
> >>
> >>
> >> Hmm, I think I can implement those somehow.
> >> But, I hope we do not have many instances like this...
> >>
> >>
> >> If you know more naive cases, please share your knowledge.
> >>
> >> Thanks!
> >>
> >>
> >> --
> >> Best Regards
> >> Masahiro Yamada
> >
> > Would get pretty bad if a test needs to consider multiple symbols.
> > Exponential explosion there...
> >
> >
> > I thought some more about the implementation of dynamic (post-parsing)
> > functions to see how bad it would get with the current implementation.
> >
> > Some background on how things work now:
> >
> >   1. All expression operands in Kconfig are symbols.
> >
> >   2. Returning '$ENV' or '$(fn foo)' as a T_WORD during parsing gets
> >      you symbols with those strings as names and S_UNKNOWN type (because
> >      they act like references to undefined symbols).
> >
> >   3. For "foo-$(fn foo)", you also get a symbol with that string as its
> >      name and S_UNKNOWN type (stored among the SYMBOL_CONST symbols)
> >
> >   4. Symbols with S_UNKNOWN type get their name as their string value,
> >      and the tristate value n.
> >
> > So, if you do string expansion on the names of symbols with S_UNKNOWN
> > type in sym_calc_value(), you're almost there with the current
> > implementation, except for the tristate case.
> >
> > Maybe you could set the tristate value of S_UNKNOWN symbols depending on
> > the string value you end up with. Things are getting pretty confusing at
> > that point.
> >
> > Could have something like S_DYNAMIC as well. More Kconfig complexity...
> >
> > Then there's other complications:
> >
> >   1. SYMBOL_CONST is no longer constant.
> >
> >   2. Dependency loop detection needs to consider symbol references
> >      within strings.
> >
> >   3. Dependency loop detection relies on static knowledge of what
> >      symbols a symbol depends on. That might get messy for certain
> >      expansions, though it might be things you wouldn't do in practice.
> >
> >   4. Symbols still need to be properly invalidated. It looks like at
> >      least menuconfig just does a dumb invalidate-everything whenever
> >      the value of a symbol is changed though, so it might not require
> >      extra work. (Bit messier in Kconfiglib, which does minimal
> >      invalidation to keep scripts fast, but just need to extract a few
> >      extra deps there.)
> >
> >
> > It looks like dynamic functions could get quite messy, but might be
> > doable if absolutely required. There's probably more devils in the
> > details though.
> >
> > I don't think the static function model precludes switching models later
> > btw, when people have more experience.
> 
> 
> 
> I really want to start with the static function model
> and see if we need the dynamic function implementation.

Yeah, let's start with static functions, IMO.

Either we'll learn that they're powerful enough in practice, and save
ourselves some work, or we'll gain experience for later. Converting from
static to dynamic functions should be painless, if needed.

My plan would be something like:

  1. Implement static functions

  2. Convert as many simple cases over to them as possible

  3. See how bad the bad cases get. If they get really bad, then decide
     what to do next (extend Kconfig, handle them in the Makefiles,
     etc.)

> 
> Here is an idea for the migration path in case
> we need to do that in the future.
> 
> 
> 
> Currently, every time user input is given,
> sym_clear_all_valid() is called.
> 
> It is not efficient to blindly re-evaluate expensive $(shell ...)

I think menuconfig only reevalutes the symbols in the menu that's
currently shown in the interface (along with their dependencies).

Maybe that'd be bad enough though.

> 
> 
> So, have a list of symbols the function depends on
> in its arguments.
> 
> For example,
> 
> config CC_HAS_SANE_STACKPROTECTOR
>         def_bool $(shell $srctree/scripts/gcc-has-stack-protector.sh
> $CC $(1), CFLAGS_BASE)
> 
> 
> Here the first argument
>   $srctree/scripts/gcc-x86-has-stack-protector.sh $CC $(1)
> 
> is the shell command.
> $(1), $(2), ... will be replaced with the values of symbols (or expressions)
> that follow when running the shell command.
> 
> 
> The second argument
> CFLAGS_BASE
> is the dependency symbol (or expression).
> 
> 
> CFLAGS_BASE can be dynamically changed like
> 
> config CFLAGS_BASE
>         string
>         default "-m64" if 64BIT
>         default "-m32"
> 
> 
> When and only when CFLAGS_BASE is updated, the function should be re-calculated.
> (This will require efforts to minimize the amount of re-evaluation.)
> 
> 
> 
> 
> cc-option will be implemented like follows:
> 
> macro cc-option $(shell $CC -Werror $$(1) $(1) -c -x c /dev/null -o
> /dev/null, CFLAGS_BASE)
> 
> 
> 
> Please notice the difference between $$(1) and $(1).
> 
> $(1) is immediately expanded by cc-option macro.
> 
> $$(1) is escaped since we want to expand it by $(shell ...), not by
> $(cc-option ...)
> 
> 
> 
> For example,
> 
>   $(cc-option -fstack-protector)
> 
> will be expanded to
> 
>   $(shell gcc -Werror $(1) -fstack-protector -c -x c /dev/null -o
> /dev/null, CFLAGS_BASE)
> 
> Since macros are just textual shorthand, so this expansion happens
> during the parse phase.
> 
> 
> 
> Then, the evaluation phase does the following every time CFLAGS_BASE is updated.
> 
> gcc -Werror [value of CFLAGS_BASE] -fstack-protector -c -x c /dev/null
> -o /dev/null
> 
> 
> This is a new form of expression, so it will be managed in AST tree
> with a flag E_SHELL (or E_FUNC) etc.
> 
> 
> Not implemented at all.  Just a rough sketch.

A simpler syntax like

	$(shell $CC -Werror {CFLAGS_BASE} -c -x c /dev/null -o /dev/null)

might work as well. Then you wouldn't have to any double escaping, and
having both $(1) and the value for it appear in the same place seems a
bit redundant.

I wonder if dependency management might get messy...

Anyway, let's just go with static functions...

Cheers,
Ulf

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Ulf Magnusson]

On Fri, Mar 02, 2018 at 10:03:26AM +0100, Ulf Magnusson wrote:
> On Fri, Mar 02, 2018 at 02:50:39PM +0900, Masahiro Yamada wrote:
> > 2018-02-22 6:39 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> > > On Wed, Feb 21, 2018 at 09:57:03PM +0900, Masahiro Yamada wrote:
> > >> 2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> > >> > On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
> > >> > <yamada.masahiro@socionext.com> wrote:
> > >> >> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> > >> >>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
> > >> >>> <yamada.masahiro@socionext.com> wrote:
> > >> >>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> > >> >>
> > >> >> Let me clarify my concern.
> > >> >>
> > >> >> When we test the compiler flag, is there a case
> > >> >> where a particular flag depends on -m{32,64} ?
> > >> >>
> > >> >> For example, is there a compiler that supports -fstack-protector
> > >> >> for 64bit mode, but unsupports it for 32bit mode?
> > >> >>
> > >> >>   $(cc-option -m32)                     ->  y
> > >> >>   $(cc-option -m64)                     ->  y
> > >> >>   $(cc-option -fstack-protector)        ->  y
> > >> >>   $(cc-option -m32 -fstack-protector)   ->  n
> > >> >>   $(cc-option -m64 -fstack-protector)   ->  y
> > >> >>
> > >> >> I guess this is unlikely to happen,
> > >> >> but I am not whether it is zero possibility.
> > >> >>
> > >> >> If this could happen,
> > >> >> $(cc-option ) must be evaluated together with
> > >> >> correct bi-arch option (either -m32 or -m64).
> > >> >>
> > >> >>
> > >> >> Currently, -m32/-m64 is specified in Makefile,
> > >> >> but we are moving compiler tests to Kconfig
> > >> >> and, CONFIG_64BIT can be dynamically toggled in Kconfig.
> > >> >
> > >> > I don't think it can happen for this particular combination (stack protector
> > >> > and word size), but I'm sure we'll eventually run into options that
> > >> > need to be tested in combination. For the current CFLAGS_KERNEL
> > >> > setting, we definitely have the case of needing the variables to be
> > >> > evaluated in a specific order.
> > >> >
> > >>
> > >>
> > >>
> > >>
> > >> I was thinking of how we can handle complex cases
> > >> in the current approach.
> > >>
> > >>
> > >>
> > >> (Case 1)
> > >>
> > >> Compiler flag -foo and -bar interacts, so
> > >> we also need to check the combination of the two.
> > >>
> > >>
> > >> config CC_HAS_FOO
> > >>         def_bool $(cc-option -foo)
> > >>
> > >> config CC_HAS_BAR
> > >>         def_bool $(cc-option -bar)
> > >>
> > >> config CC_HAS_FOO_WITH_BAR
> > >>         def_bool $(cc-option -foo -bar)
> > >>
> > >>
> > >>
> > >> (Case 2)
> > >> Compiler flag -foo is sensitive to word-size.
> > >> So, we need to test this option together with -m32/-m64.
> > >> User can toggle CONFIG_64BIT, like i386/x86_64.
> > >>
> > >>
> > >> config CC_NEEDS_M64
> > >>           def_bool $(cc-option -m64) && 64BIT
> > >>
> > >> config CC_NEEDS_M32
> > >>           def_bool $(cc-option -m32) && !64BIT
> > >>
> > >> config CC_HAS_FOO
> > >>          bool
> > >>          default $(cc-option -m64 -foo) if CC_NEEDS_M64
> > >>          default $(cc-option -m32 -foo) if CC_NEEDS_M32
> > >>          default $(cc-option -foo)
> > >>
> > >>
> > >>
> > >> (Case 3)
> > >> Compiler flag -foo is sensitive to endian-ness.
> > >>
> > >>
> > >> config CC_NEEDS_BIG_ENDIAN
> > >>           def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN
> > >>
> > >> config CC_NEEDS_LITTLE_ENDIAN
> > >>           def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN
> > >>
> > >> config CC_HAS_FOO
> > >>          bool
> > >>          default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
> > >>          default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
> > >>          default $(cc-option -foo)
> > >>
> > >>
> > >>
> > >>
> > >> Hmm, I think I can implement those somehow.
> > >> But, I hope we do not have many instances like this...
> > >>
> > >>
> > >> If you know more naive cases, please share your knowledge.
> > >>
> > >> Thanks!
> > >>
> > >>
> > >> --
> > >> Best Regards
> > >> Masahiro Yamada
> > >
> > > Would get pretty bad if a test needs to consider multiple symbols.
> > > Exponential explosion there...
> > >
> > >
> > > I thought some more about the implementation of dynamic (post-parsing)
> > > functions to see how bad it would get with the current implementation.
> > >
> > > Some background on how things work now:
> > >
> > >   1. All expression operands in Kconfig are symbols.
> > >
> > >   2. Returning '$ENV' or '$(fn foo)' as a T_WORD during parsing gets
> > >      you symbols with those strings as names and S_UNKNOWN type (because
> > >      they act like references to undefined symbols).
> > >
> > >   3. For "foo-$(fn foo)", you also get a symbol with that string as its
> > >      name and S_UNKNOWN type (stored among the SYMBOL_CONST symbols)
> > >
> > >   4. Symbols with S_UNKNOWN type get their name as their string value,
> > >      and the tristate value n.
> > >
> > > So, if you do string expansion on the names of symbols with S_UNKNOWN
> > > type in sym_calc_value(), you're almost there with the current
> > > implementation, except for the tristate case.
> > >
> > > Maybe you could set the tristate value of S_UNKNOWN symbols depending on
> > > the string value you end up with. Things are getting pretty confusing at
> > > that point.
> > >
> > > Could have something like S_DYNAMIC as well. More Kconfig complexity...
> > >
> > > Then there's other complications:
> > >
> > >   1. SYMBOL_CONST is no longer constant.
> > >
> > >   2. Dependency loop detection needs to consider symbol references
> > >      within strings.
> > >
> > >   3. Dependency loop detection relies on static knowledge of what
> > >      symbols a symbol depends on. That might get messy for certain
> > >      expansions, though it might be things you wouldn't do in practice.
> > >
> > >   4. Symbols still need to be properly invalidated. It looks like at
> > >      least menuconfig just does a dumb invalidate-everything whenever
> > >      the value of a symbol is changed though, so it might not require
> > >      extra work. (Bit messier in Kconfiglib, which does minimal
> > >      invalidation to keep scripts fast, but just need to extract a few
> > >      extra deps there.)
> > >
> > >
> > > It looks like dynamic functions could get quite messy, but might be
> > > doable if absolutely required. There's probably more devils in the
> > > details though.
> > >
> > > I don't think the static function model precludes switching models later
> > > btw, when people have more experience.
> > 
> > 
> > 
> > I really want to start with the static function model
> > and see if we need the dynamic function implementation.
> 
> Yeah, let's start with static functions, IMO.
> 
> Either we'll learn that they're powerful enough in practice, and save
> ourselves some work, or we'll gain experience for later. Converting from
> static to dynamic functions should be painless, if needed.
> 
> My plan would be something like:
> 
>   1. Implement static functions
> 
>   2. Convert as many simple cases over to them as possible
> 
>   3. See how bad the bad cases get. If they get really bad, then decide
>      what to do next (extend Kconfig, handle them in the Makefiles,
>      etc.)
> 
> > 
> > Here is an idea for the migration path in case
> > we need to do that in the future.
> > 
> > 
> > 
> > Currently, every time user input is given,
> > sym_clear_all_valid() is called.
> > 
> > It is not efficient to blindly re-evaluate expensive $(shell ...)
> 
> I think menuconfig only reevalutes the symbols in the menu that's
> currently shown in the interface (along with their dependencies).
> 
> Maybe that'd be bad enough though.
> 
> > 
> > 
> > So, have a list of symbols the function depends on
> > in its arguments.
> > 
> > For example,
> > 
> > config CC_HAS_SANE_STACKPROTECTOR
> >         def_bool $(shell $srctree/scripts/gcc-has-stack-protector.sh
> > $CC $(1), CFLAGS_BASE)
> > 
> > 
> > Here the first argument
> >   $srctree/scripts/gcc-x86-has-stack-protector.sh $CC $(1)
> > 
> > is the shell command.
> > $(1), $(2), ... will be replaced with the values of symbols (or expressions)
> > that follow when running the shell command.
> > 
> > 
> > The second argument
> > CFLAGS_BASE
> > is the dependency symbol (or expression).
> > 
> > 
> > CFLAGS_BASE can be dynamically changed like
> > 
> > config CFLAGS_BASE
> >         string
> >         default "-m64" if 64BIT
> >         default "-m32"
> > 
> > 
> > When and only when CFLAGS_BASE is updated, the function should be re-calculated.
> > (This will require efforts to minimize the amount of re-evaluation.)
> > 
> > 
> > 
> > 
> > cc-option will be implemented like follows:
> > 
> > macro cc-option $(shell $CC -Werror $$(1) $(1) -c -x c /dev/null -o
> > /dev/null, CFLAGS_BASE)
> > 
> > 
> > 
> > Please notice the difference between $$(1) and $(1).
> > 
> > $(1) is immediately expanded by cc-option macro.
> > 
> > $$(1) is escaped since we want to expand it by $(shell ...), not by
> > $(cc-option ...)
> > 
> > 
> > 
> > For example,
> > 
> >   $(cc-option -fstack-protector)
> > 
> > will be expanded to
> > 
> >   $(shell gcc -Werror $(1) -fstack-protector -c -x c /dev/null -o
> > /dev/null, CFLAGS_BASE)
> > 
> > Since macros are just textual shorthand, so this expansion happens
> > during the parse phase.
> > 
> > 
> > 
> > Then, the evaluation phase does the following every time CFLAGS_BASE is updated.
> > 
> > gcc -Werror [value of CFLAGS_BASE] -fstack-protector -c -x c /dev/null
> > -o /dev/null
> > 
> > 
> > This is a new form of expression, so it will be managed in AST tree
> > with a flag E_SHELL (or E_FUNC) etc.
> > 
> > 
> > Not implemented at all.  Just a rough sketch.
> 
> A simpler syntax like
> 
> 	$(shell $CC -Werror {CFLAGS_BASE} -c -x c /dev/null -o /dev/null)

	*$(shell $CC -Werror {CFLAGS_BASE} -fstack-protector -c -x c /dev/null -o /dev/null)