[Buildroot] [PATCH 01/16 v3] core: move pkg-utils.mk to support/

[Buildroot] [PATCH 01/16 v3] core: move pkg-utils.mk to support/

[Yann E. MORIN]

pkg-utils.mk contains various definitions that are used in the package
infrastructures and packages themselves.

However, those definitions can be useful in other parts of Buildroot,
and are already used in a few places that are not related to the package
infrastructure. Also, $(sep) will be needed early in the Makefile when
we eventually support multiple br2-external trees.

Since this file only contains definitions, we can include it anytime.

So, consider that file to no longer be specific to the package infras:
  - move it to support and rename it,
  - move a few similar definitions from the main Makefile to that file.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
---
 Makefile              |  10 +----
 package/pkg-utils.mk  |  96 -------------------------------------------
 support/misc/utils.mk | 111 ++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 113 insertions(+), 104 deletions(-)
 create mode 100644 support/misc/utils.mk

diff --git a/Makefile b/Makefile
index 027f21c..af2d982 100644
--- a/Makefile
+++ b/Makefile
@@ -106,14 +106,8 @@ else ifneq ($(filter-out $(nobuild_targets),$(MAKECMDGOALS)),)
 BR_BUILDING = y
 endif
 
-# Strip quotes and then whitespaces
-qstrip = $(strip $(subst ",,$(1)))
-#"))
-
-# Variables for use in Make constructs
-comma := ,
-empty :=
-space := $(empty) $(empty)
+# Include some helper macros and variables
+include support/misc/utils.mk
 
 ifneq ("$(origin O)", "command line")
 O := output
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index c61b3b6..28db481 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -5,43 +5,6 @@
 #
 ################################################################################
 
-# Case conversion macros. This is inspired by the 'up' macro from gmsl
-# (http://gmsl.sf.net). It is optimised very heavily because these macros
-# are used a lot. It is about 5 times faster than forking a shell and tr.
-#
-# The caseconvert-helper creates a definition of the case conversion macro.
-# After expansion by the outer $(eval ), the UPPERCASE macro is defined as:
-# $(strip $(eval __tmp := $(1))  $(eval __tmp := $(subst a,A,$(__tmp))) ... )
-# In other words, every letter is substituted one by one.
-#
-# The caseconvert-helper allows us to create this definition out of the
-# [FROM] and [TO] lists, so we don't need to write down every substition
-# manually. The uses of $ and $$ quoting are chosen in order to do as
-# much expansion as possible up-front.
-#
-# Note that it would be possible to conceive a slightly more optimal
-# implementation that avoids the use of __tmp, but that would be even
-# more unreadable and is not worth the effort.
-
-[FROM] := a b c d e f g h i j k l m n o p q r s t u v w x y z - .
-[TO]   := A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ _
-
-define caseconvert-helper
-$(1) = $$(strip \
-	$$(eval __tmp := $$(1))\
-	$(foreach c, $(2),\
-		$$(eval __tmp := $$(subst $(word 1,$(subst :, ,$c)),$(word 2,$(subst :, ,$c)),$$(__tmp))))\
-	$$(__tmp))
-endef
-
-$(eval $(call caseconvert-helper,UPPERCASE,$(join $(addsuffix :,$([FROM])),$([TO]))))
-$(eval $(call caseconvert-helper,LOWERCASE,$(join $(addsuffix :,$([TO])),$([FROM]))))
-
-# Sanitize macro cleans up generic strings so it can be used as a filename
-# and in rules. Particularly useful for VCS version strings, that can contain
-# slashes, colons (OK in filenames but not in rules), and spaces.
-sanitize = $(subst $(space),_,$(subst :,_,$(subst /,_,$(strip $(1)))))
-
 #
 # Manipulation of .config files based on the Kconfig
 # infrastructure. Used by the BusyBox package, the Linux kernel
@@ -82,65 +45,6 @@ INFLATE.tar  = cat
 # suitable-extractor(filename): returns extractor based on suffix
 suitable-extractor = $(INFLATE$(suffix $(1)))
 
-# MESSAGE Macro -- display a message in bold type
-MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG)_NAME) $($(PKG)_VERSION) $(call qstrip,$(1))$(TERM_RESET)"
-TERM_BOLD := $(shell tput smso 2>/dev/null)
-TERM_RESET := $(shell tput rmso 2>/dev/null)
-
-# Utility functions for 'find'
-# findfileclauses(filelist) => -name 'X' -o -name 'Y'
-findfileclauses = $(call notfirstword,$(patsubst %,-o -name '%',$(1)))
-# finddirclauses(base, dirlist) => -path 'base/dirX' -o -path 'base/dirY'
-finddirclauses = $(call notfirstword,$(patsubst %,-o -path '$(1)/%',$(2)))
-
-# Miscellaneous utility functions
-# notfirstword(wordlist): returns all but the first word in wordlist
-notfirstword = $(wordlist 2,$(words $(1)),$(1))
-
-# Needed for the foreach loops to loop over the list of hooks, so that
-# each hook call is properly separated by a newline.
-define sep
-
-
-endef
-
-PERCENT = %
-QUOTE = '
-# ' # Meh... syntax-highlighting
-
-# This macro properly escapes a command string, then prints it with printf:
-#
-#   - first, backslash '\' are self-escaped, so that they do not escape
-#     the following char and so that printf properly outputs a backslash;
-#
-#   - next, single quotes are escaped by closing an existing one, adding
-#     an escaped one, and re-openning a new one (see below for the reason);
-#
-#   - then '%' signs are self-escaped so that the printf does not interpret
-#     them as a format specifier, in case the variable contains an actual
-#     printf with a format;
-#
-#   - finally, $(sep) is replaced with the literal '\n' so that make does
-#     not break on the so-expanded variable, but so that the printf does
-#     correctly output an LF.
-#
-# Note: this must be escaped in this order to avoid over-escaping the
-# previously escaped elements.
-#
-# Once everything has been escaped, it is passed between single quotes
-# (that's why the single-quotes are escaped they way they are, above,
-# and why the dollar sign is not escaped) to printf(1). A trailing
-# newline is apended, too.
-#
-# Note: leading or trailing spaces are *not* stripped.
-#
-define PRINTF
-	printf '$(subst $(sep),\n,\
-			$(subst $(PERCENT),$(PERCENT)$(PERCENT),\
-				$(subst $(QUOTE),$(QUOTE)\$(QUOTE)$(QUOTE),\
-					$(subst \,\\,$(1)))))\n'
-endef
-
 # check-deprecated-variable -- throw an error on deprecated variables
 # example:
 #   $(eval $(call check-deprecated-variable,FOO_MAKE_OPT,FOO_MAKE_OPTS))
diff --git a/support/misc/utils.mk b/support/misc/utils.mk
new file mode 100644
index 0000000..990a3d1
--- /dev/null
+++ b/support/misc/utils.mk
@@ -0,0 +1,111 @@
+################################################################################
+#
+# This file contains various utility macros and variables used about
+# everywhere in make constructs.
+#
+################################################################################
+
+# Strip quotes and then whitespaces
+qstrip = $(strip $(subst ",,$(1)))
+#"))
+
+# Variables for use in Make constructs
+comma := ,
+empty :=
+space := $(empty) $(empty)
+
+# Case conversion macros. This is inspired by the 'up' macro from gmsl
+# (http://gmsl.sf.net). It is optimised very heavily because these macros
+# are used a lot. It is about 5 times faster than forking a shell and tr.
+#
+# The caseconvert-helper creates a definition of the case conversion macro.
+# After expansion by the outer $(eval ), the UPPERCASE macro is defined as:
+# $(strip $(eval __tmp := $(1))  $(eval __tmp := $(subst a,A,$(__tmp))) ... )
+# In other words, every letter is substituted one by one.
+#
+# The caseconvert-helper allows us to create this definition out of the
+# [FROM] and [TO] lists, so we don't need to write down every substition
+# manually. The uses of $ and $$ quoting are chosen in order to do as
+# much expansion as possible up-front.
+#
+# Note that it would be possible to conceive a slightly more optimal
+# implementation that avoids the use of __tmp, but that would be even
+# more unreadable and is not worth the effort.
+
+[FROM] := a b c d e f g h i j k l m n o p q r s t u v w x y z - .
+[TO]   := A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ _
+
+define caseconvert-helper
+$(1) = $$(strip \
+	$$(eval __tmp := $$(1))\
+	$(foreach c, $(2),\
+		$$(eval __tmp := $$(subst $(word 1,$(subst :, ,$c)),$(word 2,$(subst :, ,$c)),$$(__tmp))))\
+	$$(__tmp))
+endef
+
+$(eval $(call caseconvert-helper,UPPERCASE,$(join $(addsuffix :,$([FROM])),$([TO]))))
+$(eval $(call caseconvert-helper,LOWERCASE,$(join $(addsuffix :,$([TO])),$([FROM]))))
+
+# Sanitize macro cleans up generic strings so it can be used as a filename
+# and in rules. Particularly useful for VCS version strings, that can contain
+# slashes, colons (OK in filenames but not in rules), and spaces.
+sanitize = $(subst $(space),_,$(subst :,_,$(subst /,_,$(strip $(1)))))
+
+# MESSAGE Macro -- display a message in bold type
+MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG)_NAME) $($(PKG)_VERSION) $(call qstrip,$(1))$(TERM_RESET)"
+TERM_BOLD := $(shell tput smso 2>/dev/null)
+TERM_RESET := $(shell tput rmso 2>/dev/null)
+
+# Utility functions for 'find'
+# findfileclauses(filelist) => -name 'X' -o -name 'Y'
+findfileclauses = $(call notfirstword,$(patsubst %,-o -name '%',$(1)))
+# finddirclauses(base, dirlist) => -path 'base/dirX' -o -path 'base/dirY'
+finddirclauses = $(call notfirstword,$(patsubst %,-o -path '$(1)/%',$(2)))
+
+# Miscellaneous utility functions
+# notfirstword(wordlist): returns all but the first word in wordlist
+notfirstword = $(wordlist 2,$(words $(1)),$(1))
+
+# Needed for the foreach loops to loop over the list of hooks, so that
+# each hook call is properly separated by a newline.
+define sep
+
+
+endef
+
+PERCENT = %
+QUOTE = '
+# ' # Meh... syntax-highlighting
+
+# This macro properly escapes a command string, then prints it with printf:
+#
+#   - first, backslash '\' are self-escaped, so that they do not escape
+#     the following char and so that printf properly outputs a backslash;
+#
+#   - next, single quotes are escaped by closing an existing one, adding
+#     an escaped one, and re-openning a new one (see below for the reason);
+#
+#   - then '%' signs are self-escaped so that the printf does not interpret
+#     them as a format specifier, in case the variable contains an actual
+#     printf with a format;
+#
+#   - finally, $(sep) is replaced with the literal '\n' so that make does
+#     not break on the so-expanded variable, but so that the printf does
+#     correctly output an LF.
+#
+# Note: this must be escaped in this order to avoid over-escaping the
+# previously escaped elements.
+#
+# Once everything has been escaped, it is passed between single quotes
+# (that's why the single-quotes are escaped they way they are, above,
+# and why the dollar sign is not escaped) to printf(1). A trailing
+# newline is apended, too.
+#
+# Note: leading or trailing spaces are *not* stripped.
+#
+define PRINTF
+	printf '$(subst $(sep),\n,\
+		$(subst $(PERCENT),$(PERCENT)$(PERCENT),\
+			$(subst $(QUOTE),$(QUOTE)\$(QUOTE)$(QUOTE),\
+				$(subst \,\\,$(1)))))\n'
+endef
-- 
2.7.4

[Buildroot] [PATCH 02/16 v3] core: commonalise the bundled and br2-external %_defconfig rules

[Yann E. MORIN]

The code for both cases is exactly the same, and only differs in the
location where defconfig files are looked for.

We use an intermediate macro to generate the corresponding rules,
because directly generating the rules is ugly and needs lots of escaping
and double-dollar-ing for the $(eval ...) and $(foreach ...) calls to
play nicely together.

Furthermore, that will be tremendously useful when we support multiple
br2-external trees.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---
Changes v2 -> v3:
  - use lower-case macro  (Arnout)
  - move comment in the generated rule  (Arnout)
---
 Makefile | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/Makefile b/Makefile
index af2d982..7c0dcb1 100644
--- a/Makefile
+++ b/Makefile
@@ -847,14 +847,13 @@ olddefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< --defconfig$(if $(DEFCONFIG),=$(DEFCONFIG)) $(CONFIG_CONFIG_IN)
 
-# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
-%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(TOPDIR)/configs/%_defconfig outputmakefile
-	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/configs/$@ \
-		$< --defconfig=$(TOPDIR)/configs/$@ $(CONFIG_CONFIG_IN)
-
-%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig outputmakefile
-	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(BR2_EXTERNAL)/configs/$@ \
-		$< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
+define percent_defconfig
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
+	# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
+	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \
+		$$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN)
+endef
+$(eval $(foreach d,$(TOPDIR) $(BR2_EXTERNAL),$(call percent_defconfig,$(d))$(sep)))
 
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< \
-- 
2.7.4

[Buildroot] [PATCH 03/16 v3] doc/asciidoc: add possibility to define document dependencies

[Yann E. MORIN]

Currently, a document can not have dependencies, except for the purely
internal ones (like checking asciidoc version, and presence of dblatex).

For our own manual, this will come in handy when we introduce a
generated kconfig snippet, so we can actually make the manual depend on
that snippet being generated first.

For external documents, it can be used to depend on host-packages if
need be (e.g. a custom host packages that generates specific media files
included in the manual).

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 docs/manual/adding-packages-asciidoc.txt | 3 +++
 package/doc-asciidoc.mk                  | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/docs/manual/adding-packages-asciidoc.txt b/docs/manual/adding-packages-asciidoc.txt
index 6e21786..0c10fd1 100644
--- a/docs/manual/adding-packages-asciidoc.txt
+++ b/docs/manual/adding-packages-asciidoc.txt
@@ -63,6 +63,9 @@ information is (assuming the document name is +foo+) :
   to one or more directories containing so-called resources (like CSS or
   images). By default, empty.
 
+* +FOO_DEPENDENCIES+, optional, the list of packages (most probably,
+  host-packages) that must be built before building this document.
+
 There are also additional hooks (see xref:hooks[] for general information
 on hooks), that a document may set to define extra actions to be done at
 various steps:
diff --git a/package/doc-asciidoc.mk b/package/doc-asciidoc.mk
index 6ab5ad2..ede6ed1 100644
--- a/package/doc-asciidoc.mk
+++ b/package/doc-asciidoc.mk
@@ -140,7 +140,7 @@ endef
 ################################################################################
 define ASCIIDOC
 # Single line, because splitting a foreach is not easy...
-$(1)-check-dependencies: asciidoc-check-dependencies
+$(1)-check-dependencies: asciidoc-check-dependencies $$($(2)_DEPENDENCIES)
 	$$(Q)$$(foreach hook,$$($(2)_CHECK_DEPENDENCIES_HOOKS),$$(call $$(hook))$$(sep))
 
 # Single line, because splitting a foreach is not easy...
-- 
2.7.4

[Buildroot] [PATCH 04/16 v3] core: introduce an intermediate rule before the configurators

[Yann E. MORIN]

Currently, all configurators depend on generating the out-of-tree
Makefile wrapper.

In an upcoming patch, we'll need to also generate a kconfig fragment,
so it will have to kick in before we run the configurators.

Introduce a new intermediate "prepare-kconfig" rule, so we can
commonalise the dependencies of the configurators. Move the dependency
on the Makefile wrapper to that new intermediate rule.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 Makefile | 36 +++++++++++++++++++-----------------
 1 file changed, 19 insertions(+), 17 deletions(-)

diff --git a/Makefile b/Makefile
index 7c0dcb1..6f947f3 100644
--- a/Makefile
+++ b/Makefile
@@ -760,6 +760,8 @@ endif # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
+prepare-kconfig: outputmakefile
+
 $(BUILD_DIR)/buildroot-config/%onf:
 	mkdir -p $(@D)/lxdialog
 	PKG_CONFIG_PATH="$(HOST_PKG_CONFIG_PATH)" $(MAKE) CC="$(HOSTCC_NOCCACHE)" HOSTCC="$(HOSTCC_NOCCACHE)" \
@@ -779,19 +781,19 @@ COMMON_CONFIG_ENV = \
 	HOST_GCC_VERSION="$(HOSTCC_VERSION)" \
 	SKIP_LEGACY=
 
-xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
+xconfig: $(BUILD_DIR)/buildroot-config/qconf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
-gconfig: $(BUILD_DIR)/buildroot-config/gconf outputmakefile
+gconfig: $(BUILD_DIR)/buildroot-config/gconf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) srctree=$(TOPDIR) $< $(CONFIG_CONFIG_IN)
 
-menuconfig: $(BUILD_DIR)/buildroot-config/mconf outputmakefile
+menuconfig: $(BUILD_DIR)/buildroot-config/mconf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
-nconfig: $(BUILD_DIR)/buildroot-config/nconf outputmakefile
+nconfig: $(BUILD_DIR)/buildroot-config/nconf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
-config: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+config: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
 # For the config targets that automatically select options, we pass
@@ -799,22 +801,22 @@ config: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 # no values are set for the legacy options so a subsequent oldconfig
 # will query them. Therefore, run an additional olddefconfig.
 
-oldconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+oldconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< --oldconfig $(CONFIG_CONFIG_IN)
 
-randconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+randconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y $< --randconfig $(CONFIG_CONFIG_IN)
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-allyesconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+allyesconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y $< --allyesconfig $(CONFIG_CONFIG_IN)
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-allnoconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+allnoconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y $< --allnoconfig $(CONFIG_CONFIG_IN)
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-randpackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+randpackageconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
 		KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
@@ -822,7 +824,7 @@ randpackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@rm -f $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-allyespackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+allyespackageconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
 		KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
@@ -830,7 +832,7 @@ allyespackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@rm -f $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
 		KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
@@ -838,24 +840,24 @@ allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@rm -f $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-silentoldconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+silentoldconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	$(COMMON_CONFIG_ENV) $< --silentoldconfig $(CONFIG_CONFIG_IN)
 
-olddefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+olddefconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN)
 
-defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+defconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< --defconfig$(if $(DEFCONFIG),=$(DEFCONFIG)) $(CONFIG_CONFIG_IN)
 
 define percent_defconfig
-%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig prepare-kconfig
 	# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
 	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \
 		$$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN)
 endef
 $(eval $(foreach d,$(TOPDIR) $(BR2_EXTERNAL),$(call percent_defconfig,$(d))$(sep)))
 
-savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
+savedefconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 	@$(COMMON_CONFIG_ENV) $< \
 		--savedefconfig=$(if $(DEFCONFIG),$(DEFCONFIG),$(CONFIG_DIR)/defconfig) \
 		$(CONFIG_CONFIG_IN)
-- 
2.7.4

[Buildroot] [PATCH 05/16 v3] core: move rule to create basic directories

[Yann E. MORIN]

Some of those directories will be needed even during configuration, like
BUILD_DIR, where we'll store the generated kconfig snippet.

So, move the rule to create them outside the HAVE_CONFIG block.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
---
 Makefile | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/Makefile b/Makefile
index 6f947f3..d4dc1c5 100644
--- a/Makefile
+++ b/Makefile
@@ -200,6 +200,15 @@ LEGAL_LICENSES_TXT_HOST = $(LEGAL_INFO_DIR)/host-licenses.txt
 LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings
 LEGAL_REPORT = $(LEGAL_INFO_DIR)/README
 
+################################################################################
+#
+# staging and target directories do NOT list these as
+# dependencies anywhere else
+#
+################################################################################
+$(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST):
+	@mkdir -p $@
+
 BR2_CONFIG = $(CONFIG_DIR)/.config
 
 # Pull in the user's configuration file
@@ -496,15 +505,6 @@ world: target-post-image
 	legal-info legal-info-prepare legal-info-clean printvars help \
 	list-defconfigs target-finalize target-post-image source-check
 
-################################################################################
-#
-# staging and target directories do NOT list these as
-# dependencies anywhere else
-#
-################################################################################
-$(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST):
-	@mkdir -p $@
-
 # Populating the staging with the base directories is handled by the skeleton package
 $(STAGING_DIR):
 	@mkdir -p $(STAGING_DIR)
-- 
2.7.4

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Yann E. MORIN]

Add the infrastructure for adding generated kconfig snippet in the
menuconfig.

For now, the kconfig snippet is generated empty, the recipe for filling
it in will be introduced in sub-sequent patches.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Samuel Martin <s.martin49@gmail.com>
---
 Config.in             | 6 ++++++
 Makefile              | 7 ++++++-
 docs/manual/manual.mk | 3 +++
 3 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/Config.in b/Config.in
index 7bb2a01..7f533ce 100644
--- a/Config.in
+++ b/Config.in
@@ -18,6 +18,10 @@ config BR2_EXTERNAL
 	string
 	option env="BR2_EXTERNAL"
 
+config BR2_BUILD_DIR
+	string
+	option env="BUILD_DIR"
+
 # Hidden config symbols for packages to check system gcc version
 config BR2_HOST_GCC_VERSION
 	string
@@ -763,3 +767,5 @@ menu "User-provided options"
 source "$BR2_EXTERNAL/Config.in"
 
 endmenu
+
+source "$BR2_BUILD_DIR/.br2-external.in"
diff --git a/Makefile b/Makefile
index d4dc1c5..39a1507 100644
--- a/Makefile
+++ b/Makefile
@@ -760,7 +760,7 @@ endif # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
-prepare-kconfig: outputmakefile
+prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in
 
 $(BUILD_DIR)/buildroot-config/%onf:
 	mkdir -p $(@D)/lxdialog
@@ -779,6 +779,7 @@ COMMON_CONFIG_ENV = \
 	BR2_CONFIG=$(BR2_CONFIG) \
 	BR2_EXTERNAL=$(BR2_EXTERNAL) \
 	HOST_GCC_VERSION="$(HOSTCC_VERSION)" \
+	BUILD_DIR=$(BUILD_DIR) \
 	SKIP_LEGACY=
 
 xconfig: $(BUILD_DIR)/buildroot-config/qconf prepare-kconfig
@@ -879,6 +880,10 @@ ifeq ($(NEED_WRAPPER),y)
 	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
 endif
 
+.PHONY: $(BUILD_DIR)/.br2-external.in
+$(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
+	@touch $@
+
 # printvars prints all the variables currently defined in our
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
 # only the variables matching the make pattern passed in VARS are
diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
index caf080a..b26f912 100644
--- a/docs/manual/manual.mk
+++ b/docs/manual/manual.mk
@@ -7,6 +7,9 @@
 MANUAL_SOURCES = $(sort $(wildcard docs/manual/*.txt) $(wildcard docs/images/*))
 MANUAL_RESOURCES = $(TOPDIR)/docs/images
 
+# Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
+MANUAL_DEPENDENCIES += prepare-kconfig
+
 # Our manual needs to generate lists
 define MANUAL_GEN_LISTS
 	$(Q)$(call MESSAGE,"Updating the manual lists...")
-- 
2.7.4

[Buildroot] [PATCH 07/16 v3] docs/manual: prepare-config can be used as a dependency of documents

[Yann E. MORIN]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 docs/manual/adding-packages-asciidoc.txt | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/docs/manual/adding-packages-asciidoc.txt b/docs/manual/adding-packages-asciidoc.txt
index 0c10fd1..a278d44 100644
--- a/docs/manual/adding-packages-asciidoc.txt
+++ b/docs/manual/adding-packages-asciidoc.txt
@@ -65,6 +65,8 @@ information is (assuming the document name is +foo+) :
 
 * +FOO_DEPENDENCIES+, optional, the list of packages (most probably,
   host-packages) that must be built before building this document.
+  If a hook of your document needs to access the _Kconfig_ structure,
+  you may add +prepare-kconfig+ to the list of dependencies.
 
 There are also additional hooks (see xref:hooks[] for general information
 on hooks), that a document may set to define extra actions to be done at
-- 
2.7.4

[Buildroot] [PATCH 08/16 v3] core: do not hard-code inclusion of br2-external in Kconfig

[Yann E. MORIN]

Move the inclusion of br2-external's Config.in to the generated kconfig
snippet.

This will ultimately allow us to use more than one br2-external tree.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
 Config.in | 7 -------
 Makefile  | 8 +++++++-
 2 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/Config.in b/Config.in
index 7f533ce..7810bf8 100644
--- a/Config.in
+++ b/Config.in
@@ -761,11 +761,4 @@ source "package/Config.in.host"
 
 source "Config.in.legacy"
 
-menu "User-provided options"
-	depends on BR2_EXTERNAL != "support/dummy-external"
-
-source "$BR2_EXTERNAL/Config.in"
-
-endmenu
-
 source "$BR2_BUILD_DIR/.br2-external.in"
diff --git a/Makefile b/Makefile
index 39a1507..86c532e 100644
--- a/Makefile
+++ b/Makefile
@@ -882,7 +882,13 @@ endif
 
 .PHONY: $(BUILD_DIR)/.br2-external.in
 $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
-	@touch $@
+	$(Q)( \
+		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
+		printf 'menu "User-provided options"\n'; \
+		printf '\tdepends on BR2_EXTERNAL != "support/dummy-external"\n\n'; \
+		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
+		printf 'endmenu # User-provided options\n'; \
+	) >$@
 
 # printvars prints all the variables currently defined in our
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
-- 
2.7.4

[Buildroot] [PATCH 09/16 v3] core: get rid of our dummy br2-external tree

[Yann E. MORIN]

Now that we generate a kconfig snippet, we can conditionally include the
BR2_EXTERNAL's Config.in only when BR2_EXTERNAL is supplied by the user,
which means our empty/dummy Config.in is no longer useful to us.

As for external.mk, we can also include it only when BR2_EXTERNAL is
supplied by the user, which means our empty/dummy external.mk is no
longer of any use to us.

Ditch both of those files, and:

  - only generate actual content in the Kconfig snippet when we actually
    do have a BR2_EXTERNAL provided by the user (i.e. BR2_EXTERNAL is not
    empty);

  - add a variable that contains the path to the external.mk provided by
    the user, or empty if none, and include the path set in that variable
    (make can 'include' nothing without any problem! ;-) )

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
 Makefile                           | 18 +++++++-----------
 support/dummy-external/Config.in   |  0
 support/dummy-external/external.mk |  0
 3 files changed, 7 insertions(+), 11 deletions(-)
 delete mode 100644 support/dummy-external/Config.in
 delete mode 100644 support/dummy-external/external.mk

diff --git a/Makefile b/Makefile
index 86c532e..be2f586 100644
--- a/Makefile
+++ b/Makefile
@@ -147,16 +147,11 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 # on the command line, therefore the file is re-created every time make is run.
 #
 # When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
-# line), the .br-external file is removed and we point to
-# support/dummy-external. This makes sure we can unconditionally include the
-# Config.in and external.mk from the BR2_EXTERNAL directory. In this case,
-# override is necessary so the user can clear BR2_EXTERNAL from the command
-# line, but the dummy path is still used internally.
+# line), the .br-external file is removed.
 
 BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
 -include $(BR2_EXTERNAL_FILE)
 ifeq ($(BR2_EXTERNAL),)
-  override BR2_EXTERNAL = support/dummy-external
   $(shell rm -f $(BR2_EXTERNAL_FILE))
 else
   _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
@@ -165,6 +160,7 @@ else
   endif
   override BR2_EXTERNAL := $(_BR2_EXTERNAL)
   $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
+  BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
 endif
 
 # To make sure that the environment variable overrides the .config option,
@@ -461,7 +457,8 @@ include boot/common.mk
 include linux/linux.mk
 include fs/common.mk
 
-include $(BR2_EXTERNAL)/external.mk
+# Nothing to include if no BR2_EXTERNAL tree in use
+include $(BR2_EXTERNAL_MK)
 
 # Now we are sure we have all the packages scanned and defined. We now
 # check for each package in the list of enabled packages, that all its
@@ -882,13 +879,12 @@ endif
 
 .PHONY: $(BUILD_DIR)/.br2-external.in
 $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
-	$(Q)( \
+	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
 		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
-		printf 'menu "User-provided options"\n'; \
-		printf '\tdepends on BR2_EXTERNAL != "support/dummy-external"\n\n'; \
+		printf 'menu "User-provided options"\n\n'; \
 		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
 		printf 'endmenu # User-provided options\n'; \
-	) >$@
+	fi >$@
 
 # printvars prints all the variables currently defined in our
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
diff --git a/support/dummy-external/Config.in b/support/dummy-external/Config.in
deleted file mode 100644
index e69de29..0000000
diff --git a/support/dummy-external/external.mk b/support/dummy-external/external.mk
deleted file mode 100644
index e69de29..0000000
-- 
2.7.4

[Buildroot] [PATCH 10/16 v3] core: introduce per br2-external ID

[Yann E. MORIN]

This unique ID is used to construct a per br2-external tree variable,
BR2_EXTERNAL_$(ID), which contains the path to the br2-external tree.

This variable is available both from Kconfig (set in the Kconfig
snippet) and from the .mk files.

Also, display the BR2_EXTERNAL_$(ID) and its path as a comment in the
menuconfig.

This will ultimately allow us to support multiple br2-external trees at
once, with that ID (and thus BR2_EXTERNAL_$(ID)) uniquely defining which
br2-external tree is being used.

Note: since the variables in the Makefile and in Kconfig are named the
same, the one we computed early on (second hunk) will be overridden by
the one in .config when we have it. Thus, even though they are set to
the same raw value, the one from .config is quoted and, being included
later in the Makefile, will take precedence, so we must un-quote it
before we include the br2-external's makefile (third hunk). That's
unfortunate, but there is no easy way around that as we want the two
variables to be named the same in Makefile and Kconfig (and we can't
ask the user to un-quote that variable himself either), hence the
little dirty trick.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
 Makefile | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index be2f586..72b55cb 100644
--- a/Makefile
+++ b/Makefile
@@ -148,6 +148,9 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 #
 # When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
 # line), the .br-external file is removed.
+#
+# If the br2-external tree defines its ID, then export the path in the
+# BR2_EXTERNAL_$(ID) variable.
 
 BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
 -include $(BR2_EXTERNAL_FILE)
@@ -160,6 +163,13 @@ else
   endif
   override BR2_EXTERNAL := $(_BR2_EXTERNAL)
   $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
+  ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
+    BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
+    ifeq ($(BR2_EXTERNAL_ID),)
+      $(error $(BR2_EXTERNAL) has no ID (in file 'external.id'))
+    endif
+    BR2_EXTERNAL_$(BR2_EXTERNAL_ID) = $(BR2_EXTERNAL)
+  endif
   BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
 endif
 
@@ -457,6 +467,10 @@ include boot/common.mk
 include linux/linux.mk
 include fs/common.mk
 
+# If the br2-external tree defines its ID, then the BR2_EXTERNAL_$(ID)
+# variable is also present in .config, so it is quoted. We must unquote
+# it before feeding it to the br2-external makefile.
+BR2_EXTERNAL_$(BR2_EXTERNAL_ID) := $(call qstrip,$(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)))
 # Nothing to include if no BR2_EXTERNAL tree in use
 include $(BR2_EXTERNAL_MK)
 
@@ -882,7 +896,13 @@ $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
 	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
 		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
 		printf 'menu "User-provided options"\n\n'; \
-		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
+		if [ -z "$(BR2_EXTERNAL_ID)" ]; then \
+			printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
+		else \
+			printf 'config BR2_EXTERNAL_%s\n' $(BR2_EXTERNAL_ID); \
+			printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \
+			printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(BR2_EXTERNAL_ID); \
+		fi; \
 		printf 'endmenu # User-provided options\n'; \
 	fi >$@
 
-- 
2.7.4

[Buildroot] [PATCH 11/16 v3] docs/manual: document the br2-external ID

[Yann E. MORIN]

Update the manual with the new external.id mandatory file.

Take the opportunity to add a section listing all mandatory files,
Config.in, external.mk and the new external.id, instead of just hinting
about them in the external package recipes section.

Change the example to use the ID-suffixed variable instead of the raw
BR2_EXTERNAL variable. Even though it is still possible to write a
br2-external tree that is not multi-aware, we only document it with an
ID.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Jeremy Rosen <jeremy.rosen@openwide.fr>

---
Changes v2 -> v3:
  - better document the content of the ID file  (Jeremy)
---
 docs/manual/customize-outside-br.txt | 59 ++++++++++++++++++++++++++----------
 1 file changed, 43 insertions(+), 16 deletions(-)

diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
index 9ad177d..839d1b1 100644
--- a/docs/manual/customize-outside-br.txt
+++ b/docs/manual/customize-outside-br.txt
@@ -57,19 +57,50 @@ Or disable the usage of external definitions:
 buildroot/ $ make BR2_EXTERNAL= xconfig
 -----
 
-+BR2_EXTERNAL+ allows three different things:
+A +BR2_EXTERNAL+ tree must contain at least those three files:
+
++external.id+::
+   That file should contain the ID for the +BR2_EXTERNAL+ tree.
+   That ID is used to construct the +BR2_EXTERNAL_$(ID)+ variable,
+   available in +Config.in+ and +external.mk+ (see below), so that ID
+   must be a valid make and Kconfig variable. Buildroot sets
+   +BR2_EXTERNAL_$(ID)+ to the path of the +BR2_EXTERNAL+ tree.
+   The format for this file is a single line with the ID.
++
+Examples of an +external.mk+ file that declares the ID +FOO+:
++
+----
+$ cat external.mk
+FOO
+----
++
+Examples of IDs and the corresponding +BR2_EXTERNAL_$(ID)+ variable:
++
+  * +FOO+ -> +BR2_EXTERNAL_FOO+
+  * +BAR_42+ -> +BR2_EXTERNAL_BAR_42+
+  * ...
+
+
++Config.in+::
++external.mk+::
+   Those files (which may each be empty) can be used to define package
+   recipes, like for packages bundled in Buildroot itself, or other
+   custom configuration options.
+
+Using +BR2_EXTERNAL+ then allows three different things:
 
  * One can store all the board-specific configuration files there,
    such as the kernel configuration, the root filesystem overlay, or
    any other configuration file for which Buildroot allows to set its
-   location. The +BR2_EXTERNAL+ value is available within the
-   Buildroot configuration using +$(BR2_EXTERNAL)+. As an example, one
-   could set the +BR2_ROOTFS_OVERLAY+ Buildroot option to
-   +$(BR2_EXTERNAL)/board/<boardname>/overlay/+ (to specify a root
+   location. The location of the +BR2_EXTERNAL+ tree is available in the
+   Buildroot configuration using the +BR2_EXTERNAL_$(ID)+ variable. As
+   an example, (considering the ID +BAR_42+), one could set the
+   +BR2_ROOTFS_OVERLAY+ Buildroot option to
+   +$(BR2_EXTERNAL_BAR_42)/board/<boardname>/overlay/+ (to specify a root
    filesystem overlay), or the +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
    Buildroot option to
-   +$(BR2_EXTERNAL)/board/<boardname>/kernel.config+ (to specify the
-   location of the kernel configuration file).
+   +$(BR2_EXTERNAL_BAR_42)/board/<boardname>/kernel.config+ (to specify
+   the location of the kernel configuration file).
 
  * One can store package recipes (i.e. +Config.in+ and
    +<packagename>.mk+), or even custom configuration options and make
@@ -77,27 +108,23 @@ buildroot/ $ make BR2_EXTERNAL= xconfig
    make it appear in the top-level configuration menu, and includes
    +$(BR2_EXTERNAL)/external.mk+ with the rest of the makefile logic.
 +
-.Note
-Providing +Config.in+ and +external.mk+ is mandatory, but they can be
-   empty.
-+
 The main usage of this is to store package recipes. The recommended
    way to do this is to write a +$(BR2_EXTERNAL)/Config.in+ file that
    looks like:
 +
 ------
-source "$BR2_EXTERNAL/package/package1/Config.in"
-source "$BR2_EXTERNAL/package/package2/Config.in"
+source "$BR2_EXTERNAL_BAR_42/package/package1/Config.in"
+source "$BR2_EXTERNAL_BAR_42/package/package2/Config.in"
 ------
 +
 Then, have a +$(BR2_EXTERNAL)/external.mk+ file that looks like:
 +
 ------
-include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))
+include $(sort $(wildcard $(BR2_EXTERNAL_BAR_42)/package/*/*.mk))
 ------
 +
-And then in +$(BR2_EXTERNAL)/package/package1+ and
-   +$(BR2_EXTERNAL)/package/package2+ create normal Buildroot
+And then in +$(BR2_EXTERNAL_FOO_42)/package/package1+ and
+   +$(BR2_EXTERNAL_FOO_42)/package/package2+ create normal Buildroot
    package recipes, as explained in xref:adding-packages[].
    If you prefer, you can also group the packages in subdirectories
    called <boardname> and adapt the above paths accordingly.
-- 
2.7.4

[Buildroot] [PATCH 00/16 v3] br2-external: support multiple trees at once (branch yem/multi-br2-external-ID-7)

[Yann E. MORIN]

Hello All!

This rather complex series introduces support for using multiple
br2-external trees at once.


Currently, Buildroot can use what we call a br2-external tree, that is a
location where one can store customisations outisde of the Buildroot
tree:

  - defconfig files;
  - packages;
  - custom make rules and logic;
  - company-, project- and board-related configurations files, like a
    kernel config file, a rootfs overlay;
  - any other content that does not belong to the Buildroot tree...

This is very nice and very handy!

However, it is not possible to use more than one such br2-external tree
at a time; Buildroot only supports using at most one br2-external tree.


Having support for using more than one br2-external tree at the same
time would allow more flexibility organising those out-of-tree
customisations. One could use a br2-external tree to store FLOSS
packages not yet in Buildroot and a second br2-external tree for
proprietary packages; such a separation would make compliance with
FLOSS licenses easier (by allowing distributing only the br2-external
tree with FLOSS packages, keeping the proprietary bits private).

It could also be used to commonalise br2-external trees from different
internal projects, where each would provide a set of extensions (new
packages, new boards defconfigs...) that a project could cherry-pick.

And countless other possibilities.

This series is an attempt at making it possible to use such setups.


The series is organised as such:

  - patch 1 is a little cleanup that moves common Makefiles definitions
    out of the package infra and out of the main Makefile, into a
    separate file. Those definitions are generic, not specific to
    packages, and may be used throughout the Makefile.

  - patches 2..5 are preparatory work aimed at allowing to generate a
    kconfig snippet in the menuconfig.

  - patches 6..9 introduce a generated kconfig snippet, that is used to
    include the br2-external tree, rather than have conditional code in
    our existing static kconfig. They also get rid of our bundled, dummy
    br2-external tree, as it is no longer needed.

  - patches 10..11 introduce support for br2-external trees to define an
    hopefully-globally-unique ID describing this br2-external tree. See
    below for an in-depth expanation for the reason for that ID.

  - patch 12 adds a Makefile rule to generate out .br-external file,
    rather than handling it with cutom Makefile code (arguably, this
    change could have come before 10..11, but since I wrote it in this
    order and they concflict pretty hard, I left it in this position).

  - patch 13 is is the core of the series; it eventually introduces
    support for using multiple br2-external trees at once. Tanks to all
    the previous changes, it is pretty light (70+, 31-), even if a bit
    complex;

  - patch 14..15 teaches gen-manual to generate lists for packages from
    the br2-external trees,if any and if requested (default: do not
    include those packages).

  - patch 16 eventually documents support for multi br2-external trees.


The pivotal change in this series is implemented in patch 10, and aims
at providing br2-external trees with a way to find their own files,
given that their locations at runtime is completely unkown, and can
not even be deduced relative to Buildroot's own TOPDIR; this is
explained below.

Unlike Makefiles, Kconfig does not allow us to redefine a variable at
various moments of the parsing, because the values are only valid and
acted upon when the complete Kconfig code has been parsed. So, we need
a way to provide Config.in files from each br2-external with their
actual location. For that, we need different variables, one for each
br2-external tree.

So, we need a way to uniquely identify each br2-external tree, and we
need a stable scheme. We can't use indexes (because they would not be
stable across various runs or various ordering of the br2-external
trees); we can't use path-based hashing (because paths are not stable
accross users/machines/time); we can't use content-based hashing
(external.mk and/or Config.in may all be empty, giving the same hash);
we can't use hashes altogether because they are not human-friendly.

The solution is to require br2-external trees to provide a unique ID
that is used as a kind of 'name-space' for that tree: we eventually
provide a single variable, BR2_EXTERNAL_$(ID) that points to the
location of that tree; because $(ID) is known to the external tree,
it also knows BR2_EXTERNAL_$(ID) exists, so it can use it, and we
export it both in Config.in and external.mk .

Therefore, br2-external trees must now use $(BR2_EXTERNAL_$(MY_ID)) to
refer to their own files. It also prevent them from (accidentally)
including files from other br2-external trees.

Note: IDs are optional, so that existing br2-external trees continue to
work as before. However, if a br2-external tree does not provide an ID,
it will not be useable in conjunction with other br2-external trees.
Using multiple br2-external trees mandates that all of them provide an
ID.


Please have a look at the various commit logs for more details about
each change, they are usually pretty copious in details (sometimes even
much bigger than the actual change they are commenting!).


And that's about all for this time. ;-)


Changes v2 -> v3:
  - not much. ;-)

Changes RFC -> v2:
  - much. :-]


Regards,
Yann E. MORIN.


The following changes since commit 97308d8e9ae0f096ec2df8275deb063266583b91

  package/ti-gfx: list supported SoCs in Config.in help text (2016-07-16 22:53:55 +0200)


are available in the git repository at:

  https://gitlab.com/ymorin/buildroot.git

for you to fetch changes up to 8eca23fffd802ef089aab2300b9041455076e771

  docs/manual: document multi br2-external (2016-07-17 12:06:42 +0200)


----------------------------------------------------------------
Yann E. MORIN (16):
      core: move pkg-utils.mk to support/
      core: commonalise the bundled and br2-external %_defconfig rules
      doc/asciidoc: add possibility to define document dependencies
      core: introduce an intermediate rule before the configurators
      core: move rule to create basic directories
      core: introduce a generated kconfig snippet
      docs/manual: prepare-config can be used as a dependency of documents
      core: do not hard-code inclusion of br2-external in Kconfig
      core: get rid of our dummy br2-external tree
      core: introduce per br2-external ID
      docs/manual: document the br2-external ID
      core: handle .br-external in a make rule
      core: add support for multiple br2-external trees
      support/scripts: teach gen-manual-lists about BR2_EXTERNAL
      docs/manual: include packages from BR2_EXTERNAL if set
      docs/manual: document multi br2-external

 Config.in                                     |  11 +-
 Makefile                                      | 184 +++++++++++++++++---------
 docs/manual/adding-packages-asciidoc.txt      |   5 +
 docs/manual/appendix.txt                      |   5 +
 docs/manual/customize-outside-br.txt          | 119 ++++++++++++-----
 docs/manual/manual.mk                         |  21 +++
 package/Makefile.in                           |   2 +-
 package/doc-asciidoc.mk                       |   2 +-
 package/pkg-utils.mk                          |  96 --------------
 support/dummy-external/Config.in              |   0
 support/dummy-external/external.mk            |   0
 package/pkg-utils.mk => support/misc/utils.mk | 106 ++-------------
 support/scripts/gen-manual-lists.py           |  24 +++-
 13 files changed, 276 insertions(+), 299 deletions(-)
 delete mode 100644 support/dummy-external/Config.in
 delete mode 100644 support/dummy-external/external.mk
 copy package/pkg-utils.mk => support/misc/utils.mk (57%)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 12/16 v3] core: handle .br-external in a make rule

[Yann E. MORIN]

Currently, we treat the case where we have no br2-external tree
(BR2_EXTERNAL is empty) differently from the case where we do have
one (BR2_EXTERNAL is not empty).

There is now no reason to treat those two cases differently:

  - the kconfig snippet is always generated appropriately (i.e. it would
    include the br2-external tree if set, or include nothing otherwise);

  - we no longer have a dummy br-external tree either.

As such, generate the .br-external cache file in both cases.

This will make it much easier to handle when we introduce support for
multiple br2-external trees.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
 Makefile | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/Makefile b/Makefile
index 72b55cb..67334e5 100644
--- a/Makefile
+++ b/Makefile
@@ -146,23 +146,17 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 # On subsequent invocations of make, it is read in. It can still be overridden
 # on the command line, therefore the file is re-created every time make is run.
 #
-# When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
-# line), the .br-external file is removed.
-#
 # If the br2-external tree defines its ID, then export the path in the
 # BR2_EXTERNAL_$(ID) variable.
 
 BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
 -include $(BR2_EXTERNAL_FILE)
-ifeq ($(BR2_EXTERNAL),)
-  $(shell rm -f $(BR2_EXTERNAL_FILE))
-else
+ifneq ($(BR2_EXTERNAL),)
   _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
   ifeq ($(_BR2_EXTERNAL),)
     $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' does not exist, relative to $(TOPDIR))
   endif
   override BR2_EXTERNAL := $(_BR2_EXTERNAL)
-  $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
   ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
     BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
     ifeq ($(BR2_EXTERNAL_ID),)
@@ -173,6 +167,11 @@ else
   BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
 endif
 
+# This needs to be *after* we compute BR_EXTERNAL, above.
+.PHONY: $(BR2_EXTERNAL_FILE)
+$(BR2_EXTERNAL_FILE):
+	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@
+
 # To make sure that the environment variable overrides the .config option,
 # set this before including .config.
 ifneq ($(BR2_DL_DIR),)
-- 
2.7.4

[Buildroot] [PATCH 13/16 v3] core: add support for multiple br2-external trees

[Yann E. MORIN]

Currently, we only support at most one br2-external tree. Being able
to use more than one br2-external tree can be very useful.

A use-case would be for having a br2-external to contain the basic
packages, basic board defconfigs and board files, provided by one team
responsible for the "board-bringup", while other teams consume that
br2-external as a base, and complements it each with their own set of
packages, defconfigs and extra board files.

Another use-case would be for third-parties to provide their own
Buildroot packaging in a br2-external tree, along-side the archives for
their stuff.

Finally, another use-case is to be able to add FLOSS packages in a
br2-external tree, and proprietary packages in another. This allows
to not touch the Buildroot tree at all, and still be able to get in
compliance by providing only that br2-external tree(s) that contains
FLOSS packages, leaving aside the br2-external tree(s) with the
proprietary bits.

What we do is to treat BR2_EXTERNAL as a space-separated list of paths,
which we iterate to construct:

  - the list of all br2-external IDs, BR_EXTERNAL_IDS

  - the per-br2-external tree BR2_EXTERNAL_$(ID) variables, which points
    to the actual location of the corresponding tree,

  - EXTRA_ENV which now needs to contain all BR2_EXTERNAL_$(ID)
    variables, and thus needs to be set early.

Once we have all those variables, we replace references to BR2_EXTERNAL
with either:

  - a $(patsubst ...) to include the external.mk files (and docs),
    or

  - a $(foreach ...) iteration to generate other Makefile code.

To be noted: we also use $(foreach ...) in the kconfig-snippet
generating rule, because it is cleaner than doing so in the shell

Finally, we double-quote $(BR2_EXTERNAL) when assigning it to
environment variables.

Now, when more than one br2-external tree is used, each gets its own
sub-menu in the "User-provided options" menu. The sub-menu is labelled
with that br2-external tree's ID (prefixed with 'BR2_EXTERNAL_') and the
sub-menu's first item is a comment with the path to that br2-external
tree.

If there's only one br2-external tree, then there is no sub-menu, but
we still add the comment with the path to the br2-external tree (which
we did not have previously). If the br2-external tree has an ID, it is
displayed as well as a comment.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---
Changes v3 -> v4:
  - drop check for file existence  (Thomas)
  - fix check for empty ID  (Thomas)
  - simpler code to check number of br2-external trees
---
 Makefile            | 99 +++++++++++++++++++++++++++++++++++++----------------
 package/Makefile.in |  2 +-
 2 files changed, 70 insertions(+), 31 deletions(-)

diff --git a/Makefile b/Makefile
index 67334e5..7b3542c 100644
--- a/Makefile
+++ b/Makefile
@@ -145,32 +145,57 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 # The value of BR2_EXTERNAL is stored in .br-external in the output directory.
 # On subsequent invocations of make, it is read in. It can still be overridden
 # on the command line, therefore the file is re-created every time make is run.
-#
-# If the br2-external tree defines its ID, then export the path in the
-# BR2_EXTERNAL_$(ID) variable.
 
 BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
 -include $(BR2_EXTERNAL_FILE)
-ifneq ($(BR2_EXTERNAL),)
-  _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
-  ifeq ($(_BR2_EXTERNAL),)
-    $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' does not exist, relative to $(TOPDIR))
-  endif
-  override BR2_EXTERNAL := $(_BR2_EXTERNAL)
-  ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
-    BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
-    ifeq ($(BR2_EXTERNAL_ID),)
-      $(error $(BR2_EXTERNAL) has no ID (in file 'external.id'))
-    endif
-    BR2_EXTERNAL_$(BR2_EXTERNAL_ID) = $(BR2_EXTERNAL)
-  endif
-  BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
-endif
 
-# This needs to be *after* we compute BR_EXTERNAL, above.
 .PHONY: $(BR2_EXTERNAL_FILE)
 $(BR2_EXTERNAL_FILE):
-	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@
+	@echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) >$@
+
+# Those two variables need to be defined as simply-expanded variables, they
+# can't be recursively-expanded, because the values they are assigned change
+# with each iteration of the foreach, below.
+BR_EXTERNAL_IDS :=
+EXTRA_ENV :=
+
+# If there is no or one br2-external trees used, then we don't require (but
+# accept) an ID; otherwise (i.e. there are two or more br2-external trees)
+# we require they do all define their ID.
+ifeq ($(filter-out 0 1,$(words $(BR2_EXTERNAL))),)
+BR2_EXTERNAL_NEED_ID := loose
+else
+BR2_EXTERNAL_NEED_ID := strict
+endif
+
+# Validate the br2-external tree passed as $(1):
+# - check the directory actually exists
+# - check if we need and have a non-empty ID
+# - check the ID is not a duplicate
+# - set variables for later use
+define BR2_EXTERNAL_VALIDATE
+  _BR_EXT_DIR := $$(shell cd $(1) >/dev/null 2>&1 && pwd)
+  ifeq ($$(_BR_EXT_DIR),)
+    $$(error BR2_EXTERNAL='$(1)' does not exist, relative to $$(TOPDIR))
+  endif
+  _BR_EXT_ID := $$(shell cat $$(_BR_EXT_DIR)/external.id 2>/dev/null)
+  ifeq ($$(_BR_EXT_ID),)
+    ifeq ($(BR2_EXTERNAL_NEED_ID),strict)
+      $$(error BR2_EXTERNAL='$(1)' has no ID (in file 'external.id'),\
+               mandatory to use more than one br2-external tree at once)
+    endif # BR2_EXTERNAL_NEED_ID strict
+  endif # No ID
+  ifneq ($$(filter $$(_BR_EXT_ID),$$(BR_EXTERNAL_IDS)),)
+    $$(error Duplicate ID '$$(_BR_EXT_ID)' in '$(1)', previously defined in '$$(BR2_EXTERNAL_$$(_BR_EXT_ID))')
+  endif
+  ifneq ($$(_BR_EXT_ID),)
+    BR2_EXTERNAL_$$(_BR_EXT_ID) := $$(_BR_EXT_DIR)
+    BR_EXTERNAL_IDS += $$(_BR_EXT_ID)
+    EXTRA_ENV += BR2_EXTERNAL_$$(_BR_EXT_ID)=$$(_BR_EXT_DIR)
+  endif # _BR_EXT_ID not empty
+endef # BR2_EXTERNAL_VALIDATE
+
+$(eval $(foreach d,$(BR2_EXTERNAL),$(call BR2_EXTERNAL_VALIDATE,$(d))$(sep)))
 
 # To make sure that the environment variable overrides the .config option,
 # set this before including .config.
@@ -469,9 +494,10 @@ include fs/common.mk
 # If the br2-external tree defines its ID, then the BR2_EXTERNAL_$(ID)
 # variable is also present in .config, so it is quoted. We must unquote
 # it before feeding it to the br2-external makefile.
-BR2_EXTERNAL_$(BR2_EXTERNAL_ID) := $(call qstrip,$(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)))
+$(eval $(foreach id,$(BR_EXTERNAL_IDS),\
+	BR2_EXTERNAL_$(id) := $(call qstrip,$(BR2_EXTERNAL_$(id)))$(sep)))
 # Nothing to include if no BR2_EXTERNAL tree in use
-include $(BR2_EXTERNAL_MK)
+include $(patsubst %,%/external.mk,$(BR2_EXTERNAL))
 
 # Now we are sure we have all the packages scanned and defined. We now
 # check for each package in the list of enabled packages, that all its
@@ -787,7 +813,7 @@ COMMON_CONFIG_ENV = \
 	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
 	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
 	BR2_CONFIG=$(BR2_CONFIG) \
-	BR2_EXTERNAL=$(BR2_EXTERNAL) \
+	BR2_EXTERNAL="$(BR2_EXTERNAL)" \
 	HOST_GCC_VERSION="$(HOSTCC_VERSION)" \
 	BUILD_DIR=$(BUILD_DIR) \
 	SKIP_LEGACY=
@@ -895,12 +921,25 @@ $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
 	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
 		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
 		printf 'menu "User-provided options"\n\n'; \
-		if [ -z "$(BR2_EXTERNAL_ID)" ]; then \
+		if [ -z "$(call strip,$(BR_EXTERNAL_IDS))" ]; then \
+			printf 'comment "%s"\n\n' $(BR2_EXTERNAL); \
 			printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
 		else \
-			printf 'config BR2_EXTERNAL_%s\n' $(BR2_EXTERNAL_ID); \
-			printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \
-			printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(BR2_EXTERNAL_ID); \
+			$(foreach id,$(BR_EXTERNAL_IDS),\
+				for i in $$(seq 1 80); do printf '#'; done; printf '\n\n'; \
+				if [ $(words $(call strip,$(BR_EXTERNAL_IDS))) -gt 1 ]; then \
+					printf 'menu "BR2_EXTERNAL_%s"\n\n' $(id); \
+				else \
+					printf 'comment "BR2_EXTERNAL_%s"\n\n' $(id); \
+				fi; \
+				printf 'comment "%s"\n\n' $(BR2_EXTERNAL_$(id)); \
+				printf 'config BR2_EXTERNAL_%s\n' $(id); \
+				printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(id)); \
+				printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(id); \
+				if [ $(words $(call strip,$(BR_EXTERNAL_IDS))) -gt 1 ]; then \
+					printf 'endmenu # BR2_EXTERNAL_%s\n\n' $(id); \
+				fi; ) \
+			for i in $$(seq 1 80); do printf '#'; done; printf '\n\n'; \
 		fi; \
 		printf 'endmenu # User-provided options\n'; \
 	fi >$@
@@ -1004,10 +1043,10 @@ list-defconfigs:
 	@echo 'Built-in configs:'
 	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
-ifneq ($(wildcard $(BR2_EXTERNAL)/configs/*_defconfig),)
+ifneq ($(wildcard $(patsubst %,%/configs/*_defconfig,$(BR2_EXTERNAL))),)
 	@echo
 	@echo 'User-provided configs:'
-	@$(foreach b, $(sort $(notdir $(wildcard $(BR2_EXTERNAL)/configs/*_defconfig))), \
+	@$(foreach b, $(sort $(notdir $(wildcard $(patsubst %,%/configs/*_defconfig,$(BR2_EXTERNAL))))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 endif
 	@echo
@@ -1029,7 +1068,7 @@ print-version:
 	@echo $(BR2_VERSION_FULL)
 
 include docs/manual/manual.mk
--include $(BR2_EXTERNAL)/docs/*/*.mk
+-include $(patsubst %,%/docs/*/*.mk,$(BR2_EXTERNAL))
 
 .PHONY: $(noconfig_targets)
 
diff --git a/package/Makefile.in b/package/Makefile.in
index afd5d3a..fe8ee68 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -319,7 +319,7 @@ EXTRA_ENV = \
 	PATH=$(BR_PATH) \
 	BR2_DL_DIR=$(BR2_DL_DIR) \
 	BUILD_DIR=$(BUILD_DIR) \
-	BR2_EXTERNAL=$(BR2_EXTERNAL)
+	BR2_EXTERNAL="$(BR2_EXTERNAL)"
 
 ################################################################################
 # settings we need to pass to configure
-- 
2.7.4

[Buildroot] [PATCH 14/16 v3] support/scripts: teach gen-manual-lists about BR2_EXTERNAL

[Yann E. MORIN]

When using BR2_EXTERNAL, generate the list of packages from there.

This is currently effectively unused for now, as BR2_EXTERNAL is not
available (not exported in the environment); that will be updated in a
later patch.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 support/scripts/gen-manual-lists.py | 24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)

diff --git a/support/scripts/gen-manual-lists.py b/support/scripts/gen-manual-lists.py
index d231eda..a672b7b 100644
--- a/support/scripts/gen-manual-lists.py
+++ b/support/scripts/gen-manual-lists.py
@@ -141,7 +141,6 @@ class Buildroot:
 
     """
     root_config = "Config.in"
-    package_dirname = "package"
     package_prefixes = ["BR2_PACKAGE_", "BR2_PACKAGE_HOST_"]
     re_pkg_prefix = re.compile(r"^(" + "|".join(package_prefixes) + ").*")
     deprecated_symbol = "BR2_DEPRECATED"
@@ -182,12 +181,22 @@ class Buildroot:
             'format': "_format_symbol_prompt_location",
             'sorted': False,
         },
+        'external': {
+            'filename': "br2-external-package-list",
+            'root_menu': "User-provided options",
+            'filter': "_is_real_package",
+            'format': "_format_symbol_prompt_location",
+            'sorted': True,
+        },
     }
 
     def __init__(self):
         self.base_dir = os.environ.get("TOPDIR")
         self.output_dir = os.environ.get("O")
-        self.package_dir = os.path.join(self.base_dir, self.package_dirname)
+        self.package_dirs = []
+        self.package_dirs.append(os.path.join(self.base_dir, "package"))
+        if os.getenv('BR2_EXTERNAL'):
+            self.package_dirs.extend(os.environ["BR2_EXTERNAL"].split())
         self.config = kconfiglib.Config(os.path.join(self.base_dir,
                                                      self.root_config),
                                         self.base_dir)
@@ -259,9 +268,10 @@ class Buildroot:
         #   symbol.
         if not hasattr(self, "_package_list"):
             pkg_list = []
-            for _, _, files in os.walk(self.package_dir):
-                for file_ in (f for f in files if f.endswith(".mk")):
-                    pkg_list.append(re.sub(r"(.*?)\.mk", r"\1", file_))
+            for d in self.package_dirs:
+                for _, _, files in os.walk(d):
+                    for file_ in (f for f in files if f.endswith(".mk")):
+                        pkg_list.append(re.sub(r"(.*?)\.mk", r"\1", file_))
             setattr(self, "_package_list", pkg_list)
         for pkg in getattr(self, "_package_list"):
             if type == 'real':
@@ -491,6 +501,8 @@ class Buildroot:
 
 if __name__ == '__main__':
     list_types = ['target-packages', 'host-packages', 'virtual-packages', 'deprecated']
+    if os.getenv('BR2_EXTERNAL'):
+        list_types.append('external')
     parser = ArgumentParser()
     parser.add_argument("list_type", nargs="?", choices=list_types,
                         help="""\
@@ -505,6 +517,8 @@ Generate the given list (generate all lists if unspecified)""")
                         help="Output virtual package file")
     parser.add_argument("--output-deprecated", dest="output_deprecated",
                         help="Output deprecated file")
+    parser.add_argument("--output-external", dest="output_external",
+                        help="Output br2-external file")
     args = parser.parse_args()
     lists = [args.list_type] if args.list_type else list_types
     buildroot = Buildroot()
-- 
2.7.4

[Buildroot] [PATCH 15/16 v3] docs/manual: include packages from BR2_EXTERNAL if set

[Yann E. MORIN]

If the user is using one or more br2-external trees, add the list of
packages therein to the manual.

Note that we do re-introduce BR2_EXTERNAL (when it was removed in
[-SHA1-] (docs/manual: do not override BR2_EXTERNAL). The reason for
the removal is that BR2_EXTERNAL is no longer used as a Kconfig option.

However, BR2_EXTERNAL is now used by the gen-manual-lists.py script,
which needs it to find packages' .mk files in the br2-external trees,
not as a Kconfig option (which it is not).

So, even though that looks like a revert, that's not: the purpose is
different, and we only pass it when not empty.

In case we have no br2-external, we want to create an empty file, so
we can't just touch it, in case it was previously generated with
br2-external trees; instead, we printf an empty string in there to
ensure the file exists and is empty.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <jacmet@uclibc.org>
---
 docs/manual/appendix.txt |  5 +++++
 docs/manual/manual.mk    | 18 ++++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/docs/manual/appendix.txt b/docs/manual/appendix.txt
index 87a20bd..7ef8bbc 100644
--- a/docs/manual/appendix.txt
+++ b/docs/manual/appendix.txt
@@ -40,3 +40,8 @@ year following deprecation. For example, a symbol depending on
 +BR2_DEPRECATED_SINCE_2013_05+ can be removed from 2014.05 onwards.
 
 include::deprecated-list.txt[]
+
+// This file is generated:
+// - with the list of br2-external packages if BR2_EXTERNAL is set
+// - empty otherwise
+include::br2-external.txt[]
diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
index b26f912..2e711b8 100644
--- a/docs/manual/manual.mk
+++ b/docs/manual/manual.mk
@@ -10,14 +10,32 @@ MANUAL_RESOURCES = $(TOPDIR)/docs/images
 # Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
 MANUAL_DEPENDENCIES += prepare-kconfig
 
+# If using one or more BR2_EXTERNAL trees, include the list of
+# packages therein, otherwise, generate an empty file
+ifeq ($(BR2_EXTERNAL),)
+define MANUAL_GEN_LISTS_BR2_EXTERNAL
+	$(Q)printf "" >$(@D)/br2-external.txt
+endef
+else
+define MANUAL_GEN_LISTS_BR2_EXTERNAL
+	$(Q)( \
+		printf "== List of BR2_EXTERNAL packages\n\n"; \
+		printf "include::br2-external-package-list.txt[]\n"; \
+	) >$(@D)/br2-external.txt
+endef
+MANUAL_GEN_LISTS_ENV = BR2_EXTERNAL="$(BR2_EXTERNAL)"
+endif
+
 # Our manual needs to generate lists
 define MANUAL_GEN_LISTS
 	$(Q)$(call MESSAGE,"Updating the manual lists...")
 	$(Q)$(COMMON_CONFIG_ENV) \
+		$(MANUAL_GEN_LISTS_ENV) \
 		BR2_DEFCONFIG="" \
 		TOPDIR=$(TOPDIR) \
 		O=$(@D) \
 		python -B $(TOPDIR)/support/scripts/gen-manual-lists.py
+	$(MANUAL_GEN_LISTS_BR2_EXTERNAL)
 endef
 MANUAL_POST_RSYNC_HOOKS += MANUAL_GEN_LISTS
 
-- 
2.7.4

[Buildroot] [PATCH 16/16 v3] docs/manual: document multi br2-external

[Yann E. MORIN]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 docs/manual/customize-outside-br.txt | 62 ++++++++++++++++++++++++------------
 1 file changed, 41 insertions(+), 21 deletions(-)

diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
index 839d1b1..ed4a949 100644
--- a/docs/manual/customize-outside-br.txt
+++ b/docs/manual/customize-outside-br.txt
@@ -17,19 +17,18 @@ place project-specific customizations in two locations:
    having them nicely integrated in the build logic. This section
    explains how to use +BR2_EXTERNAL+.
 
-+BR2_EXTERNAL+ is an environment variable that can be used to point to
-a directory that contains Buildroot customizations. It can be passed
-to any Buildroot +make+ invocation. It is automatically saved in the
-hidden +.br-external+ file in the output directory. Thanks to this,
-there is no need to pass +BR2_EXTERNAL+ at every +make+ invocation. It
-can however be changed at any time by passing a new value, and can be
++BR2_EXTERNAL+ is an environment variable that can be set to contain the
+paths to one or more directories that contains Buildroot customizations.
+It can be passed to any Buildroot +make+ invocation. It is automatically
+saved in the hidden +.br-external+ file in the output directory. Thanks
+to this, there is no need to pass +BR2_EXTERNAL+ at every +make+ invocation.
+It can however be changed at any time by passing a new value, and can be
 removed by passing an empty value.
 
 .Note
-The +BR2_EXTERNAL+ path can be either an absolute or a relative path,
-but if it's passed as a relative path, it is important to note that it
-is interpreted relative to the main Buildroot source directory, *not*
-to the Buildroot output directory.
+Paths in +BR2_EXTERNAL+ can be either absolute or relative paths, but it
+is important to note that relative paths are interpreted relative to the
+main Buildroot source directory, *not* to the Buildroot output directory.
 
 Some examples:
 
@@ -51,6 +50,12 @@ We can switch to another external definitions directory at any time:
 buildroot/ $ make BR2_EXTERNAL=/where/we/have/barfoo xconfig
 -----
 
+We can also use multiple +BR2_EXTERNAL+ locations:
+
+----
+buildroot/ $ make BR2_EXTERNAL="/path/to/foobar /where/we/have/barfoo" menuconfig
+----
+
 Or disable the usage of external definitions:
 
 -----
@@ -60,13 +65,23 @@ buildroot/ $ make BR2_EXTERNAL= xconfig
 A +BR2_EXTERNAL+ tree must contain at least those three files:
 
 +external.id+::
-   That file should contain the ID for the +BR2_EXTERNAL+ tree.
+   That file should contain the ID for that +BR2_EXTERNAL+ tree.
    That ID is used to construct the +BR2_EXTERNAL_$(ID)+ variable,
    available in +Config.in+ and +external.mk+ (see below), so that ID
    must be a valid make and Kconfig variable. Buildroot sets
    +BR2_EXTERNAL_$(ID)+ to the path of the +BR2_EXTERNAL+ tree.
    The format for this file is a single line with the ID.
 +
+.Note:
+Since it is possible to use multiple +BR2_EXTERNAL+ trees at once, this
+  ID is used by Buildroot to generate variables for each of those trees.
+  That ID is used to identify your +BR2_EXTERNAL+ tree, so try to come up
+  with an ID that really describes your +BR2_EXTERNAL+ tree, in order for
+  it to be relatively unique, so that it does not clash with another ID
+  from another +BR2_EXTERNAL+ tree, especially if you are planning on
+  somehow sharing your +BR2_EXTERNAL+ tree with third parties or using
+  +BR2_EXTERNAL+ trees from third parties.
++
 Examples of an +external.mk+ file that declares the ID +FOO+:
 +
 ----
@@ -102,22 +117,22 @@ Using +BR2_EXTERNAL+ then allows three different things:
    +$(BR2_EXTERNAL_BAR_42)/board/<boardname>/kernel.config+ (to specify
    the location of the kernel configuration file).
 
- * One can store package recipes (i.e. +Config.in+ and
-   +<packagename>.mk+), or even custom configuration options and make
-   logic. Buildroot automatically includes +$(BR2_EXTERNAL)/Config.in+ to
-   make it appear in the top-level configuration menu, and includes
-   +$(BR2_EXTERNAL)/external.mk+ with the rest of the makefile logic.
+ * One can store package recipes (i.e. +Config.in+ and +<packagename>.mk+),
+   or even custom configuration options and make logic. Buildroot
+   automatically includes the +Config.in+ from each paths in +BR2_EXTERNAL+
+   to make it appear in a sub-menu labelled _User-provided options_ at the
+   root of the main configuration menu, and includes the +external.mk+ from
+   each paths in +BR2_EXTERNAL+ with the rest of the makefile logic.
 +
 The main usage of this is to store package recipes. The recommended
-   way to do this is to write a +$(BR2_EXTERNAL)/Config.in+ file that
-   looks like:
+   way to do this is to write +Config.in+ files that look like:
 +
 ------
 source "$BR2_EXTERNAL_BAR_42/package/package1/Config.in"
 source "$BR2_EXTERNAL_BAR_42/package/package2/Config.in"
 ------
 +
-Then, have a +$(BR2_EXTERNAL)/external.mk+ file that looks like:
+Then, write +/external.mk+ files that look like:
 +
 ------
 include $(sort $(wildcard $(BR2_EXTERNAL_BAR_42)/package/*/*.mk))
@@ -130,7 +145,12 @@ And then in +$(BR2_EXTERNAL_FOO_42)/package/package1+ and
    called <boardname> and adapt the above paths accordingly.
 
  * One can store Buildroot defconfigs in the +configs+ subdirectory of
-   +$(BR2_EXTERNAL)+. Buildroot will automatically show them in the
+   a +BR2_EXTERNAL+ tree. Buildroot will automatically show them in the
    output of +make list-defconfigs+ and allow them to be loaded with the
    normal +make <name>_defconfig+ command. They will be visible under the
-   +User-provided configs+' label in the 'make list-defconfigs' output.
+   +User-provided configs+ label in the 'make list-defconfigs' output.
++
+.Note
+If a defconfig file is present in more than one +BR2_EXTERNAL+ location,
+   then the last one is used. It is also possible to override a defconfig
+   bundled in Buildroot.
-- 
2.7.4

[Buildroot] [PATCH 12/16 v3] core: handle .br-external in a make rule

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> Currently, we treat the case where we have no br2-external tree
> (BR2_EXTERNAL is empty) differently from the case where we do have
> one (BR2_EXTERNAL is not empty).
> 
> There is now no reason to treat those two cases differently:
> 
>   - the kconfig snippet is always generated appropriately (i.e. it would
>     include the br2-external tree if set, or include nothing otherwise);
> 
>   - we no longer have a dummy br-external tree either.
> 
> As such, generate the .br-external cache file in both cases.
> 
> This will make it much easier to handle when we introduce support for
> multiple br2-external trees.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> ---
>  Makefile | 13 ++++++-------
>  1 file changed, 6 insertions(+), 7 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 72b55cb..67334e5 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -146,23 +146,17 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
>  # On subsequent invocations of make, it is read in. It can still be overridden
>  # on the command line, therefore the file is re-created every time make is run.
>  #
> -# When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
> -# line), the .br-external file is removed.
> -#
>  # If the br2-external tree defines its ID, then export the path in the
>  # BR2_EXTERNAL_$(ID) variable.
>  
>  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
>  -include $(BR2_EXTERNAL_FILE)
> -ifeq ($(BR2_EXTERNAL),)
> -  $(shell rm -f $(BR2_EXTERNAL_FILE))
> -else
> +ifneq ($(BR2_EXTERNAL),)
>    _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
>    ifeq ($(_BR2_EXTERNAL),)
>      $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' does not exist, relative to $(TOPDIR))
>    endif
>    override BR2_EXTERNAL := $(_BR2_EXTERNAL)
> -  $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
>    ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
>      BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
>      ifeq ($(BR2_EXTERNAL_ID),)
> @@ -173,6 +167,11 @@ else
>    BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
>  endif
>  
> +# This needs to be *after* we compute BR_EXTERNAL, above.
> +.PHONY: $(BR2_EXTERNAL_FILE)
> +$(BR2_EXTERNAL_FILE):
> +	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@

s/BR_EXTERNAL/BR2_EXTERNAL/

It's probably a rebase issue since it's fixed in the following patch ;-)

Best regards,
Romain


> +
>  # To make sure that the environment variable overrides the .config option,
>  # set this before including .config.
>  ifneq ($(BR2_DL_DIR),)
> 

[Buildroot] [PATCH 13/16 v3] core: add support for multiple br2-external trees

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> Currently, we only support at most one br2-external tree. Being able
> to use more than one br2-external tree can be very useful.
> 
> A use-case would be for having a br2-external to contain the basic
> packages, basic board defconfigs and board files, provided by one team
> responsible for the "board-bringup", while other teams consume that
> br2-external as a base, and complements it each with their own set of
> packages, defconfigs and extra board files.
> 
> Another use-case would be for third-parties to provide their own
> Buildroot packaging in a br2-external tree, along-side the archives for
> their stuff.
> 
> Finally, another use-case is to be able to add FLOSS packages in a
> br2-external tree, and proprietary packages in another. This allows
> to not touch the Buildroot tree at all, and still be able to get in
> compliance by providing only that br2-external tree(s) that contains
> FLOSS packages, leaving aside the br2-external tree(s) with the
> proprietary bits.
> 
> What we do is to treat BR2_EXTERNAL as a space-separated list of paths,
> which we iterate to construct:
> 
>   - the list of all br2-external IDs, BR_EXTERNAL_IDS
                                          ^
There are several occurrence of BR_EXTERNAL_IDS instead of BR2_EXTERNAL_IDS.

Did you named BR_EXTERNAL_IDS to avoid confusion between BR2_EXTERNAL_ID and
BR2_EXTERNAL_IDS.

Ha ok, BR_ variables are Buildroot internal variables not saved in .config file,
right ?

Best regards,
Romain

> 
>   - the per-br2-external tree BR2_EXTERNAL_$(ID) variables, which points
>     to the actual location of the corresponding tree,
> 
>   - EXTRA_ENV which now needs to contain all BR2_EXTERNAL_$(ID)
>     variables, and thus needs to be set early.
> 
> Once we have all those variables, we replace references to BR2_EXTERNAL
> with either:
> 
>   - a $(patsubst ...) to include the external.mk files (and docs),
>     or
> 
>   - a $(foreach ...) iteration to generate other Makefile code.
> 
> To be noted: we also use $(foreach ...) in the kconfig-snippet
> generating rule, because it is cleaner than doing so in the shell
> 
> Finally, we double-quote $(BR2_EXTERNAL) when assigning it to
> environment variables.
> 
> Now, when more than one br2-external tree is used, each gets its own
> sub-menu in the "User-provided options" menu. The sub-menu is labelled
> with that br2-external tree's ID (prefixed with 'BR2_EXTERNAL_') and the
> sub-menu's first item is a comment with the path to that br2-external
> tree.
> 
> If there's only one br2-external tree, then there is no sub-menu, but
> we still add the comment with the path to the br2-external tree (which
> we did not have previously). If the br2-external tree has an ID, it is
> displayed as well as a comment.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> 
> ---
> Changes v3 -> v4:
>   - drop check for file existence  (Thomas)
>   - fix check for empty ID  (Thomas)
>   - simpler code to check number of br2-external trees
> ---
>  Makefile            | 99 +++++++++++++++++++++++++++++++++++++----------------
>  package/Makefile.in |  2 +-
>  2 files changed, 70 insertions(+), 31 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 67334e5..7b3542c 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -145,32 +145,57 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
>  # The value of BR2_EXTERNAL is stored in .br-external in the output directory.
>  # On subsequent invocations of make, it is read in. It can still be overridden
>  # on the command line, therefore the file is re-created every time make is run.
> -#
> -# If the br2-external tree defines its ID, then export the path in the
> -# BR2_EXTERNAL_$(ID) variable.
>  
>  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
>  -include $(BR2_EXTERNAL_FILE)
> -ifneq ($(BR2_EXTERNAL),)
> -  _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
> -  ifeq ($(_BR2_EXTERNAL),)
> -    $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' does not exist, relative to $(TOPDIR))
> -  endif
> -  override BR2_EXTERNAL := $(_BR2_EXTERNAL)
> -  ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
> -    BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
> -    ifeq ($(BR2_EXTERNAL_ID),)
> -      $(error $(BR2_EXTERNAL) has no ID (in file 'external.id'))
> -    endif
> -    BR2_EXTERNAL_$(BR2_EXTERNAL_ID) = $(BR2_EXTERNAL)
> -  endif
> -  BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
> -endif
>  
> -# This needs to be *after* we compute BR_EXTERNAL, above.
>  .PHONY: $(BR2_EXTERNAL_FILE)
>  $(BR2_EXTERNAL_FILE):
> -	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@
> +	@echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) >$@
> +
> +# Those two variables need to be defined as simply-expanded variables, they
> +# can't be recursively-expanded, because the values they are assigned change
> +# with each iteration of the foreach, below.
> +BR_EXTERNAL_IDS :=
> +EXTRA_ENV :=
> +
> +# If there is no or one br2-external trees used, then we don't require (but
> +# accept) an ID; otherwise (i.e. there are two or more br2-external trees)
> +# we require they do all define their ID.
> +ifeq ($(filter-out 0 1,$(words $(BR2_EXTERNAL))),)
> +BR2_EXTERNAL_NEED_ID := loose
> +else
> +BR2_EXTERNAL_NEED_ID := strict
> +endif
> +
> +# Validate the br2-external tree passed as $(1):
> +# - check the directory actually exists
> +# - check if we need and have a non-empty ID
> +# - check the ID is not a duplicate
> +# - set variables for later use
> +define BR2_EXTERNAL_VALIDATE
> +  _BR_EXT_DIR := $$(shell cd $(1) >/dev/null 2>&1 && pwd)
> +  ifeq ($$(_BR_EXT_DIR),)
> +    $$(error BR2_EXTERNAL='$(1)' does not exist, relative to $$(TOPDIR))
> +  endif
> +  _BR_EXT_ID := $$(shell cat $$(_BR_EXT_DIR)/external.id 2>/dev/null)
> +  ifeq ($$(_BR_EXT_ID),)
> +    ifeq ($(BR2_EXTERNAL_NEED_ID),strict)
> +      $$(error BR2_EXTERNAL='$(1)' has no ID (in file 'external.id'),\
> +               mandatory to use more than one br2-external tree at once)
> +    endif # BR2_EXTERNAL_NEED_ID strict
> +  endif # No ID
> +  ifneq ($$(filter $$(_BR_EXT_ID),$$(BR_EXTERNAL_IDS)),)
> +    $$(error Duplicate ID '$$(_BR_EXT_ID)' in '$(1)', previously defined in '$$(BR2_EXTERNAL_$$(_BR_EXT_ID))')
> +  endif
> +  ifneq ($$(_BR_EXT_ID),)
> +    BR2_EXTERNAL_$$(_BR_EXT_ID) := $$(_BR_EXT_DIR)
> +    BR_EXTERNAL_IDS += $$(_BR_EXT_ID)
> +    EXTRA_ENV += BR2_EXTERNAL_$$(_BR_EXT_ID)=$$(_BR_EXT_DIR)
> +  endif # _BR_EXT_ID not empty
> +endef # BR2_EXTERNAL_VALIDATE
> +
> +$(eval $(foreach d,$(BR2_EXTERNAL),$(call BR2_EXTERNAL_VALIDATE,$(d))$(sep)))
>  
>  # To make sure that the environment variable overrides the .config option,
>  # set this before including .config.
> @@ -469,9 +494,10 @@ include fs/common.mk
>  # If the br2-external tree defines its ID, then the BR2_EXTERNAL_$(ID)
>  # variable is also present in .config, so it is quoted. We must unquote
>  # it before feeding it to the br2-external makefile.
> -BR2_EXTERNAL_$(BR2_EXTERNAL_ID) := $(call qstrip,$(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)))
> +$(eval $(foreach id,$(BR_EXTERNAL_IDS),\
> +	BR2_EXTERNAL_$(id) := $(call qstrip,$(BR2_EXTERNAL_$(id)))$(sep)))
>  # Nothing to include if no BR2_EXTERNAL tree in use
> -include $(BR2_EXTERNAL_MK)
> +include $(patsubst %,%/external.mk,$(BR2_EXTERNAL))
>  
>  # Now we are sure we have all the packages scanned and defined. We now
>  # check for each package in the list of enabled packages, that all its
> @@ -787,7 +813,7 @@ COMMON_CONFIG_ENV = \
>  	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
>  	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
>  	BR2_CONFIG=$(BR2_CONFIG) \
> -	BR2_EXTERNAL=$(BR2_EXTERNAL) \
> +	BR2_EXTERNAL="$(BR2_EXTERNAL)" \
>  	HOST_GCC_VERSION="$(HOSTCC_VERSION)" \
>  	BUILD_DIR=$(BUILD_DIR) \
>  	SKIP_LEGACY=
> @@ -895,12 +921,25 @@ $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
>  	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
>  		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
>  		printf 'menu "User-provided options"\n\n'; \
> -		if [ -z "$(BR2_EXTERNAL_ID)" ]; then \
> +		if [ -z "$(call strip,$(BR_EXTERNAL_IDS))" ]; then \
> +			printf 'comment "%s"\n\n' $(BR2_EXTERNAL); \
>  			printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
>  		else \
> -			printf 'config BR2_EXTERNAL_%s\n' $(BR2_EXTERNAL_ID); \
> -			printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \
> -			printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(BR2_EXTERNAL_ID); \
> +			$(foreach id,$(BR_EXTERNAL_IDS),\
> +				for i in $$(seq 1 80); do printf '#'; done; printf '\n\n'; \
> +				if [ $(words $(call strip,$(BR_EXTERNAL_IDS))) -gt 1 ]; then \
> +					printf 'menu "BR2_EXTERNAL_%s"\n\n' $(id); \
> +				else \
> +					printf 'comment "BR2_EXTERNAL_%s"\n\n' $(id); \
> +				fi; \
> +				printf 'comment "%s"\n\n' $(BR2_EXTERNAL_$(id)); \
> +				printf 'config BR2_EXTERNAL_%s\n' $(id); \
> +				printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(id)); \
> +				printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(id); \
> +				if [ $(words $(call strip,$(BR_EXTERNAL_IDS))) -gt 1 ]; then \
> +					printf 'endmenu # BR2_EXTERNAL_%s\n\n' $(id); \
> +				fi; ) \
> +			for i in $$(seq 1 80); do printf '#'; done; printf '\n\n'; \
>  		fi; \
>  		printf 'endmenu # User-provided options\n'; \
>  	fi >$@
> @@ -1004,10 +1043,10 @@ list-defconfigs:
>  	@echo 'Built-in configs:'
>  	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
>  	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
> -ifneq ($(wildcard $(BR2_EXTERNAL)/configs/*_defconfig),)
> +ifneq ($(wildcard $(patsubst %,%/configs/*_defconfig,$(BR2_EXTERNAL))),)
>  	@echo
>  	@echo 'User-provided configs:'
> -	@$(foreach b, $(sort $(notdir $(wildcard $(BR2_EXTERNAL)/configs/*_defconfig))), \
> +	@$(foreach b, $(sort $(notdir $(wildcard $(patsubst %,%/configs/*_defconfig,$(BR2_EXTERNAL))))), \
>  	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
>  endif
>  	@echo
> @@ -1029,7 +1068,7 @@ print-version:
>  	@echo $(BR2_VERSION_FULL)
>  
>  include docs/manual/manual.mk
> --include $(BR2_EXTERNAL)/docs/*/*.mk
> +-include $(patsubst %,%/docs/*/*.mk,$(BR2_EXTERNAL))
>  
>  .PHONY: $(noconfig_targets)
>  
> diff --git a/package/Makefile.in b/package/Makefile.in
> index afd5d3a..fe8ee68 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -319,7 +319,7 @@ EXTRA_ENV = \
>  	PATH=$(BR_PATH) \
>  	BR2_DL_DIR=$(BR2_DL_DIR) \
>  	BUILD_DIR=$(BUILD_DIR) \
> -	BR2_EXTERNAL=$(BR2_EXTERNAL)
> +	BR2_EXTERNAL="$(BR2_EXTERNAL)"
>  
>  ################################################################################
>  # settings we need to pass to configure
> 

[Buildroot] [PATCH 01/16 v3] core: move pkg-utils.mk to support/

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> pkg-utils.mk contains various definitions that are used in the package
> infrastructures and packages themselves.
> 
> However, those definitions can be useful in other parts of Buildroot,
> and are already used in a few places that are not related to the package
> infrastructure. Also, $(sep) will be needed early in the Makefile when
> we eventually support multiple br2-external trees.
> 
> Since this file only contains definitions, we can include it anytime.
> 
> So, consider that file to no longer be specific to the package infras:
>   - move it to support and rename it,
>   - move a few similar definitions from the main Makefile to that file.

Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> ---
>  Makefile              |  10 +----
>  package/pkg-utils.mk  |  96 -------------------------------------------
>  support/misc/utils.mk | 111 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 113 insertions(+), 104 deletions(-)
>  create mode 100644 support/misc/utils.mk
> 
> diff --git a/Makefile b/Makefile
> index 027f21c..af2d982 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -106,14 +106,8 @@ else ifneq ($(filter-out $(nobuild_targets),$(MAKECMDGOALS)),)
>  BR_BUILDING = y
>  endif
>  
> -# Strip quotes and then whitespaces
> -qstrip = $(strip $(subst ",,$(1)))
> -#"))
> -
> -# Variables for use in Make constructs
> -comma := ,
> -empty :=
> -space := $(empty) $(empty)
> +# Include some helper macros and variables
> +include support/misc/utils.mk
>  
>  ifneq ("$(origin O)", "command line")
>  O := output
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index c61b3b6..28db481 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -5,43 +5,6 @@
>  #
>  ################################################################################
>  
> -# Case conversion macros. This is inspired by the 'up' macro from gmsl
> -# (http://gmsl.sf.net). It is optimised very heavily because these macros
> -# are used a lot. It is about 5 times faster than forking a shell and tr.
> -#
> -# The caseconvert-helper creates a definition of the case conversion macro.
> -# After expansion by the outer $(eval ), the UPPERCASE macro is defined as:
> -# $(strip $(eval __tmp := $(1))  $(eval __tmp := $(subst a,A,$(__tmp))) ... )
> -# In other words, every letter is substituted one by one.
> -#
> -# The caseconvert-helper allows us to create this definition out of the
> -# [FROM] and [TO] lists, so we don't need to write down every substition
> -# manually. The uses of $ and $$ quoting are chosen in order to do as
> -# much expansion as possible up-front.
> -#
> -# Note that it would be possible to conceive a slightly more optimal
> -# implementation that avoids the use of __tmp, but that would be even
> -# more unreadable and is not worth the effort.
> -
> -[FROM] := a b c d e f g h i j k l m n o p q r s t u v w x y z - .
> -[TO]   := A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ _
> -
> -define caseconvert-helper
> -$(1) = $$(strip \
> -	$$(eval __tmp := $$(1))\
> -	$(foreach c, $(2),\
> -		$$(eval __tmp := $$(subst $(word 1,$(subst :, ,$c)),$(word 2,$(subst :, ,$c)),$$(__tmp))))\
> -	$$(__tmp))
> -endef
> -
> -$(eval $(call caseconvert-helper,UPPERCASE,$(join $(addsuffix :,$([FROM])),$([TO]))))
> -$(eval $(call caseconvert-helper,LOWERCASE,$(join $(addsuffix :,$([TO])),$([FROM]))))
> -
> -# Sanitize macro cleans up generic strings so it can be used as a filename
> -# and in rules. Particularly useful for VCS version strings, that can contain
> -# slashes, colons (OK in filenames but not in rules), and spaces.
> -sanitize = $(subst $(space),_,$(subst :,_,$(subst /,_,$(strip $(1)))))
> -
>  #
>  # Manipulation of .config files based on the Kconfig
>  # infrastructure. Used by the BusyBox package, the Linux kernel
> @@ -82,65 +45,6 @@ INFLATE.tar  = cat
>  # suitable-extractor(filename): returns extractor based on suffix
>  suitable-extractor = $(INFLATE$(suffix $(1)))
>  
> -# MESSAGE Macro -- display a message in bold type
> -MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG)_NAME) $($(PKG)_VERSION) $(call qstrip,$(1))$(TERM_RESET)"
> -TERM_BOLD := $(shell tput smso 2>/dev/null)
> -TERM_RESET := $(shell tput rmso 2>/dev/null)
> -
> -# Utility functions for 'find'
> -# findfileclauses(filelist) => -name 'X' -o -name 'Y'
> -findfileclauses = $(call notfirstword,$(patsubst %,-o -name '%',$(1)))
> -# finddirclauses(base, dirlist) => -path 'base/dirX' -o -path 'base/dirY'
> -finddirclauses = $(call notfirstword,$(patsubst %,-o -path '$(1)/%',$(2)))
> -
> -# Miscellaneous utility functions
> -# notfirstword(wordlist): returns all but the first word in wordlist
> -notfirstword = $(wordlist 2,$(words $(1)),$(1))
> -
> -# Needed for the foreach loops to loop over the list of hooks, so that
> -# each hook call is properly separated by a newline.
> -define sep
> -
> -
> -endef
> -
> -PERCENT = %
> -QUOTE = '
> -# ' # Meh... syntax-highlighting
> -
> -# This macro properly escapes a command string, then prints it with printf:
> -#
> -#   - first, backslash '\' are self-escaped, so that they do not escape
> -#     the following char and so that printf properly outputs a backslash;
> -#
> -#   - next, single quotes are escaped by closing an existing one, adding
> -#     an escaped one, and re-openning a new one (see below for the reason);
> -#
> -#   - then '%' signs are self-escaped so that the printf does not interpret
> -#     them as a format specifier, in case the variable contains an actual
> -#     printf with a format;
> -#
> -#   - finally, $(sep) is replaced with the literal '\n' so that make does
> -#     not break on the so-expanded variable, but so that the printf does
> -#     correctly output an LF.
> -#
> -# Note: this must be escaped in this order to avoid over-escaping the
> -# previously escaped elements.
> -#
> -# Once everything has been escaped, it is passed between single quotes
> -# (that's why the single-quotes are escaped they way they are, above,
> -# and why the dollar sign is not escaped) to printf(1). A trailing
> -# newline is apended, too.
> -#
> -# Note: leading or trailing spaces are *not* stripped.
> -#
> -define PRINTF
> -	printf '$(subst $(sep),\n,\
> -			$(subst $(PERCENT),$(PERCENT)$(PERCENT),\
> -				$(subst $(QUOTE),$(QUOTE)\$(QUOTE)$(QUOTE),\
> -					$(subst \,\\,$(1)))))\n'
> -endef
> -
>  # check-deprecated-variable -- throw an error on deprecated variables
>  # example:
>  #   $(eval $(call check-deprecated-variable,FOO_MAKE_OPT,FOO_MAKE_OPTS))
> diff --git a/support/misc/utils.mk b/support/misc/utils.mk
> new file mode 100644
> index 0000000..990a3d1
> --- /dev/null
> +++ b/support/misc/utils.mk
> @@ -0,0 +1,111 @@
> +################################################################################
> +#
> +# This file contains various utility macros and variables used about
> +# everywhere in make constructs.
> +#
> +################################################################################
> +
> +# Strip quotes and then whitespaces
> +qstrip = $(strip $(subst ",,$(1)))
> +#"))
> +
> +# Variables for use in Make constructs
> +comma := ,
> +empty :=
> +space := $(empty) $(empty)
> +
> +# Case conversion macros. This is inspired by the 'up' macro from gmsl
> +# (http://gmsl.sf.net). It is optimised very heavily because these macros
> +# are used a lot. It is about 5 times faster than forking a shell and tr.
> +#
> +# The caseconvert-helper creates a definition of the case conversion macro.
> +# After expansion by the outer $(eval ), the UPPERCASE macro is defined as:
> +# $(strip $(eval __tmp := $(1))  $(eval __tmp := $(subst a,A,$(__tmp))) ... )
> +# In other words, every letter is substituted one by one.
> +#
> +# The caseconvert-helper allows us to create this definition out of the
> +# [FROM] and [TO] lists, so we don't need to write down every substition
> +# manually. The uses of $ and $$ quoting are chosen in order to do as
> +# much expansion as possible up-front.
> +#
> +# Note that it would be possible to conceive a slightly more optimal
> +# implementation that avoids the use of __tmp, but that would be even
> +# more unreadable and is not worth the effort.
> +
> +[FROM] := a b c d e f g h i j k l m n o p q r s t u v w x y z - .
> +[TO]   := A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ _
> +
> +define caseconvert-helper
> +$(1) = $$(strip \
> +	$$(eval __tmp := $$(1))\
> +	$(foreach c, $(2),\
> +		$$(eval __tmp := $$(subst $(word 1,$(subst :, ,$c)),$(word 2,$(subst :, ,$c)),$$(__tmp))))\
> +	$$(__tmp))
> +endef
> +
> +$(eval $(call caseconvert-helper,UPPERCASE,$(join $(addsuffix :,$([FROM])),$([TO]))))
> +$(eval $(call caseconvert-helper,LOWERCASE,$(join $(addsuffix :,$([TO])),$([FROM]))))
> +
> +# Sanitize macro cleans up generic strings so it can be used as a filename
> +# and in rules. Particularly useful for VCS version strings, that can contain
> +# slashes, colons (OK in filenames but not in rules), and spaces.
> +sanitize = $(subst $(space),_,$(subst :,_,$(subst /,_,$(strip $(1)))))
> +
> +# MESSAGE Macro -- display a message in bold type
> +MESSAGE = echo "$(TERM_BOLD)>>> $($(PKG)_NAME) $($(PKG)_VERSION) $(call qstrip,$(1))$(TERM_RESET)"
> +TERM_BOLD := $(shell tput smso 2>/dev/null)
> +TERM_RESET := $(shell tput rmso 2>/dev/null)
> +
> +# Utility functions for 'find'
> +# findfileclauses(filelist) => -name 'X' -o -name 'Y'
> +findfileclauses = $(call notfirstword,$(patsubst %,-o -name '%',$(1)))
> +# finddirclauses(base, dirlist) => -path 'base/dirX' -o -path 'base/dirY'
> +finddirclauses = $(call notfirstword,$(patsubst %,-o -path '$(1)/%',$(2)))
> +
> +# Miscellaneous utility functions
> +# notfirstword(wordlist): returns all but the first word in wordlist
> +notfirstword = $(wordlist 2,$(words $(1)),$(1))
> +
> +# Needed for the foreach loops to loop over the list of hooks, so that
> +# each hook call is properly separated by a newline.
> +define sep
> +
> +
> +endef
> +
> +PERCENT = %
> +QUOTE = '
> +# ' # Meh... syntax-highlighting
> +
> +# This macro properly escapes a command string, then prints it with printf:
> +#
> +#   - first, backslash '\' are self-escaped, so that they do not escape
> +#     the following char and so that printf properly outputs a backslash;
> +#
> +#   - next, single quotes are escaped by closing an existing one, adding
> +#     an escaped one, and re-openning a new one (see below for the reason);
> +#
> +#   - then '%' signs are self-escaped so that the printf does not interpret
> +#     them as a format specifier, in case the variable contains an actual
> +#     printf with a format;
> +#
> +#   - finally, $(sep) is replaced with the literal '\n' so that make does
> +#     not break on the so-expanded variable, but so that the printf does
> +#     correctly output an LF.
> +#
> +# Note: this must be escaped in this order to avoid over-escaping the
> +# previously escaped elements.
> +#
> +# Once everything has been escaped, it is passed between single quotes
> +# (that's why the single-quotes are escaped they way they are, above,
> +# and why the dollar sign is not escaped) to printf(1). A trailing
> +# newline is apended, too.
> +#
> +# Note: leading or trailing spaces are *not* stripped.
> +#
> +define PRINTF
> +	printf '$(subst $(sep),\n,\
> +		$(subst $(PERCENT),$(PERCENT)$(PERCENT),\
> +			$(subst $(QUOTE),$(QUOTE)\$(QUOTE)$(QUOTE),\
> +				$(subst \,\\,$(1)))))\n'
> +endef
> 

[Buildroot] [PATCH 12/16 v3] core: handle .br-external in a make rule

[Yann E. MORIN]

Romain, All,

On 2016-07-24 22:05 +0200, Romain Naour spake thusly:
> Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> > Currently, we treat the case where we have no br2-external tree
> > (BR2_EXTERNAL is empty) differently from the case where we do have
> > one (BR2_EXTERNAL is not empty).
> > 
> > There is now no reason to treat those two cases differently:
> > 
> >   - the kconfig snippet is always generated appropriately (i.e. it would
> >     include the br2-external tree if set, or include nothing otherwise);
> > 
> >   - we no longer have a dummy br-external tree either.
> > 
> > As such, generate the .br-external cache file in both cases.
> > 
> > This will make it much easier to handle when we introduce support for
> > multiple br2-external trees.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > Cc: Peter Korsgaard <jacmet@uclibc.org>
> > Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
> > ---
> >  Makefile | 13 ++++++-------
> >  1 file changed, 6 insertions(+), 7 deletions(-)
> > 
> > diff --git a/Makefile b/Makefile
> > index 72b55cb..67334e5 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -146,23 +146,17 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
> >  # On subsequent invocations of make, it is read in. It can still be overridden
> >  # on the command line, therefore the file is re-created every time make is run.
> >  #
> > -# When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
> > -# line), the .br-external file is removed.
> > -#
> >  # If the br2-external tree defines its ID, then export the path in the
> >  # BR2_EXTERNAL_$(ID) variable.
> >  
> >  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
> >  -include $(BR2_EXTERNAL_FILE)
> > -ifeq ($(BR2_EXTERNAL),)
> > -  $(shell rm -f $(BR2_EXTERNAL_FILE))
> > -else
> > +ifneq ($(BR2_EXTERNAL),)
> >    _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
> >    ifeq ($(_BR2_EXTERNAL),)
> >      $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' does not exist, relative to $(TOPDIR))
> >    endif
> >    override BR2_EXTERNAL := $(_BR2_EXTERNAL)
> > -  $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
> >    ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
> >      BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
> >      ifeq ($(BR2_EXTERNAL_ID),)
> > @@ -173,6 +167,11 @@ else
> >    BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
> >  endif
> >  
> > +# This needs to be *after* we compute BR_EXTERNAL, above.
> > +.PHONY: $(BR2_EXTERNAL_FILE)
> > +$(BR2_EXTERNAL_FILE):
> > +	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@
> 
> s/BR_EXTERNAL/BR2_EXTERNAL/

Right, but it should be: s/BR2_EXTERNAL/_BR2_EXTERNAL/  (i.e. with a
leading underscore) as we want to store the full path, which is
constructed with the first line in the ifdef block.

> It's probably a rebase issue since it's fixed in the following patch ;-)

Probably, probably... ;-)

Thanks for spotting.

Regards,
Yann E. MORIN.


-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 13/16 v3] core: add support for multiple br2-external trees

[Yann E. MORIN]

On 2016-07-24 22:25 +0200, Romain Naour spake thusly:
> Hi Yann,
> 
> Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> > Currently, we only support at most one br2-external tree. Being able
> > to use more than one br2-external tree can be very useful.
> > 
> > A use-case would be for having a br2-external to contain the basic
> > packages, basic board defconfigs and board files, provided by one team
> > responsible for the "board-bringup", while other teams consume that
> > br2-external as a base, and complements it each with their own set of
> > packages, defconfigs and extra board files.
> > 
> > Another use-case would be for third-parties to provide their own
> > Buildroot packaging in a br2-external tree, along-side the archives for
> > their stuff.
> > 
> > Finally, another use-case is to be able to add FLOSS packages in a
> > br2-external tree, and proprietary packages in another. This allows
> > to not touch the Buildroot tree at all, and still be able to get in
> > compliance by providing only that br2-external tree(s) that contains
> > FLOSS packages, leaving aside the br2-external tree(s) with the
> > proprietary bits.
> > 
> > What we do is to treat BR2_EXTERNAL as a space-separated list of paths,
> > which we iterate to construct:
> > 
> >   - the list of all br2-external IDs, BR_EXTERNAL_IDS
>                                           ^
> There are several occurrence of BR_EXTERNAL_IDS instead of BR2_EXTERNAL_IDS.
> 
> Did you named BR_EXTERNAL_IDS to avoid confusion between BR2_EXTERNAL_ID and
> BR2_EXTERNAL_IDS.
> 
> Ha ok, BR_ variables are Buildroot internal variables not saved in .config file,
> right ?

Mostly, yes.

BR2_ variables are user-defined, either variables from the .config (i.e.
from kconfig), variables from the environment, variables from variables
on the command line or variables we pass to the user.

BR_ variables are internal variables, thus-prefixed to avoid
namespace-clashing.

So yes, BR_EXTERNAL_IDS is an internal variable, so is not BR2_-prefixed.

Regards,
Yann E. MORIN.

> Best regards,
> Romain
> 
> > 
> >   - the per-br2-external tree BR2_EXTERNAL_$(ID) variables, which points
> >     to the actual location of the corresponding tree,
> > 
> >   - EXTRA_ENV which now needs to contain all BR2_EXTERNAL_$(ID)
> >     variables, and thus needs to be set early.
> > 
> > Once we have all those variables, we replace references to BR2_EXTERNAL
> > with either:
> > 
> >   - a $(patsubst ...) to include the external.mk files (and docs),
> >     or
> > 
> >   - a $(foreach ...) iteration to generate other Makefile code.
> > 
> > To be noted: we also use $(foreach ...) in the kconfig-snippet
> > generating rule, because it is cleaner than doing so in the shell
> > 
> > Finally, we double-quote $(BR2_EXTERNAL) when assigning it to
> > environment variables.
> > 
> > Now, when more than one br2-external tree is used, each gets its own
> > sub-menu in the "User-provided options" menu. The sub-menu is labelled
> > with that br2-external tree's ID (prefixed with 'BR2_EXTERNAL_') and the
> > sub-menu's first item is a comment with the path to that br2-external
> > tree.
> > 
> > If there's only one br2-external tree, then there is no sub-menu, but
> > we still add the comment with the path to the br2-external tree (which
> > we did not have previously). If the br2-external tree has an ID, it is
> > displayed as well as a comment.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > Cc: Peter Korsgaard <jacmet@uclibc.org>
> > Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
> > 
> > ---
> > Changes v3 -> v4:
> >   - drop check for file existence  (Thomas)
> >   - fix check for empty ID  (Thomas)
> >   - simpler code to check number of br2-external trees
> > ---
> >  Makefile            | 99 +++++++++++++++++++++++++++++++++++++----------------
> >  package/Makefile.in |  2 +-
> >  2 files changed, 70 insertions(+), 31 deletions(-)
> > 
> > diff --git a/Makefile b/Makefile
> > index 67334e5..7b3542c 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -145,32 +145,57 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
> >  # The value of BR2_EXTERNAL is stored in .br-external in the output directory.
> >  # On subsequent invocations of make, it is read in. It can still be overridden
> >  # on the command line, therefore the file is re-created every time make is run.
> > -#
> > -# If the br2-external tree defines its ID, then export the path in the
> > -# BR2_EXTERNAL_$(ID) variable.
> >  
> >  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
> >  -include $(BR2_EXTERNAL_FILE)
> > -ifneq ($(BR2_EXTERNAL),)
> > -  _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
> > -  ifeq ($(_BR2_EXTERNAL),)
> > -    $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' does not exist, relative to $(TOPDIR))
> > -  endif
> > -  override BR2_EXTERNAL := $(_BR2_EXTERNAL)
> > -  ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
> > -    BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
> > -    ifeq ($(BR2_EXTERNAL_ID),)
> > -      $(error $(BR2_EXTERNAL) has no ID (in file 'external.id'))
> > -    endif
> > -    BR2_EXTERNAL_$(BR2_EXTERNAL_ID) = $(BR2_EXTERNAL)
> > -  endif
> > -  BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
> > -endif
> >  
> > -# This needs to be *after* we compute BR_EXTERNAL, above.
> >  .PHONY: $(BR2_EXTERNAL_FILE)
> >  $(BR2_EXTERNAL_FILE):
> > -	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@
> > +	@echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) >$@
> > +
> > +# Those two variables need to be defined as simply-expanded variables, they
> > +# can't be recursively-expanded, because the values they are assigned change
> > +# with each iteration of the foreach, below.
> > +BR_EXTERNAL_IDS :=
> > +EXTRA_ENV :=
> > +
> > +# If there is no or one br2-external trees used, then we don't require (but
> > +# accept) an ID; otherwise (i.e. there are two or more br2-external trees)
> > +# we require they do all define their ID.
> > +ifeq ($(filter-out 0 1,$(words $(BR2_EXTERNAL))),)
> > +BR2_EXTERNAL_NEED_ID := loose
> > +else
> > +BR2_EXTERNAL_NEED_ID := strict
> > +endif
> > +
> > +# Validate the br2-external tree passed as $(1):
> > +# - check the directory actually exists
> > +# - check if we need and have a non-empty ID
> > +# - check the ID is not a duplicate
> > +# - set variables for later use
> > +define BR2_EXTERNAL_VALIDATE
> > +  _BR_EXT_DIR := $$(shell cd $(1) >/dev/null 2>&1 && pwd)
> > +  ifeq ($$(_BR_EXT_DIR),)
> > +    $$(error BR2_EXTERNAL='$(1)' does not exist, relative to $$(TOPDIR))
> > +  endif
> > +  _BR_EXT_ID := $$(shell cat $$(_BR_EXT_DIR)/external.id 2>/dev/null)
> > +  ifeq ($$(_BR_EXT_ID),)
> > +    ifeq ($(BR2_EXTERNAL_NEED_ID),strict)
> > +      $$(error BR2_EXTERNAL='$(1)' has no ID (in file 'external.id'),\
> > +               mandatory to use more than one br2-external tree at once)
> > +    endif # BR2_EXTERNAL_NEED_ID strict
> > +  endif # No ID
> > +  ifneq ($$(filter $$(_BR_EXT_ID),$$(BR_EXTERNAL_IDS)),)
> > +    $$(error Duplicate ID '$$(_BR_EXT_ID)' in '$(1)', previously defined in '$$(BR2_EXTERNAL_$$(_BR_EXT_ID))')
> > +  endif
> > +  ifneq ($$(_BR_EXT_ID),)
> > +    BR2_EXTERNAL_$$(_BR_EXT_ID) := $$(_BR_EXT_DIR)
> > +    BR_EXTERNAL_IDS += $$(_BR_EXT_ID)
> > +    EXTRA_ENV += BR2_EXTERNAL_$$(_BR_EXT_ID)=$$(_BR_EXT_DIR)
> > +  endif # _BR_EXT_ID not empty
> > +endef # BR2_EXTERNAL_VALIDATE
> > +
> > +$(eval $(foreach d,$(BR2_EXTERNAL),$(call BR2_EXTERNAL_VALIDATE,$(d))$(sep)))
> >  
> >  # To make sure that the environment variable overrides the .config option,
> >  # set this before including .config.
> > @@ -469,9 +494,10 @@ include fs/common.mk
> >  # If the br2-external tree defines its ID, then the BR2_EXTERNAL_$(ID)
> >  # variable is also present in .config, so it is quoted. We must unquote
> >  # it before feeding it to the br2-external makefile.
> > -BR2_EXTERNAL_$(BR2_EXTERNAL_ID) := $(call qstrip,$(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)))
> > +$(eval $(foreach id,$(BR_EXTERNAL_IDS),\
> > +	BR2_EXTERNAL_$(id) := $(call qstrip,$(BR2_EXTERNAL_$(id)))$(sep)))
> >  # Nothing to include if no BR2_EXTERNAL tree in use
> > -include $(BR2_EXTERNAL_MK)
> > +include $(patsubst %,%/external.mk,$(BR2_EXTERNAL))
> >  
> >  # Now we are sure we have all the packages scanned and defined. We now
> >  # check for each package in the list of enabled packages, that all its
> > @@ -787,7 +813,7 @@ COMMON_CONFIG_ENV = \
> >  	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
> >  	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
> >  	BR2_CONFIG=$(BR2_CONFIG) \
> > -	BR2_EXTERNAL=$(BR2_EXTERNAL) \
> > +	BR2_EXTERNAL="$(BR2_EXTERNAL)" \
> >  	HOST_GCC_VERSION="$(HOSTCC_VERSION)" \
> >  	BUILD_DIR=$(BUILD_DIR) \
> >  	SKIP_LEGACY=
> > @@ -895,12 +921,25 @@ $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
> >  	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
> >  		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
> >  		printf 'menu "User-provided options"\n\n'; \
> > -		if [ -z "$(BR2_EXTERNAL_ID)" ]; then \
> > +		if [ -z "$(call strip,$(BR_EXTERNAL_IDS))" ]; then \
> > +			printf 'comment "%s"\n\n' $(BR2_EXTERNAL); \
> >  			printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> >  		else \
> > -			printf 'config BR2_EXTERNAL_%s\n' $(BR2_EXTERNAL_ID); \
> > -			printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \
> > -			printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(BR2_EXTERNAL_ID); \
> > +			$(foreach id,$(BR_EXTERNAL_IDS),\
> > +				for i in $$(seq 1 80); do printf '#'; done; printf '\n\n'; \
> > +				if [ $(words $(call strip,$(BR_EXTERNAL_IDS))) -gt 1 ]; then \
> > +					printf 'menu "BR2_EXTERNAL_%s"\n\n' $(id); \
> > +				else \
> > +					printf 'comment "BR2_EXTERNAL_%s"\n\n' $(id); \
> > +				fi; \
> > +				printf 'comment "%s"\n\n' $(BR2_EXTERNAL_$(id)); \
> > +				printf 'config BR2_EXTERNAL_%s\n' $(id); \
> > +				printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(id)); \
> > +				printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(id); \
> > +				if [ $(words $(call strip,$(BR_EXTERNAL_IDS))) -gt 1 ]; then \
> > +					printf 'endmenu # BR2_EXTERNAL_%s\n\n' $(id); \
> > +				fi; ) \
> > +			for i in $$(seq 1 80); do printf '#'; done; printf '\n\n'; \
> >  		fi; \
> >  		printf 'endmenu # User-provided options\n'; \
> >  	fi >$@
> > @@ -1004,10 +1043,10 @@ list-defconfigs:
> >  	@echo 'Built-in configs:'
> >  	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
> >  	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
> > -ifneq ($(wildcard $(BR2_EXTERNAL)/configs/*_defconfig),)
> > +ifneq ($(wildcard $(patsubst %,%/configs/*_defconfig,$(BR2_EXTERNAL))),)
> >  	@echo
> >  	@echo 'User-provided configs:'
> > -	@$(foreach b, $(sort $(notdir $(wildcard $(BR2_EXTERNAL)/configs/*_defconfig))), \
> > +	@$(foreach b, $(sort $(notdir $(wildcard $(patsubst %,%/configs/*_defconfig,$(BR2_EXTERNAL))))), \
> >  	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
> >  endif
> >  	@echo
> > @@ -1029,7 +1068,7 @@ print-version:
> >  	@echo $(BR2_VERSION_FULL)
> >  
> >  include docs/manual/manual.mk
> > --include $(BR2_EXTERNAL)/docs/*/*.mk
> > +-include $(patsubst %,%/docs/*/*.mk,$(BR2_EXTERNAL))
> >  
> >  .PHONY: $(noconfig_targets)
> >  
> > diff --git a/package/Makefile.in b/package/Makefile.in
> > index afd5d3a..fe8ee68 100644
> > --- a/package/Makefile.in
> > +++ b/package/Makefile.in
> > @@ -319,7 +319,7 @@ EXTRA_ENV = \
> >  	PATH=$(BR_PATH) \
> >  	BR2_DL_DIR=$(BR2_DL_DIR) \
> >  	BUILD_DIR=$(BUILD_DIR) \
> > -	BR2_EXTERNAL=$(BR2_EXTERNAL)
> > +	BR2_EXTERNAL="$(BR2_EXTERNAL)"
> >  
> >  ################################################################################
> >  # settings we need to pass to configure
> > 
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 02/16 v3] core: commonalise the bundled and br2-external %_defconfig rules

[Romain Naour]

Hi Yann, All,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> The code for both cases is exactly the same, and only differs in the
> location where defconfig files are looked for.
> 
> We use an intermediate macro to generate the corresponding rules,
> because directly generating the rules is ugly and needs lots of escaping
> and double-dollar-ing for the $(eval ...) and $(foreach ...) calls to
> play nicely together.
> 
> Furthermore, that will be tremendously useful when we support multiple
> br2-external trees.

This patch is similar to the one sent by Sam Bobroff [1]. I reviewed it during
the summer camp but it was not applied since it was in a series we rejected.

Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

[1] http://lists.busybox.net/pipermail/buildroot/2016-June/165382.html

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> 
> ---
> Changes v2 -> v3:
>   - use lower-case macro  (Arnout)
>   - move comment in the generated rule  (Arnout)
> ---
>  Makefile | 15 +++++++--------
>  1 file changed, 7 insertions(+), 8 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index af2d982..7c0dcb1 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -847,14 +847,13 @@ olddefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  	@$(COMMON_CONFIG_ENV) $< --defconfig$(if $(DEFCONFIG),=$(DEFCONFIG)) $(CONFIG_CONFIG_IN)
>  
> -# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
> -%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(TOPDIR)/configs/%_defconfig outputmakefile
> -	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/configs/$@ \
> -		$< --defconfig=$(TOPDIR)/configs/$@ $(CONFIG_CONFIG_IN)
> -
> -%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig outputmakefile
> -	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(BR2_EXTERNAL)/configs/$@ \
> -		$< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
> +define percent_defconfig
> +%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
> +	# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
> +	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \
> +		$$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN)
> +endef
> +$(eval $(foreach d,$(TOPDIR) $(BR2_EXTERNAL),$(call percent_defconfig,$(d))$(sep)))
>  
>  savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  	@$(COMMON_CONFIG_ENV) $< \
> 

[Buildroot] [PATCH 03/16 v3] doc/asciidoc: add possibility to define document dependencies

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> Currently, a document can not have dependencies, except for the purely
> internal ones (like checking asciidoc version, and presence of dblatex).
> 
> For our own manual, this will come in handy when we introduce a
> generated kconfig snippet, so we can actually make the manual depend on
> that snippet being generated first.
> 
> For external documents, it can be used to depend on host-packages if
> need be (e.g. a custom host packages that generates specific media files
> included in the manual).

Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  docs/manual/adding-packages-asciidoc.txt | 3 +++
>  package/doc-asciidoc.mk                  | 2 +-
>  2 files changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/manual/adding-packages-asciidoc.txt b/docs/manual/adding-packages-asciidoc.txt
> index 6e21786..0c10fd1 100644
> --- a/docs/manual/adding-packages-asciidoc.txt
> +++ b/docs/manual/adding-packages-asciidoc.txt
> @@ -63,6 +63,9 @@ information is (assuming the document name is +foo+) :
>    to one or more directories containing so-called resources (like CSS or
>    images). By default, empty.
>  
> +* +FOO_DEPENDENCIES+, optional, the list of packages (most probably,
> +  host-packages) that must be built before building this document.
> +
>  There are also additional hooks (see xref:hooks[] for general information
>  on hooks), that a document may set to define extra actions to be done at
>  various steps:
> diff --git a/package/doc-asciidoc.mk b/package/doc-asciidoc.mk
> index 6ab5ad2..ede6ed1 100644
> --- a/package/doc-asciidoc.mk
> +++ b/package/doc-asciidoc.mk
> @@ -140,7 +140,7 @@ endef
>  ################################################################################
>  define ASCIIDOC
>  # Single line, because splitting a foreach is not easy...
> -$(1)-check-dependencies: asciidoc-check-dependencies
> +$(1)-check-dependencies: asciidoc-check-dependencies $$($(2)_DEPENDENCIES)
>  	$$(Q)$$(foreach hook,$$($(2)_CHECK_DEPENDENCIES_HOOKS),$$(call $$(hook))$$(sep))
>  
>  # Single line, because splitting a foreach is not easy...
> 

[Buildroot] [PATCH 04/16 v3] core: introduce an intermediate rule before the configurators

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> Currently, all configurators depend on generating the out-of-tree
> Makefile wrapper.
> 
> In an upcoming patch, we'll need to also generate a kconfig fragment,
> so it will have to kick in before we run the configurators.
> 
> Introduce a new intermediate "prepare-kconfig" rule, so we can
> commonalise the dependencies of the configurators. Move the dependency
> on the Makefile wrapper to that new intermediate rule.

I think we need to add prepare-kconfig to PHONY target list (for consistency as
you said on IRC ;-))

With that fixed:
Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  Makefile | 36 +++++++++++++++++++-----------------
>  1 file changed, 19 insertions(+), 17 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 7c0dcb1..6f947f3 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -760,6 +760,8 @@ endif # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
>  HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
>  export HOSTCFLAGS
>  
> +prepare-kconfig: outputmakefile
> +
>  $(BUILD_DIR)/buildroot-config/%onf:
>  	mkdir -p $(@D)/lxdialog
>  	PKG_CONFIG_PATH="$(HOST_PKG_CONFIG_PATH)" $(MAKE) CC="$(HOSTCC_NOCCACHE)" HOSTCC="$(HOSTCC_NOCCACHE)" \
> @@ -779,19 +781,19 @@ COMMON_CONFIG_ENV = \
>  	HOST_GCC_VERSION="$(HOSTCC_VERSION)" \
>  	SKIP_LEGACY=
>  
> -xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
> +xconfig: $(BUILD_DIR)/buildroot-config/qconf prepare-kconfig
>  	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
>  
> -gconfig: $(BUILD_DIR)/buildroot-config/gconf outputmakefile
> +gconfig: $(BUILD_DIR)/buildroot-config/gconf prepare-kconfig
>  	@$(COMMON_CONFIG_ENV) srctree=$(TOPDIR) $< $(CONFIG_CONFIG_IN)
>  
> -menuconfig: $(BUILD_DIR)/buildroot-config/mconf outputmakefile
> +menuconfig: $(BUILD_DIR)/buildroot-config/mconf prepare-kconfig
>  	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
>  
> -nconfig: $(BUILD_DIR)/buildroot-config/nconf outputmakefile
> +nconfig: $(BUILD_DIR)/buildroot-config/nconf prepare-kconfig
>  	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
>  
> -config: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +config: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
>  
>  # For the config targets that automatically select options, we pass
> @@ -799,22 +801,22 @@ config: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  # no values are set for the legacy options so a subsequent oldconfig
>  # will query them. Therefore, run an additional olddefconfig.
>  
> -oldconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +oldconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	@$(COMMON_CONFIG_ENV) $< --oldconfig $(CONFIG_CONFIG_IN)
>  
> -randconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +randconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y $< --randconfig $(CONFIG_CONFIG_IN)
>  	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
>  
> -allyesconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +allyesconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y $< --allyesconfig $(CONFIG_CONFIG_IN)
>  	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
>  
> -allnoconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +allnoconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y $< --allnoconfig $(CONFIG_CONFIG_IN)
>  	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
>  
> -randpackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +randpackageconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
>  	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
>  		KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
> @@ -822,7 +824,7 @@ randpackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  	@rm -f $(CONFIG_DIR)/.config.nopkg
>  	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
>  
> -allyespackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +allyespackageconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
>  	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
>  		KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
> @@ -830,7 +832,7 @@ allyespackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  	@rm -f $(CONFIG_DIR)/.config.nopkg
>  	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
>  
> -allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
>  	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
>  		KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
> @@ -838,24 +840,24 @@ allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  	@rm -f $(CONFIG_DIR)/.config.nopkg
>  	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
>  
> -silentoldconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +silentoldconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	$(COMMON_CONFIG_ENV) $< --silentoldconfig $(CONFIG_CONFIG_IN)
>  
> -olddefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +olddefconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN)
>  
> -defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +defconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	@$(COMMON_CONFIG_ENV) $< --defconfig$(if $(DEFCONFIG),=$(DEFCONFIG)) $(CONFIG_CONFIG_IN)
>  
>  define percent_defconfig
> -%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
> +%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig prepare-kconfig
>  	# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
>  	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \
>  		$$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN)
>  endef
>  $(eval $(foreach d,$(TOPDIR) $(BR2_EXTERNAL),$(call percent_defconfig,$(d))$(sep)))
>  
> -savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> +savedefconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
>  	@$(COMMON_CONFIG_ENV) $< \
>  		--savedefconfig=$(if $(DEFCONFIG),$(DEFCONFIG),$(CONFIG_DIR)/defconfig) \
>  		$(CONFIG_CONFIG_IN)
> 

[Buildroot] [PATCH 05/16 v3] core: move rule to create basic directories

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> Some of those directories will be needed even during configuration, like
> BUILD_DIR, where we'll store the generated kconfig snippet.
> 
> So, move the rule to create them outside the HAVE_CONFIG block.

You mean BR2_HAVE_DOT_CONFIG here, right ?

With that fixed:
Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> ---
>  Makefile | 18 +++++++++---------
>  1 file changed, 9 insertions(+), 9 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 6f947f3..d4dc1c5 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -200,6 +200,15 @@ LEGAL_LICENSES_TXT_HOST = $(LEGAL_INFO_DIR)/host-licenses.txt
>  LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings
>  LEGAL_REPORT = $(LEGAL_INFO_DIR)/README
>  
> +################################################################################
> +#
> +# staging and target directories do NOT list these as
> +# dependencies anywhere else
> +#
> +################################################################################
> +$(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST):
> +	@mkdir -p $@
> +
>  BR2_CONFIG = $(CONFIG_DIR)/.config
>  
>  # Pull in the user's configuration file
> @@ -496,15 +505,6 @@ world: target-post-image
>  	legal-info legal-info-prepare legal-info-clean printvars help \
>  	list-defconfigs target-finalize target-post-image source-check
>  
> -################################################################################
> -#
> -# staging and target directories do NOT list these as
> -# dependencies anywhere else
> -#
> -################################################################################
> -$(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST):
> -	@mkdir -p $@
> -
>  # Populating the staging with the base directories is handled by the skeleton package
>  $(STAGING_DIR):
>  	@mkdir -p $(STAGING_DIR)
> 

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> Add the infrastructure for adding generated kconfig snippet in the
> menuconfig.
> 
> For now, the kconfig snippet is generated empty, the recipe for filling
> it in will be introduced in sub-sequent patches.

Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Samuel Martin <s.martin49@gmail.com>
> ---
>  Config.in             | 6 ++++++
>  Makefile              | 7 ++++++-
>  docs/manual/manual.mk | 3 +++
>  3 files changed, 15 insertions(+), 1 deletion(-)
> 
> diff --git a/Config.in b/Config.in
> index 7bb2a01..7f533ce 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -18,6 +18,10 @@ config BR2_EXTERNAL
>  	string
>  	option env="BR2_EXTERNAL"
>  
> +config BR2_BUILD_DIR
> +	string
> +	option env="BUILD_DIR"
> +
>  # Hidden config symbols for packages to check system gcc version
>  config BR2_HOST_GCC_VERSION
>  	string
> @@ -763,3 +767,5 @@ menu "User-provided options"
>  source "$BR2_EXTERNAL/Config.in"
>  
>  endmenu
> +
> +source "$BR2_BUILD_DIR/.br2-external.in"
> diff --git a/Makefile b/Makefile
> index d4dc1c5..39a1507 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -760,7 +760,7 @@ endif # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
>  HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
>  export HOSTCFLAGS
>  
> -prepare-kconfig: outputmakefile
> +prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in
>  
>  $(BUILD_DIR)/buildroot-config/%onf:
>  	mkdir -p $(@D)/lxdialog
> @@ -779,6 +779,7 @@ COMMON_CONFIG_ENV = \
>  	BR2_CONFIG=$(BR2_CONFIG) \
>  	BR2_EXTERNAL=$(BR2_EXTERNAL) \
>  	HOST_GCC_VERSION="$(HOSTCC_VERSION)" \
> +	BUILD_DIR=$(BUILD_DIR) \
>  	SKIP_LEGACY=
>  
>  xconfig: $(BUILD_DIR)/buildroot-config/qconf prepare-kconfig
> @@ -879,6 +880,10 @@ ifeq ($(NEED_WRAPPER),y)
>  	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
>  endif
>  
> +.PHONY: $(BUILD_DIR)/.br2-external.in
> +$(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
> +	@touch $@
> +
>  # printvars prints all the variables currently defined in our
>  # Makefiles. Alternatively, if a non-empty VARS variable is passed,
>  # only the variables matching the make pattern passed in VARS are
> diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
> index caf080a..b26f912 100644
> --- a/docs/manual/manual.mk
> +++ b/docs/manual/manual.mk
> @@ -7,6 +7,9 @@
>  MANUAL_SOURCES = $(sort $(wildcard docs/manual/*.txt) $(wildcard docs/images/*))
>  MANUAL_RESOURCES = $(TOPDIR)/docs/images
>  
> +# Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
> +MANUAL_DEPENDENCIES += prepare-kconfig
> +
>  # Our manual needs to generate lists
>  define MANUAL_GEN_LISTS
>  	$(Q)$(call MESSAGE,"Updating the manual lists...")
> 

[Buildroot] [PATCH 07/16 v3] docs/manual: prepare-config can be used as a dependency of documents

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :

Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain


> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  docs/manual/adding-packages-asciidoc.txt | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/docs/manual/adding-packages-asciidoc.txt b/docs/manual/adding-packages-asciidoc.txt
> index 0c10fd1..a278d44 100644
> --- a/docs/manual/adding-packages-asciidoc.txt
> +++ b/docs/manual/adding-packages-asciidoc.txt
> @@ -65,6 +65,8 @@ information is (assuming the document name is +foo+) :
>  
>  * +FOO_DEPENDENCIES+, optional, the list of packages (most probably,
>    host-packages) that must be built before building this document.
> +  If a hook of your document needs to access the _Kconfig_ structure,
> +  you may add +prepare-kconfig+ to the list of dependencies.
>  
>  There are also additional hooks (see xref:hooks[] for general information
>  on hooks), that a document may set to define extra actions to be done at
> 

[Buildroot] [PATCH 08/16 v3] core: do not hard-code inclusion of br2-external in Kconfig

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> Move the inclusion of br2-external's Config.in to the generated kconfig
> snippet.
> 
> This will ultimately allow us to use more than one br2-external tree.

Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> ---
>  Config.in | 7 -------
>  Makefile  | 8 +++++++-
>  2 files changed, 7 insertions(+), 8 deletions(-)
> 
> diff --git a/Config.in b/Config.in
> index 7f533ce..7810bf8 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -761,11 +761,4 @@ source "package/Config.in.host"
>  
>  source "Config.in.legacy"
>  
> -menu "User-provided options"
> -	depends on BR2_EXTERNAL != "support/dummy-external"
> -
> -source "$BR2_EXTERNAL/Config.in"
> -
> -endmenu
> -
>  source "$BR2_BUILD_DIR/.br2-external.in"
> diff --git a/Makefile b/Makefile
> index 39a1507..86c532e 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -882,7 +882,13 @@ endif
>  
>  .PHONY: $(BUILD_DIR)/.br2-external.in
>  $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
> -	@touch $@
> +	$(Q)( \
> +		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
> +		printf 'menu "User-provided options"\n'; \
> +		printf '\tdepends on BR2_EXTERNAL != "support/dummy-external"\n\n'; \
> +		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> +		printf 'endmenu # User-provided options\n'; \
> +	) >$@
>  
>  # printvars prints all the variables currently defined in our
>  # Makefiles. Alternatively, if a non-empty VARS variable is passed,
> 

[Buildroot] [PATCH 09/16 v3] core: get rid of our dummy br2-external tree

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> Now that we generate a kconfig snippet, we can conditionally include the
> BR2_EXTERNAL's Config.in only when BR2_EXTERNAL is supplied by the user,
> which means our empty/dummy Config.in is no longer useful to us.
> 
> As for external.mk, we can also include it only when BR2_EXTERNAL is
> supplied by the user, which means our empty/dummy external.mk is no
> longer of any use to us.
> 
> Ditch both of those files, and:
> 
>   - only generate actual content in the Kconfig snippet when we actually
>     do have a BR2_EXTERNAL provided by the user (i.e. BR2_EXTERNAL is not
>     empty);
> 
>   - add a variable that contains the path to the external.mk provided by
>     the user, or empty if none, and include the path set in that variable
>     (make can 'include' nothing without any problem! ;-) )

Maybe a dumb question but it's true with old make version (i.e 3.81) ?

Otherwise:
Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> ---
>  Makefile                           | 18 +++++++-----------
>  support/dummy-external/Config.in   |  0
>  support/dummy-external/external.mk |  0
>  3 files changed, 7 insertions(+), 11 deletions(-)
>  delete mode 100644 support/dummy-external/Config.in
>  delete mode 100644 support/dummy-external/external.mk
> 
> diff --git a/Makefile b/Makefile
> index 86c532e..be2f586 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -147,16 +147,11 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
>  # on the command line, therefore the file is re-created every time make is run.
>  #
>  # When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
> -# line), the .br-external file is removed and we point to
> -# support/dummy-external. This makes sure we can unconditionally include the
> -# Config.in and external.mk from the BR2_EXTERNAL directory. In this case,
> -# override is necessary so the user can clear BR2_EXTERNAL from the command
> -# line, but the dummy path is still used internally.
> +# line), the .br-external file is removed.
>  
>  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
>  -include $(BR2_EXTERNAL_FILE)
>  ifeq ($(BR2_EXTERNAL),)
> -  override BR2_EXTERNAL = support/dummy-external
>    $(shell rm -f $(BR2_EXTERNAL_FILE))
>  else
>    _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
> @@ -165,6 +160,7 @@ else
>    endif
>    override BR2_EXTERNAL := $(_BR2_EXTERNAL)
>    $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
> +  BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
>  endif
>  
>  # To make sure that the environment variable overrides the .config option,
> @@ -461,7 +457,8 @@ include boot/common.mk
>  include linux/linux.mk
>  include fs/common.mk
>  
> -include $(BR2_EXTERNAL)/external.mk
> +# Nothing to include if no BR2_EXTERNAL tree in use
> +include $(BR2_EXTERNAL_MK)
>  
>  # Now we are sure we have all the packages scanned and defined. We now
>  # check for each package in the list of enabled packages, that all its
> @@ -882,13 +879,12 @@ endif
>  
>  .PHONY: $(BUILD_DIR)/.br2-external.in
>  $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
> -	$(Q)( \
> +	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
>  		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
> -		printf 'menu "User-provided options"\n'; \
> -		printf '\tdepends on BR2_EXTERNAL != "support/dummy-external"\n\n'; \
> +		printf 'menu "User-provided options"\n\n'; \
>  		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
>  		printf 'endmenu # User-provided options\n'; \
> -	) >$@
> +	fi >$@
>  
>  # printvars prints all the variables currently defined in our
>  # Makefiles. Alternatively, if a non-empty VARS variable is passed,
> diff --git a/support/dummy-external/Config.in b/support/dummy-external/Config.in
> deleted file mode 100644
> index e69de29..0000000
> diff --git a/support/dummy-external/external.mk b/support/dummy-external/external.mk
> deleted file mode 100644
> index e69de29..0000000
> 

[Buildroot] [PATCH 09/16 v3] core: get rid of our dummy br2-external tree

[Yann E. MORIN]

Romain, All,

On 2016-08-06 17:30 +0200, Romain Naour spake thusly:
> Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> > Now that we generate a kconfig snippet, we can conditionally include the
> > BR2_EXTERNAL's Config.in only when BR2_EXTERNAL is supplied by the user,
> > which means our empty/dummy Config.in is no longer useful to us.
> > 
> > As for external.mk, we can also include it only when BR2_EXTERNAL is
> > supplied by the user, which means our empty/dummy external.mk is no
> > longer of any use to us.
> > 
> > Ditch both of those files, and:
> > 
> >   - only generate actual content in the Kconfig snippet when we actually
> >     do have a BR2_EXTERNAL provided by the user (i.e. BR2_EXTERNAL is not
> >     empty);
> > 
> >   - add a variable that contains the path to the external.mk provided by
> >     the user, or empty if none, and include the path set in that variable
> >     (make can 'include' nothing without any problem! ;-) )
> 
> Maybe a dumb question but it's true with old make version (i.e 3.81) ?

As discussed on IRC: yes, make-3.81 does not whine.

> Otherwise:
> Reviewed-by: Romain Naour <romain.naour@gmail.com>

Thanks!

I'll wait for your complete review of the series before replying to the
other comments (hwen needed), then I'll fix the series.

Thanks! :-)

Regards,
Yann E. MORIN.

> Best regards,
> Romain
> 
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > Cc: Peter Korsgaard <jacmet@uclibc.org>
> > Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
> > ---
> >  Makefile                           | 18 +++++++-----------
> >  support/dummy-external/Config.in   |  0
> >  support/dummy-external/external.mk |  0
> >  3 files changed, 7 insertions(+), 11 deletions(-)
> >  delete mode 100644 support/dummy-external/Config.in
> >  delete mode 100644 support/dummy-external/external.mk
> > 
> > diff --git a/Makefile b/Makefile
> > index 86c532e..be2f586 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -147,16 +147,11 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
> >  # on the command line, therefore the file is re-created every time make is run.
> >  #
> >  # When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
> > -# line), the .br-external file is removed and we point to
> > -# support/dummy-external. This makes sure we can unconditionally include the
> > -# Config.in and external.mk from the BR2_EXTERNAL directory. In this case,
> > -# override is necessary so the user can clear BR2_EXTERNAL from the command
> > -# line, but the dummy path is still used internally.
> > +# line), the .br-external file is removed.
> >  
> >  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
> >  -include $(BR2_EXTERNAL_FILE)
> >  ifeq ($(BR2_EXTERNAL),)
> > -  override BR2_EXTERNAL = support/dummy-external
> >    $(shell rm -f $(BR2_EXTERNAL_FILE))
> >  else
> >    _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
> > @@ -165,6 +160,7 @@ else
> >    endif
> >    override BR2_EXTERNAL := $(_BR2_EXTERNAL)
> >    $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
> > +  BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
> >  endif
> >  
> >  # To make sure that the environment variable overrides the .config option,
> > @@ -461,7 +457,8 @@ include boot/common.mk
> >  include linux/linux.mk
> >  include fs/common.mk
> >  
> > -include $(BR2_EXTERNAL)/external.mk
> > +# Nothing to include if no BR2_EXTERNAL tree in use
> > +include $(BR2_EXTERNAL_MK)
> >  
> >  # Now we are sure we have all the packages scanned and defined. We now
> >  # check for each package in the list of enabled packages, that all its
> > @@ -882,13 +879,12 @@ endif
> >  
> >  .PHONY: $(BUILD_DIR)/.br2-external.in
> >  $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
> > -	$(Q)( \
> > +	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
> >  		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
> > -		printf 'menu "User-provided options"\n'; \
> > -		printf '\tdepends on BR2_EXTERNAL != "support/dummy-external"\n\n'; \
> > +		printf 'menu "User-provided options"\n\n'; \
> >  		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> >  		printf 'endmenu # User-provided options\n'; \
> > -	) >$@
> > +	fi >$@
> >  
> >  # printvars prints all the variables currently defined in our
> >  # Makefiles. Alternatively, if a non-empty VARS variable is passed,
> > diff --git a/support/dummy-external/Config.in b/support/dummy-external/Config.in
> > deleted file mode 100644
> > index e69de29..0000000
> > diff --git a/support/dummy-external/external.mk b/support/dummy-external/external.mk
> > deleted file mode 100644
> > index e69de29..0000000
> > 
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 10/16 v3] core: introduce per br2-external ID

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> This unique ID is used to construct a per br2-external tree variable,
> BR2_EXTERNAL_$(ID), which contains the path to the br2-external tree.
> 
> This variable is available both from Kconfig (set in the Kconfig
> snippet) and from the .mk files.
> 
> Also, display the BR2_EXTERNAL_$(ID) and its path as a comment in the
> menuconfig.
> 
> This will ultimately allow us to support multiple br2-external trees at
> once, with that ID (and thus BR2_EXTERNAL_$(ID)) uniquely defining which
> br2-external tree is being used.
> 
> Note: since the variables in the Makefile and in Kconfig are named the
> same, the one we computed early on (second hunk) will be overridden by
> the one in .config when we have it. Thus, even though they are set to
> the same raw value, the one from .config is quoted and, being included
> later in the Makefile, will take precedence, so we must un-quote it
> before we include the br2-external's makefile (third hunk). That's
> unfortunate, but there is no easy way around that as we want the two
> variables to be named the same in Makefile and Kconfig (and we can't
> ask the user to un-quote that variable himself either), hence the
> little dirty trick.

Well, I have no other proposal to avoid the "little dirty trick".

Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> ---
>  Makefile | 22 +++++++++++++++++++++-
>  1 file changed, 21 insertions(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index be2f586..72b55cb 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -148,6 +148,9 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
>  #
>  # When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
>  # line), the .br-external file is removed.
> +#
> +# If the br2-external tree defines its ID, then export the path in the
> +# BR2_EXTERNAL_$(ID) variable.
>  
>  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
>  -include $(BR2_EXTERNAL_FILE)
> @@ -160,6 +163,13 @@ else
>    endif
>    override BR2_EXTERNAL := $(_BR2_EXTERNAL)
>    $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
> +  ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
> +    BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
> +    ifeq ($(BR2_EXTERNAL_ID),)
> +      $(error $(BR2_EXTERNAL) has no ID (in file 'external.id'))
> +    endif
> +    BR2_EXTERNAL_$(BR2_EXTERNAL_ID) = $(BR2_EXTERNAL)
> +  endif
>    BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
>  endif
>  
> @@ -457,6 +467,10 @@ include boot/common.mk
>  include linux/linux.mk
>  include fs/common.mk
>  
> +# If the br2-external tree defines its ID, then the BR2_EXTERNAL_$(ID)
> +# variable is also present in .config, so it is quoted. We must unquote
> +# it before feeding it to the br2-external makefile.
> +BR2_EXTERNAL_$(BR2_EXTERNAL_ID) := $(call qstrip,$(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)))
>  # Nothing to include if no BR2_EXTERNAL tree in use
>  include $(BR2_EXTERNAL_MK)
>  
> @@ -882,7 +896,13 @@ $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
>  	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
>  		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
>  		printf 'menu "User-provided options"\n\n'; \
> -		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> +		if [ -z "$(BR2_EXTERNAL_ID)" ]; then \
> +			printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> +		else \
> +			printf 'config BR2_EXTERNAL_%s\n' $(BR2_EXTERNAL_ID); \
> +			printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \
> +			printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(BR2_EXTERNAL_ID); \
> +		fi; \
>  		printf 'endmenu # User-provided options\n'; \
>  	fi >$@
>  
> 

[Buildroot] [PATCH 11/16 v3] docs/manual: document the br2-external ID

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> Update the manual with the new external.id mandatory file.
> 
> Take the opportunity to add a section listing all mandatory files,
> Config.in, external.mk and the new external.id, instead of just hinting
> about them in the external package recipes section.
> 
> Change the example to use the ID-suffixed variable instead of the raw
> BR2_EXTERNAL variable. Even though it is still possible to write a
> br2-external tree that is not multi-aware, we only document it with an
> ID.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Jeremy Rosen <jeremy.rosen@openwide.fr>
> 
> ---
> Changes v2 -> v3:
>   - better document the content of the ID file  (Jeremy)
> ---
>  docs/manual/customize-outside-br.txt | 59 ++++++++++++++++++++++++++----------
>  1 file changed, 43 insertions(+), 16 deletions(-)
> 
> diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
> index 9ad177d..839d1b1 100644
> --- a/docs/manual/customize-outside-br.txt
> +++ b/docs/manual/customize-outside-br.txt
> @@ -57,19 +57,50 @@ Or disable the usage of external definitions:
>  buildroot/ $ make BR2_EXTERNAL= xconfig
>  -----
>  
> -+BR2_EXTERNAL+ allows three different things:
> +A +BR2_EXTERNAL+ tree must contain at least those three files:
> +
> ++external.id+::
> +   That file should contain the ID for the +BR2_EXTERNAL+ tree.
> +   That ID is used to construct the +BR2_EXTERNAL_$(ID)+ variable,
> +   available in +Config.in+ and +external.mk+ (see below), so that ID
> +   must be a valid make and Kconfig variable. Buildroot sets
> +   +BR2_EXTERNAL_$(ID)+ to the path of the +BR2_EXTERNAL+ tree.
> +   The format for this file is a single line with the ID.
> ++
> +Examples of an +external.mk+ file that declares the ID +FOO+:
> ++
> +----
> +$ cat external.mk
> +FOO
> +----
> ++
> +Examples of IDs and the corresponding +BR2_EXTERNAL_$(ID)+ variable:
> ++
> +  * +FOO+ -> +BR2_EXTERNAL_FOO+
> +  * +BAR_42+ -> +BR2_EXTERNAL_BAR_42+
> +  * ...
> +
> +
> ++Config.in+::
> ++external.mk+::
> +   Those files (which may each be empty) can be used to define package
> +   recipes, like for packages bundled in Buildroot itself, or other
> +   custom configuration options.
> +
> +Using +BR2_EXTERNAL+ then allows three different things:
>  
>   * One can store all the board-specific configuration files there,
>     such as the kernel configuration, the root filesystem overlay, or
>     any other configuration file for which Buildroot allows to set its
> -   location. The +BR2_EXTERNAL+ value is available within the
> -   Buildroot configuration using +$(BR2_EXTERNAL)+. As an example, one
> -   could set the +BR2_ROOTFS_OVERLAY+ Buildroot option to
> -   +$(BR2_EXTERNAL)/board/<boardname>/overlay/+ (to specify a root
> +   location. The location of the +BR2_EXTERNAL+ tree is available in the
> +   Buildroot configuration using the +BR2_EXTERNAL_$(ID)+ variable. As
> +   an example, (considering the ID +BAR_42+), one could set the
> +   +BR2_ROOTFS_OVERLAY+ Buildroot option to
> +   +$(BR2_EXTERNAL_BAR_42)/board/<boardname>/overlay/+ (to specify a root
>     filesystem overlay), or the +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
>     Buildroot option to
> -   +$(BR2_EXTERNAL)/board/<boardname>/kernel.config+ (to specify the
> -   location of the kernel configuration file).
> +   +$(BR2_EXTERNAL_BAR_42)/board/<boardname>/kernel.config+ (to specify
> +   the location of the kernel configuration file).
>  
>   * One can store package recipes (i.e. +Config.in+ and
>     +<packagename>.mk+), or even custom configuration options and make
> @@ -77,27 +108,23 @@ buildroot/ $ make BR2_EXTERNAL= xconfig
>     make it appear in the top-level configuration menu, and includes
>     +$(BR2_EXTERNAL)/external.mk+ with the rest of the makefile logic.
>  +
> -.Note
> -Providing +Config.in+ and +external.mk+ is mandatory, but they can be
> -   empty.
> -+
>  The main usage of this is to store package recipes. The recommended
>     way to do this is to write a +$(BR2_EXTERNAL)/Config.in+ file that
>     looks like:
>  +
>  ------
> -source "$BR2_EXTERNAL/package/package1/Config.in"
> -source "$BR2_EXTERNAL/package/package2/Config.in"
> +source "$BR2_EXTERNAL_BAR_42/package/package1/Config.in"
> +source "$BR2_EXTERNAL_BAR_42/package/package2/Config.in"
>  ------
>  +
>  Then, have a +$(BR2_EXTERNAL)/external.mk+ file that looks like:
>  +
>  ------
> -include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))
> +include $(sort $(wildcard $(BR2_EXTERNAL_BAR_42)/package/*/*.mk))
>  ------
>  +
> -And then in +$(BR2_EXTERNAL)/package/package1+ and
> -   +$(BR2_EXTERNAL)/package/package2+ create normal Buildroot
> +And then in +$(BR2_EXTERNAL_FOO_42)/package/package1+ and
> +   +$(BR2_EXTERNAL_FOO_42)/package/package2+ create normal Buildroot

Here BR2_EXTERNAL_FOO or BR2_EXTERNAL_BAR_42 ?
BR2_EXTERNAL_BAR_42 it seems ;-)

Best regards,
Romain

>     package recipes, as explained in xref:adding-packages[].
>     If you prefer, you can also group the packages in subdirectories
>     called <boardname> and adapt the above paths accordingly.
> 

[Buildroot] [PATCH 14/16 v3] support/scripts: teach gen-manual-lists about BR2_EXTERNAL

[Romain Naour]

Hi Yann,

I already replied to patch 12-13, where BR2_EXTERNAL should be renamed to
_BR_EXTERNAL.

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> When using BR2_EXTERNAL, generate the list of packages from there.
> 
> This is currently effectively unused for now, as BR2_EXTERNAL is not
> available (not exported in the environment); that will be updated in a
> later patch.

Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  support/scripts/gen-manual-lists.py | 24 +++++++++++++++++++-----
>  1 file changed, 19 insertions(+), 5 deletions(-)
> 
> diff --git a/support/scripts/gen-manual-lists.py b/support/scripts/gen-manual-lists.py
> index d231eda..a672b7b 100644
> --- a/support/scripts/gen-manual-lists.py
> +++ b/support/scripts/gen-manual-lists.py
> @@ -141,7 +141,6 @@ class Buildroot:
>  
>      """
>      root_config = "Config.in"
> -    package_dirname = "package"
>      package_prefixes = ["BR2_PACKAGE_", "BR2_PACKAGE_HOST_"]
>      re_pkg_prefix = re.compile(r"^(" + "|".join(package_prefixes) + ").*")
>      deprecated_symbol = "BR2_DEPRECATED"
> @@ -182,12 +181,22 @@ class Buildroot:
>              'format': "_format_symbol_prompt_location",
>              'sorted': False,
>          },
> +        'external': {
> +            'filename': "br2-external-package-list",
> +            'root_menu': "User-provided options",
> +            'filter': "_is_real_package",
> +            'format': "_format_symbol_prompt_location",
> +            'sorted': True,
> +        },
>      }
>  
>      def __init__(self):
>          self.base_dir = os.environ.get("TOPDIR")
>          self.output_dir = os.environ.get("O")
> -        self.package_dir = os.path.join(self.base_dir, self.package_dirname)
> +        self.package_dirs = []
> +        self.package_dirs.append(os.path.join(self.base_dir, "package"))
> +        if os.getenv('BR2_EXTERNAL'):
> +            self.package_dirs.extend(os.environ["BR2_EXTERNAL"].split())
>          self.config = kconfiglib.Config(os.path.join(self.base_dir,
>                                                       self.root_config),
>                                          self.base_dir)
> @@ -259,9 +268,10 @@ class Buildroot:
>          #   symbol.
>          if not hasattr(self, "_package_list"):
>              pkg_list = []
> -            for _, _, files in os.walk(self.package_dir):
> -                for file_ in (f for f in files if f.endswith(".mk")):
> -                    pkg_list.append(re.sub(r"(.*?)\.mk", r"\1", file_))
> +            for d in self.package_dirs:
> +                for _, _, files in os.walk(d):
> +                    for file_ in (f for f in files if f.endswith(".mk")):
> +                        pkg_list.append(re.sub(r"(.*?)\.mk", r"\1", file_))
>              setattr(self, "_package_list", pkg_list)
>          for pkg in getattr(self, "_package_list"):
>              if type == 'real':
> @@ -491,6 +501,8 @@ class Buildroot:
>  
>  if __name__ == '__main__':
>      list_types = ['target-packages', 'host-packages', 'virtual-packages', 'deprecated']
> +    if os.getenv('BR2_EXTERNAL'):
> +        list_types.append('external')
>      parser = ArgumentParser()
>      parser.add_argument("list_type", nargs="?", choices=list_types,
>                          help="""\
> @@ -505,6 +517,8 @@ Generate the given list (generate all lists if unspecified)""")
>                          help="Output virtual package file")
>      parser.add_argument("--output-deprecated", dest="output_deprecated",
>                          help="Output deprecated file")
> +    parser.add_argument("--output-external", dest="output_external",
> +                        help="Output br2-external file")
>      args = parser.parse_args()
>      lists = [args.list_type] if args.list_type else list_types
>      buildroot = Buildroot()
> 

[Buildroot] [PATCH 15/16 v3] docs/manual: include packages from BR2_EXTERNAL if set

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> If the user is using one or more br2-external trees, add the list of
> packages therein to the manual.
> 
> Note that we do re-introduce BR2_EXTERNAL (when it was removed in
> [-SHA1-] (docs/manual: do not override BR2_EXTERNAL). The reason for

As reported on IRC, the SHA1 must be updated when this patch is applied.

> the removal is that BR2_EXTERNAL is no longer used as a Kconfig option.
> 
> However, BR2_EXTERNAL is now used by the gen-manual-lists.py script,
> which needs it to find packages' .mk files in the br2-external trees,
> not as a Kconfig option (which it is not).
> 
> So, even though that looks like a revert, that's not: the purpose is
> different, and we only pass it when not empty.
> 
> In case we have no br2-external, we want to create an empty file, so
> we can't just touch it, in case it was previously generated with
> br2-external trees; instead, we printf an empty string in there to
> ensure the file exists and is empty.

Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> ---
>  docs/manual/appendix.txt |  5 +++++
>  docs/manual/manual.mk    | 18 ++++++++++++++++++
>  2 files changed, 23 insertions(+)
> 
> diff --git a/docs/manual/appendix.txt b/docs/manual/appendix.txt
> index 87a20bd..7ef8bbc 100644
> --- a/docs/manual/appendix.txt
> +++ b/docs/manual/appendix.txt
> @@ -40,3 +40,8 @@ year following deprecation. For example, a symbol depending on
>  +BR2_DEPRECATED_SINCE_2013_05+ can be removed from 2014.05 onwards.
>  
>  include::deprecated-list.txt[]
> +
> +// This file is generated:
> +// - with the list of br2-external packages if BR2_EXTERNAL is set
> +// - empty otherwise
> +include::br2-external.txt[]
> diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
> index b26f912..2e711b8 100644
> --- a/docs/manual/manual.mk
> +++ b/docs/manual/manual.mk
> @@ -10,14 +10,32 @@ MANUAL_RESOURCES = $(TOPDIR)/docs/images
>  # Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
>  MANUAL_DEPENDENCIES += prepare-kconfig
>  
> +# If using one or more BR2_EXTERNAL trees, include the list of
> +# packages therein, otherwise, generate an empty file
> +ifeq ($(BR2_EXTERNAL),)
> +define MANUAL_GEN_LISTS_BR2_EXTERNAL
> +	$(Q)printf "" >$(@D)/br2-external.txt
> +endef
> +else
> +define MANUAL_GEN_LISTS_BR2_EXTERNAL
> +	$(Q)( \
> +		printf "== List of BR2_EXTERNAL packages\n\n"; \
> +		printf "include::br2-external-package-list.txt[]\n"; \
> +	) >$(@D)/br2-external.txt
> +endef
> +MANUAL_GEN_LISTS_ENV = BR2_EXTERNAL="$(BR2_EXTERNAL)"
> +endif
> +
>  # Our manual needs to generate lists
>  define MANUAL_GEN_LISTS
>  	$(Q)$(call MESSAGE,"Updating the manual lists...")
>  	$(Q)$(COMMON_CONFIG_ENV) \
> +		$(MANUAL_GEN_LISTS_ENV) \
>  		BR2_DEFCONFIG="" \
>  		TOPDIR=$(TOPDIR) \
>  		O=$(@D) \
>  		python -B $(TOPDIR)/support/scripts/gen-manual-lists.py
> +	$(MANUAL_GEN_LISTS_BR2_EXTERNAL)
>  endef
>  MANUAL_POST_RSYNC_HOOKS += MANUAL_GEN_LISTS
>  
> 

[Buildroot] [PATCH 16/16 v3] docs/manual: document multi br2-external

[Romain Naour]

Hi Yann,

Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :

Reviewed-by: Romain Naour <romain.naour@gmail.com>

Best regards,
Romain

> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> ---
>  docs/manual/customize-outside-br.txt | 62 ++++++++++++++++++++++++------------
>  1 file changed, 41 insertions(+), 21 deletions(-)
> 
> diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
> index 839d1b1..ed4a949 100644
> --- a/docs/manual/customize-outside-br.txt
> +++ b/docs/manual/customize-outside-br.txt
> @@ -17,19 +17,18 @@ place project-specific customizations in two locations:
>     having them nicely integrated in the build logic. This section
>     explains how to use +BR2_EXTERNAL+.
>  
> -+BR2_EXTERNAL+ is an environment variable that can be used to point to
> -a directory that contains Buildroot customizations. It can be passed
> -to any Buildroot +make+ invocation. It is automatically saved in the
> -hidden +.br-external+ file in the output directory. Thanks to this,
> -there is no need to pass +BR2_EXTERNAL+ at every +make+ invocation. It
> -can however be changed at any time by passing a new value, and can be
> ++BR2_EXTERNAL+ is an environment variable that can be set to contain the
> +paths to one or more directories that contains Buildroot customizations.
> +It can be passed to any Buildroot +make+ invocation. It is automatically
> +saved in the hidden +.br-external+ file in the output directory. Thanks
> +to this, there is no need to pass +BR2_EXTERNAL+ at every +make+ invocation.
> +It can however be changed at any time by passing a new value, and can be
>  removed by passing an empty value.
>  
>  .Note
> -The +BR2_EXTERNAL+ path can be either an absolute or a relative path,
> -but if it's passed as a relative path, it is important to note that it
> -is interpreted relative to the main Buildroot source directory, *not*
> -to the Buildroot output directory.
> +Paths in +BR2_EXTERNAL+ can be either absolute or relative paths, but it
> +is important to note that relative paths are interpreted relative to the
> +main Buildroot source directory, *not* to the Buildroot output directory.
>  
>  Some examples:
>  
> @@ -51,6 +50,12 @@ We can switch to another external definitions directory at any time:
>  buildroot/ $ make BR2_EXTERNAL=/where/we/have/barfoo xconfig
>  -----
>  
> +We can also use multiple +BR2_EXTERNAL+ locations:
> +
> +----
> +buildroot/ $ make BR2_EXTERNAL="/path/to/foobar /where/we/have/barfoo" menuconfig
> +----
> +
>  Or disable the usage of external definitions:
>  
>  -----
> @@ -60,13 +65,23 @@ buildroot/ $ make BR2_EXTERNAL= xconfig
>  A +BR2_EXTERNAL+ tree must contain at least those three files:
>  
>  +external.id+::
> -   That file should contain the ID for the +BR2_EXTERNAL+ tree.
> +   That file should contain the ID for that +BR2_EXTERNAL+ tree.
>     That ID is used to construct the +BR2_EXTERNAL_$(ID)+ variable,
>     available in +Config.in+ and +external.mk+ (see below), so that ID
>     must be a valid make and Kconfig variable. Buildroot sets
>     +BR2_EXTERNAL_$(ID)+ to the path of the +BR2_EXTERNAL+ tree.
>     The format for this file is a single line with the ID.
>  +
> +.Note:
> +Since it is possible to use multiple +BR2_EXTERNAL+ trees at once, this
> +  ID is used by Buildroot to generate variables for each of those trees.
> +  That ID is used to identify your +BR2_EXTERNAL+ tree, so try to come up
> +  with an ID that really describes your +BR2_EXTERNAL+ tree, in order for
> +  it to be relatively unique, so that it does not clash with another ID
> +  from another +BR2_EXTERNAL+ tree, especially if you are planning on
> +  somehow sharing your +BR2_EXTERNAL+ tree with third parties or using
> +  +BR2_EXTERNAL+ trees from third parties.
> ++
>  Examples of an +external.mk+ file that declares the ID +FOO+:
>  +
>  ----
> @@ -102,22 +117,22 @@ Using +BR2_EXTERNAL+ then allows three different things:
>     +$(BR2_EXTERNAL_BAR_42)/board/<boardname>/kernel.config+ (to specify
>     the location of the kernel configuration file).
>  
> - * One can store package recipes (i.e. +Config.in+ and
> -   +<packagename>.mk+), or even custom configuration options and make
> -   logic. Buildroot automatically includes +$(BR2_EXTERNAL)/Config.in+ to
> -   make it appear in the top-level configuration menu, and includes
> -   +$(BR2_EXTERNAL)/external.mk+ with the rest of the makefile logic.
> + * One can store package recipes (i.e. +Config.in+ and +<packagename>.mk+),
> +   or even custom configuration options and make logic. Buildroot
> +   automatically includes the +Config.in+ from each paths in +BR2_EXTERNAL+
> +   to make it appear in a sub-menu labelled _User-provided options_ at the
> +   root of the main configuration menu, and includes the +external.mk+ from
> +   each paths in +BR2_EXTERNAL+ with the rest of the makefile logic.
>  +
>  The main usage of this is to store package recipes. The recommended
> -   way to do this is to write a +$(BR2_EXTERNAL)/Config.in+ file that
> -   looks like:
> +   way to do this is to write +Config.in+ files that look like:
>  +
>  ------
>  source "$BR2_EXTERNAL_BAR_42/package/package1/Config.in"
>  source "$BR2_EXTERNAL_BAR_42/package/package2/Config.in"
>  ------
>  +
> -Then, have a +$(BR2_EXTERNAL)/external.mk+ file that looks like:
> +Then, write +/external.mk+ files that look like:
>  +
>  ------
>  include $(sort $(wildcard $(BR2_EXTERNAL_BAR_42)/package/*/*.mk))
> @@ -130,7 +145,12 @@ And then in +$(BR2_EXTERNAL_FOO_42)/package/package1+ and
>     called <boardname> and adapt the above paths accordingly.
>  
>   * One can store Buildroot defconfigs in the +configs+ subdirectory of
> -   +$(BR2_EXTERNAL)+. Buildroot will automatically show them in the
> +   a +BR2_EXTERNAL+ tree. Buildroot will automatically show them in the
>     output of +make list-defconfigs+ and allow them to be loaded with the
>     normal +make <name>_defconfig+ command. They will be visible under the
> -   +User-provided configs+' label in the 'make list-defconfigs' output.
> +   +User-provided configs+ label in the 'make list-defconfigs' output.
> ++
> +.Note
> +If a defconfig file is present in more than one +BR2_EXTERNAL+ location,
> +   then the last one is used. It is also possible to override a defconfig
> +   bundled in Buildroot.
> 

[Buildroot] [PATCH 01/16 v3] core: move pkg-utils.mk to support/

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:21 +0200, Yann E. MORIN wrote:
> pkg-utils.mk contains various definitions that are used in the package
> infrastructures and packages themselves.
> 
> However, those definitions can be useful in other parts of Buildroot,
> and are already used in a few places that are not related to the package
> infrastructure. Also, $(sep) will be needed early in the Makefile when
> we eventually support multiple br2-external trees.
> 
> Since this file only contains definitions, we can include it anytime.
> 
> So, consider that file to no longer be specific to the package infras:
>   - move it to support and rename it,
>   - move a few similar definitions from the main Makefile to that file.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> ---
>  Makefile              |  10 +----
>  package/pkg-utils.mk  |  96 -------------------------------------------
>  support/misc/utils.mk | 111 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 113 insertions(+), 104 deletions(-)
>  create mode 100644 support/misc/utils.mk

Applied to next, thanks. I'm glad to see a few lines disappearing from
the main Makefile. Sadly, the remaining of your series adds some more
lines to this file :-/

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 02/16 v3] core: commonalise the bundled and br2-external %_defconfig rules

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:22 +0200, Yann E. MORIN wrote:

> +define percent_defconfig
> +%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
> +	# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig

This comment in here was a bit annoying, because it appears on stdout
when you run "make <foo>_defconfig". So I've moved it outside of the
target definition instead, in order to preserve the existing behavior
in terms of what appears on stdout when loading a defconfig.

Applied to next with this change.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 03/16 v3] doc/asciidoc: add possibility to define document dependencies

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:23 +0200, Yann E. MORIN wrote:

> diff --git a/docs/manual/adding-packages-asciidoc.txt b/docs/manual/adding-packages-asciidoc.txt
> index 6e21786..0c10fd1 100644
> --- a/docs/manual/adding-packages-asciidoc.txt
> +++ b/docs/manual/adding-packages-asciidoc.txt
> @@ -63,6 +63,9 @@ information is (assuming the document name is +foo+) :
>    to one or more directories containing so-called resources (like CSS or
>    images). By default, empty.
>  
> +* +FOO_DEPENDENCIES+, optional, the list of packages (most probably,
> +  host-packages) that must be built before building this document.
> +
>  There are also additional hooks (see xref:hooks[] for general information
>  on hooks), that a document may set to define extra actions to be done at
>  various steps:
> diff --git a/package/doc-asciidoc.mk b/package/doc-asciidoc.mk
> index 6ab5ad2..ede6ed1 100644
> --- a/package/doc-asciidoc.mk
> +++ b/package/doc-asciidoc.mk
> @@ -140,7 +140,7 @@ endef
>  ################################################################################
>  define ASCIIDOC
>  # Single line, because splitting a foreach is not easy...
> -$(1)-check-dependencies: asciidoc-check-dependencies
> +$(1)-check-dependencies: asciidoc-check-dependencies $$($(2)_DEPENDENCIES)

My concern is that $(2)_DEPENDENCIES has nothing to do with *checking*
dependencies, so it's a bit weird to see this added to a target called
<something>-check-dependencies.

Could you refresh me on how this $(1)-check-dependencies target is
used? I had a quick look at the code around and couldn't remember, and
wanted to look at some other patches in your series.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 05/16 v3] core: move rule to create basic directories

[Thomas Petazzoni]

Hello,

On Sat, 6 Aug 2016 17:16:20 +0200, Romain Naour wrote:
> Hi Yann,
> 
> Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> > Some of those directories will be needed even during configuration, like
> > BUILD_DIR, where we'll store the generated kconfig snippet.
> > 
> > So, move the rule to create them outside the HAVE_CONFIG block.  
> 
> You mean BR2_HAVE_DOT_CONFIG here, right ?

Yann, could you reply to this comment? I think Romain is right, but I
just want to be sure before fixing up myself when applying.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:26 +0200, Yann E. MORIN wrote:

> +.PHONY: $(BUILD_DIR)/.br2-external.in

Why is a real file that exists on the filesystem a phony target?

> +# Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
> +MANUAL_DEPENDENCIES += prepare-kconfig

It's a bit crazy the complexity that we have just to allow our manual
to have the list of all packages in Buildroot. I'm not even sure this
section of the manual is that useful, to be honest. But oh well, that's
supported today, so we have to continue support it.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 02/16 v3] core: commonalise the bundled and br2-external %_defconfig rules

[Yann E. MORIN]

Thomas, All,

On 2016-08-27 16:12 +0200, Thomas Petazzoni spake thusly:
> On Sun, 17 Jul 2016 12:34:22 +0200, Yann E. MORIN wrote:
> 
> > +define percent_defconfig
> > +%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
> > +	# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
> 
> This comment in here was a bit annoying, because it appears on stdout
> when you run "make <foo>_defconfig". So I've moved it outside of the
> target definition instead, in order to preserve the existing behavior
> in terms of what appears on stdout when loading a defconfig.

Well, that's how I did it first, but Arnout contended that the comment
should be moved like I did (or at least that's what I understood from
his review).

I'm happy you moved it. ;-)

Thanks!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 05/16 v3] core: move rule to create basic directories

[Yann E. MORIN]

Romain, Thomas, All,

On 2016-08-27 16:14 +0200, Thomas Petazzoni spake thusly:
> On Sat, 6 Aug 2016 17:16:20 +0200, Romain Naour wrote:
> > Hi Yann,
> > 
> > Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> > > Some of those directories will be needed even during configuration, like
> > > BUILD_DIR, where we'll store the generated kconfig snippet.
> > > 
> > > So, move the rule to create them outside the HAVE_CONFIG block.  
> > 
> > You mean BR2_HAVE_DOT_CONFIG here, right ?
> 
> Yann, could you reply to this comment? I think Romain is right, but I
> just want to be sure before fixing up myself when applying.

Yes, he's right, this should have been 'BR2_HAVE_DOT_CONFIG'.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 03/16 v3] doc/asciidoc: add possibility to define document dependencies

[Yann E. MORIN]

Thomas, All,

On 2016-08-27 16:14 +0200, Thomas Petazzoni spake thusly:
> On Sun, 17 Jul 2016 12:34:23 +0200, Yann E. MORIN wrote:
> 
> > diff --git a/docs/manual/adding-packages-asciidoc.txt b/docs/manual/adding-packages-asciidoc.txt
> > index 6e21786..0c10fd1 100644
> > --- a/docs/manual/adding-packages-asciidoc.txt
> > +++ b/docs/manual/adding-packages-asciidoc.txt
> > @@ -63,6 +63,9 @@ information is (assuming the document name is +foo+) :
> >    to one or more directories containing so-called resources (like CSS or
> >    images). By default, empty.
> >  
> > +* +FOO_DEPENDENCIES+, optional, the list of packages (most probably,
> > +  host-packages) that must be built before building this document.
> > +
> >  There are also additional hooks (see xref:hooks[] for general information
> >  on hooks), that a document may set to define extra actions to be done at
> >  various steps:
> > diff --git a/package/doc-asciidoc.mk b/package/doc-asciidoc.mk
> > index 6ab5ad2..ede6ed1 100644
> > --- a/package/doc-asciidoc.mk
> > +++ b/package/doc-asciidoc.mk
> > @@ -140,7 +140,7 @@ endef
> >  ################################################################################
> >  define ASCIIDOC
> >  # Single line, because splitting a foreach is not easy...
> > -$(1)-check-dependencies: asciidoc-check-dependencies
> > +$(1)-check-dependencies: asciidoc-check-dependencies $$($(2)_DEPENDENCIES)
> 
> My concern is that $(2)_DEPENDENCIES has nothing to do with *checking*
> dependencies, so it's a bit weird to see this added to a target called
> <something>-check-dependencies.

Indeed, that's not the best solution.

It ensures oredering, for sure, and it ensures it is done only once for
the document, even if multiple formats are rquested.

> Could you refresh me on how this $(1)-check-dependencies target is
> used? I had a quick look at the code around and couldn't remember, and
> wanted to look at some other patches in your series.

The $(1)-check-depednencies rule is defined outside of the ASCIIDOC_INNER
macro, in the ASCIIDOC macro, so that we do not redefine it for every
formats.

The dependency on $(1)-check-dependencies is however set in the
INNER_MACRO, as a dependency of the output file $(1).$(6) .

I can see at making that cleaner, but we'll probably have to define
another rule for that. Let's see what I can come up with...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 04/16 v3] core: introduce an intermediate rule before the configurators

[Yann E. MORIN]

Romain, All,

On 2016-08-06 17:13 +0200, Romain Naour spake thusly:
> Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> > Currently, all configurators depend on generating the out-of-tree
> > Makefile wrapper.
> > 
> > In an upcoming patch, we'll need to also generate a kconfig fragment,
> > so it will have to kick in before we run the configurators.
> > 
> > Introduce a new intermediate "prepare-kconfig" rule, so we can
> > commonalise the dependencies of the configurators. Move the dependency
> > on the Makefile wrapper to that new intermediate rule.
> 
> I think we need to add prepare-kconfig to PHONY target list (for consistency as
> you said on IRC ;-))

Yes, indeed, it should be made .PHONY ...

> With that fixed:
> Reviewed-by: Romain Naour <romain.naour@gmail.com>

Thanks!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Yann E. MORIN]

Thomas, All,

On 2016-08-27 16:16 +0200, Thomas Petazzoni spake thusly:
> On Sun, 17 Jul 2016 12:34:26 +0200, Yann E. MORIN wrote:
> 
> > +.PHONY: $(BUILD_DIR)/.br2-external.in
> 
> Why is a real file that exists on the filesystem a phony target?

Because we want to regenerate it every time we run, in case the user
passes a new set of BR2_EXTERNAL.

With only this patch, of course that does not bring anything, as we
simply touch that file. It becones usefull in followup patches.

> > +# Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
> > +MANUAL_DEPENDENCIES += prepare-kconfig
> 
> It's a bit crazy the complexity that we have just to allow our manual
> to have the list of all packages in Buildroot. I'm not even sure this
> section of the manual is that useful, to be honest. But oh well, that's
> supported today, so we have to continue support it.

Well, I am prersonally not very impressed with the list of packags in
the manual either...

That being said, some users find it interesting and, as you said, we
already have it and the code is already there...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 11/16 v3] docs/manual: document the br2-external ID

[Yann E. MORIN]

Romain, All,

On 2016-08-06 17:48 +0200, Romain Naour spake thusly:
> Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> > Update the manual with the new external.id mandatory file.
[--SNIP--]
> > diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
> > index 9ad177d..839d1b1 100644
> > --- a/docs/manual/customize-outside-br.txt
> > +++ b/docs/manual/customize-outside-br.txt
[--SNIP--]
> > @@ -77,27 +108,23 @@ buildroot/ $ make BR2_EXTERNAL= xconfig
> >     make it appear in the top-level configuration menu, and includes
> >     +$(BR2_EXTERNAL)/external.mk+ with the rest of the makefile logic.
> >  +
> > -.Note
> > -Providing +Config.in+ and +external.mk+ is mandatory, but they can be
> > -   empty.
> > -+
> >  The main usage of this is to store package recipes. The recommended
> >     way to do this is to write a +$(BR2_EXTERNAL)/Config.in+ file that
> >     looks like:
> >  +
> >  ------
> > -source "$BR2_EXTERNAL/package/package1/Config.in"
> > -source "$BR2_EXTERNAL/package/package2/Config.in"
> > +source "$BR2_EXTERNAL_BAR_42/package/package1/Config.in"
> > +source "$BR2_EXTERNAL_BAR_42/package/package2/Config.in"
> >  ------
> >  +
> >  Then, have a +$(BR2_EXTERNAL)/external.mk+ file that looks like:

s/BR2_EXTERNAL/BR2_EXTERNAL_BR_42/

> >  ------
> > -include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))
> > +include $(sort $(wildcard $(BR2_EXTERNAL_BAR_42)/package/*/*.mk))
> >  ------
> >  +
> > -And then in +$(BR2_EXTERNAL)/package/package1+ and
> > -   +$(BR2_EXTERNAL)/package/package2+ create normal Buildroot
> > +And then in +$(BR2_EXTERNAL_FOO_42)/package/package1+ and
> > +   +$(BR2_EXTERNAL_FOO_42)/package/package2+ create normal Buildroot
> 
> Here BR2_EXTERNAL_FOO or BR2_EXTERNAL_BAR_42 ?
> BR2_EXTERNAL_BAR_42 it seems ;-)

Yes.

Thanks! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 12/16 v3] core: handle .br-external in a make rule

[Yann E. MORIN]

Romain, All,

On 2016-07-25 23:13 +0200, Yann E. MORIN spake thusly:
> On 2016-07-24 22:05 +0200, Romain Naour spake thusly:
> > Le 17/07/2016 ? 12:34, Yann E. MORIN a ?crit :
> > > Currently, we treat the case where we have no br2-external tree
> > > (BR2_EXTERNAL is empty) differently from the case where we do have
> > > one (BR2_EXTERNAL is not empty).
> > > 
> > > There is now no reason to treat those two cases differently:
> > > 
> > >   - the kconfig snippet is always generated appropriately (i.e. it would
> > >     include the br2-external tree if set, or include nothing otherwise);
> > > 
> > >   - we no longer have a dummy br-external tree either.
> > > 
> > > As such, generate the .br-external cache file in both cases.
> > > 
> > > This will make it much easier to handle when we introduce support for
> > > multiple br2-external trees.
> > > 
> > > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > > Cc: Peter Korsgaard <jacmet@uclibc.org>
> > > Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> > > Cc: Arnout Vandecappelle <arnout@mind.be>
> > > ---
> > >  Makefile | 13 ++++++-------
> > >  1 file changed, 6 insertions(+), 7 deletions(-)
> > > 
> > > diff --git a/Makefile b/Makefile
> > > index 72b55cb..67334e5 100644
> > > --- a/Makefile
> > > +++ b/Makefile
> > > @@ -146,23 +146,17 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
> > >  # On subsequent invocations of make, it is read in. It can still be overridden
> > >  # on the command line, therefore the file is re-created every time make is run.
> > >  #
> > > -# When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
> > > -# line), the .br-external file is removed.
> > > -#
> > >  # If the br2-external tree defines its ID, then export the path in the
> > >  # BR2_EXTERNAL_$(ID) variable.
> > >  
> > >  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
> > >  -include $(BR2_EXTERNAL_FILE)
> > > -ifeq ($(BR2_EXTERNAL),)
> > > -  $(shell rm -f $(BR2_EXTERNAL_FILE))
> > > -else
> > > +ifneq ($(BR2_EXTERNAL),)
> > >    _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
> > >    ifeq ($(_BR2_EXTERNAL),)
> > >      $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' does not exist, relative to $(TOPDIR))
> > >    endif
> > >    override BR2_EXTERNAL := $(_BR2_EXTERNAL)
> > > -  $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
> > >    ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
> > >      BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
> > >      ifeq ($(BR2_EXTERNAL_ID),)
> > > @@ -173,6 +167,11 @@ else
> > >    BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
> > >  endif
> > >  
> > > +# This needs to be *after* we compute BR_EXTERNAL, above.
> > > +.PHONY: $(BR2_EXTERNAL_FILE)
> > > +$(BR2_EXTERNAL_FILE):
> > > +	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@
> > 
> > s/BR_EXTERNAL/BR2_EXTERNAL/
> 
> Right, but it should be: s/BR2_EXTERNAL/_BR2_EXTERNAL/  (i.e. with a
> leading underscore) as we want to store the full path, which is
> constructed with the first line in the ifdef block.

No, it should really be BR2_EXTERNAL as you noticed.

Regards,
Yann E. MORIN.

> > It's probably a rebase issue since it's fixed in the following patch ;-)
> 
> Probably, probably... ;-)
> 
> Thanks for spotting.
> 
> Regards,
> Yann E. MORIN.
> 
> 
> -- 
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 02/16 v3] core: commonalise the bundled and br2-external %_defconfig rules

[Arnout Vandecappelle]

On 27-08-16 16:16, Yann E. MORIN wrote:
> Thomas, All,
> 
> On 2016-08-27 16:12 +0200, Thomas Petazzoni spake thusly:
>> On Sun, 17 Jul 2016 12:34:22 +0200, Yann E. MORIN wrote:
>>
>>> +define percent_defconfig
>>> +%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
>>> +	# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
>>
>> This comment in here was a bit annoying, because it appears on stdout
>> when you run "make <foo>_defconfig". So I've moved it outside of the
>> target definition instead, in order to preserve the existing behavior
>> in terms of what appears on stdout when loading a defconfig.
> 
> Well, that's how I did it first, but Arnout contended that the comment
> should be moved like I did (or at least that's what I understood from
> his review).

 With "inside the define" I meant exactly what Thomas did: inside the define
(not inside the rule).

 Regards,
 Arnout

> 
> I'm happy you moved it. ;-)
> 
> Thanks!
> 
> Regards,
> Yann E. MORIN.
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 04/16 v3] core: introduce an intermediate rule before the configurators

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:24 +0200, Yann E. MORIN wrote:
> Currently, all configurators depend on generating the out-of-tree
> Makefile wrapper.
> 
> In an upcoming patch, we'll need to also generate a kconfig fragment,
> so it will have to kick in before we run the configurators.
> 
> Introduce a new intermediate "prepare-kconfig" rule, so we can
> commonalise the dependencies of the configurators. Move the dependency
> on the Makefile wrapper to that new intermediate rule.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  Makefile | 36 +++++++++++++++++++-----------------
>  1 file changed, 19 insertions(+), 17 deletions(-)

Applied to next, after marking the new prepare-kconfig target as PHONY.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 05/16 v3] core: move rule to create basic directories

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:25 +0200, Yann E. MORIN wrote:
> Some of those directories will be needed even during configuration, like
> BUILD_DIR, where we'll store the generated kconfig snippet.
> 
> So, move the rule to create them outside the HAVE_CONFIG block.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>

I've applied this patch to next, after fixing the typo in the commit
log noticed by Romain Naour. One question below, though.

> +################################################################################
> +#
> +# staging and target directories do NOT list these as
> +# dependencies anywhere else
> +#
> +################################################################################

Do you understand what this comment means? It seems to date back from
the "major rework" commit 08782ae7d8812933ad3f932132955ea1c0fb1e52 in
2002, and I don't really understand the meaning of this comment.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:26 +0200, Yann E. MORIN wrote:

> +.PHONY: $(BUILD_DIR)/.br2-external.in
> +$(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
> +	@touch $@

I've added a comment that explains why this target is PHONY, and
applied this patch to the next branch. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 03/16 v3] doc/asciidoc: add possibility to define document dependencies

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:23 +0200, Yann E. MORIN wrote:
> Currently, a document can not have dependencies, except for the purely
> internal ones (like checking asciidoc version, and presence of dblatex).
> 
> For our own manual, this will come in handy when we introduce a
> generated kconfig snippet, so we can actually make the manual depend on
> that snippet being generated first.
> 
> For external documents, it can be used to depend on host-packages if
> need be (e.g. a custom host packages that generates specific media files
> included in the manual).
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  docs/manual/adding-packages-asciidoc.txt | 3 +++
>  package/doc-asciidoc.mk                  | 2 +-
>  2 files changed, 4 insertions(+), 1 deletion(-)

Applied to next, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 07/16 v3] docs/manual: prepare-config can be used as a dependency of documents

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:27 +0200, Yann E. MORIN wrote:
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  docs/manual/adding-packages-asciidoc.txt | 2 ++
>  1 file changed, 2 insertions(+)

In the title: s/prepare-config/prepare-kconfig/, I've fixed that up
when applying (to the next branch, obviously).

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 08/16 v3] core: do not hard-code inclusion of br2-external in Kconfig

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:28 +0200, Yann E. MORIN wrote:

>  .PHONY: $(BUILD_DIR)/.br2-external.in
>  $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
> -	@touch $@
> +	$(Q)( \
> +		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
> +		printf 'menu "User-provided options"\n'; \
> +		printf '\tdepends on BR2_EXTERNAL != "support/dummy-external"\n\n'; \
> +		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> +		printf 'endmenu # User-provided options\n'; \
> +	) >$@

Shouldn't the logic that generates the .br2-external.in be moved to a
shell script?

I'm generally in favor of having most of the logic in make, but here,
it is really ugly. Did you try having this in a shell script?

Also, what about using readlink or realpath to convert the path to an
absolute path?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 10/16 v3] core: introduce per br2-external ID

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:30 +0200, Yann E. MORIN wrote:
> This unique ID is used to construct a per br2-external tree variable,
> BR2_EXTERNAL_$(ID), which contains the path to the br2-external tree.

I would suggest to use the terminology "name" rather than "ID". I
think it's much easier to understand when using "name" that using "ID".
It's a bit longer, but not much, and will make things clearer.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 11/16 v3] docs/manual: document the br2-external ID

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:31 +0200, Yann E. MORIN wrote:

> diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
> index 9ad177d..839d1b1 100644
> --- a/docs/manual/customize-outside-br.txt
> +++ b/docs/manual/customize-outside-br.txt
> @@ -57,19 +57,50 @@ Or disable the usage of external definitions:
>  buildroot/ $ make BR2_EXTERNAL= xconfig
>  -----
>  
> -+BR2_EXTERNAL+ allows three different things:
> +A +BR2_EXTERNAL+ tree must contain at least those three files:
> +
> ++external.id+::

I thought this file was optional when there was a single external tree,
but you're saying it's mandatory here. Is it because the "no
external.id file" case is only supported for backward compat, and we
expect that new users write an external.id, regardless of whether they
are using a single or multiple external trees ?

> +   That file should contain the ID for the +BR2_EXTERNAL+ tree.
> +   That ID is used to construct the +BR2_EXTERNAL_$(ID)+ variable,
> +   available in +Config.in+ and +external.mk+ (see below), so that ID
> +   must be a valid make and Kconfig variable. Buildroot sets
> +   +BR2_EXTERNAL_$(ID)+ to the path of the +BR2_EXTERNAL+ tree.
> +   The format for this file is a single line with the ID.
> ++
> +Examples of an +external.mk+ file that declares the ID +FOO+:

Huh? Is it really external.mk that contains the ID ?

> ++
> +----
> +$ cat external.mk

Ditto.

> +FOO
> +----
> ++
> +Examples of IDs and the corresponding +BR2_EXTERNAL_$(ID)+ variable:
> ++
> +  * +FOO+ -> +BR2_EXTERNAL_FOO+
> +  * +BAR_42+ -> +BR2_EXTERNAL_BAR_42+
> +  * ...

Maybe mention which characters are allowed and which ones are not
allowed?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 12/16 v3] core: handle .br-external in a make rule

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:32 +0200, Yann E. MORIN wrote:

>  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
>  -include $(BR2_EXTERNAL_FILE)
> -ifeq ($(BR2_EXTERNAL),)
> -  $(shell rm -f $(BR2_EXTERNAL_FILE))
> -else
> +ifneq ($(BR2_EXTERNAL),)
>    _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
>    ifeq ($(_BR2_EXTERNAL),)
>      $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' does not exist, relative to $(TOPDIR))
>    endif
>    override BR2_EXTERNAL := $(_BR2_EXTERNAL)
> -  $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
>    ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
>      BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
>      ifeq ($(BR2_EXTERNAL_ID),)
> @@ -173,6 +167,11 @@ else
>    BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
>  endif
>  
> +# This needs to be *after* we compute BR_EXTERNAL, above.
> +.PHONY: $(BR2_EXTERNAL_FILE)
> +$(BR2_EXTERNAL_FILE):
> +	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@
> +

I'm probably missing something, but where is this make target being
"triggered" ?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 13/16 v3] core: add support for multiple br2-external trees

[Thomas Petazzoni]

Hello,

On Sun, 17 Jul 2016 12:34:33 +0200, Yann E. MORIN wrote:

> What we do is to treat BR2_EXTERNAL as a space-separated list of paths,
> which we iterate to construct:

As discussed on IRC, is the space-separated list the best choice? Most
environment variables that contain paths in Unix are colon-separated
instead (PATH, LD_LIBRARY_PATH, PKG_CONFIG_PATH, etc, etc.)

Making it space-separated makes it a bit annoying IMO.

> -# This needs to be *after* we compute BR_EXTERNAL, above.
>  .PHONY: $(BR2_EXTERNAL_FILE)
>  $(BR2_EXTERNAL_FILE):
> -	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@
> +	@echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) >$@
> +
> +# Those two variables need to be defined as simply-expanded variables, they
> +# can't be recursively-expanded, because the values they are assigned change
> +# with each iteration of the foreach, below.
> +BR_EXTERNAL_IDS :=
> +EXTRA_ENV :=
> +
> +# If there is no or one br2-external trees used, then we don't require (but
> +# accept) an ID; otherwise (i.e. there are two or more br2-external trees)
> +# we require they do all define their ID.
> +ifeq ($(filter-out 0 1,$(words $(BR2_EXTERNAL))),)
> +BR2_EXTERNAL_NEED_ID := loose
> +else
> +BR2_EXTERNAL_NEED_ID := strict
> +endif
> +
> +# Validate the br2-external tree passed as $(1):
> +# - check the directory actually exists
> +# - check if we need and have a non-empty ID
> +# - check the ID is not a duplicate
> +# - set variables for later use
> +define BR2_EXTERNAL_VALIDATE
> +  _BR_EXT_DIR := $$(shell cd $(1) >/dev/null 2>&1 && pwd)
> +  ifeq ($$(_BR_EXT_DIR),)
> +    $$(error BR2_EXTERNAL='$(1)' does not exist, relative to $$(TOPDIR))
> +  endif
> +  _BR_EXT_ID := $$(shell cat $$(_BR_EXT_DIR)/external.id 2>/dev/null)
> +  ifeq ($$(_BR_EXT_ID),)
> +    ifeq ($(BR2_EXTERNAL_NEED_ID),strict)
> +      $$(error BR2_EXTERNAL='$(1)' has no ID (in file 'external.id'),\
> +               mandatory to use more than one br2-external tree at once)
> +    endif # BR2_EXTERNAL_NEED_ID strict
> +  endif # No ID
> +  ifneq ($$(filter $$(_BR_EXT_ID),$$(BR_EXTERNAL_IDS)),)
> +    $$(error Duplicate ID '$$(_BR_EXT_ID)' in '$(1)', previously defined in '$$(BR2_EXTERNAL_$$(_BR_EXT_ID))')
> +  endif
> +  ifneq ($$(_BR_EXT_ID),)
> +    BR2_EXTERNAL_$$(_BR_EXT_ID) := $$(_BR_EXT_DIR)
> +    BR_EXTERNAL_IDS += $$(_BR_EXT_ID)
> +    EXTRA_ENV += BR2_EXTERNAL_$$(_BR_EXT_ID)=$$(_BR_EXT_DIR)
> +  endif # _BR_EXT_ID not empty
> +endef # BR2_EXTERNAL_VALIDATE

This is really long and not so pretty to look at. Perhaps all the
"validation" aspects can be handled in the shell script that generates
the .br2-external.in Config.in snippet?

>  # Now we are sure we have all the packages scanned and defined. We now
>  # check for each package in the list of enabled packages, that all its
> @@ -787,7 +813,7 @@ COMMON_CONFIG_ENV = \
>  	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
>  	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
>  	BR2_CONFIG=$(BR2_CONFIG) \
> -	BR2_EXTERNAL=$(BR2_EXTERNAL) \
> +	BR2_EXTERNAL="$(BR2_EXTERNAL)" \
>  	HOST_GCC_VERSION="$(HOSTCC_VERSION)" \
>  	BUILD_DIR=$(BUILD_DIR) \
>  	SKIP_LEGACY=
> @@ -895,12 +921,25 @@ $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
>  	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
>  		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
>  		printf 'menu "User-provided options"\n\n'; \
> -		if [ -z "$(BR2_EXTERNAL_ID)" ]; then \
> +		if [ -z "$(call strip,$(BR_EXTERNAL_IDS))" ]; then \
> +			printf 'comment "%s"\n\n' $(BR2_EXTERNAL); \
>  			printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
>  		else \
> -			printf 'config BR2_EXTERNAL_%s\n' $(BR2_EXTERNAL_ID); \
> -			printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \
> -			printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(BR2_EXTERNAL_ID); \
> +			$(foreach id,$(BR_EXTERNAL_IDS),\
> +				for i in $$(seq 1 80); do printf '#'; done; printf '\n\n'; \
> +				if [ $(words $(call strip,$(BR_EXTERNAL_IDS))) -gt 1 ]; then \
> +					printf 'menu "BR2_EXTERNAL_%s"\n\n' $(id); \
> +				else \
> +					printf 'comment "BR2_EXTERNAL_%s"\n\n' $(id); \
> +				fi; \
> +				printf 'comment "%s"\n\n' $(BR2_EXTERNAL_$(id)); \
> +				printf 'config BR2_EXTERNAL_%s\n' $(id); \
> +				printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(id)); \
> +				printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(id); \
> +				if [ $(words $(call strip,$(BR_EXTERNAL_IDS))) -gt 1 ]; then \
> +					printf 'endmenu # BR2_EXTERNAL_%s\n\n' $(id); \
> +				fi; ) \
> +			for i in $$(seq 1 80); do printf '#'; done; printf '\n\n'; \
>  		fi; \
>  		printf 'endmenu # User-provided options\n'; \

This is really where you want to start having a helper shell script IMO.

> -ifneq ($(wildcard $(BR2_EXTERNAL)/configs/*_defconfig),)
> +ifneq ($(wildcard $(patsubst %,%/configs/*_defconfig,$(BR2_EXTERNAL))),)

Everywhere you use a patsubst like this, a foreach would be more
appropriate I believe:

ifneq ($(wildcard $(foreach d,$(BR2_EXTERNAL),$(d)/configs/*_defconfig))

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 10/16 v3] core: introduce per br2-external ID

[Arnout Vandecappelle]

 I guess you're about to send a v3 so I'll try to still do a quick review of the
remaining things.


On 17-07-16 12:34, Yann E. MORIN wrote:
> This unique ID is used to construct a per br2-external tree variable,
> BR2_EXTERNAL_$(ID), which contains the path to the br2-external tree.
> 
> This variable is available both from Kconfig (set in the Kconfig
> snippet) and from the .mk files.
> 
> Also, display the BR2_EXTERNAL_$(ID) and its path as a comment in the
> menuconfig.
> 
> This will ultimately allow us to support multiple br2-external trees at
> once, with that ID (and thus BR2_EXTERNAL_$(ID)) uniquely defining which
> br2-external tree is being used.
> 
> Note: since the variables in the Makefile and in Kconfig are named the
> same, the one we computed early on (second hunk) will be overridden by
> the one in .config when we have it. Thus, even though they are set to
> the same raw value, the one from .config is quoted and, being included
> later in the Makefile, will take precedence, so we must un-quote it
> before we include the br2-external's makefile (third hunk). That's
> unfortunate, but there is no easy way around that as we want the two
> variables to be named the same in Makefile and Kconfig (and we can't
> ask the user to un-quote that variable himself either), hence the
> little dirty trick.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> ---
>  Makefile | 22 +++++++++++++++++++++-
>  1 file changed, 21 insertions(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index be2f586..72b55cb 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -148,6 +148,9 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
>  #
>  # When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
>  # line), the .br-external file is removed.
> +#
> +# If the br2-external tree defines its ID, then export the path in the
> +# BR2_EXTERNAL_$(ID) variable.

 I agree with Thomas that NAME is better than ID.

 The name can be anything, including, for example, MK - which would create a
conflict with BR2_EXTERNAL_MK. Therefore I think there should be an additional
prefix, e.g. BR2_EXTERNAL_DIR_$(NAME).

>  
>  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
>  -include $(BR2_EXTERNAL_FILE)
> @@ -160,6 +163,13 @@ else
>    endif
>    override BR2_EXTERNAL := $(_BR2_EXTERNAL)
>    $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
> +  ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
> +    BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)

 BR2_EXTERNAL_ID is an internal variable, so it should be EXTERNAL_NAME I think.
Admittedly many of the currently existing BR2_EXTERNAL variables wrongly have
the BR2 prefix, but let's not make it worse :-)


> +    ifeq ($(BR2_EXTERNAL_ID),)
> +      $(error $(BR2_EXTERNAL) has no ID (in file 'external.id'))
> +    endif
> +    BR2_EXTERNAL_$(BR2_EXTERNAL_ID) = $(BR2_EXTERNAL)
> +  endif
>    BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
>  endif
>  
> @@ -457,6 +467,10 @@ include boot/common.mk
>  include linux/linux.mk
>  include fs/common.mk
>  
> +# If the br2-external tree defines its ID, then the BR2_EXTERNAL_$(ID)
> +# variable is also present in .config, so it is quoted. We must unquote
> +# it before feeding it to the br2-external makefile.
> +BR2_EXTERNAL_$(BR2_EXTERNAL_ID) := $(call qstrip,$(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)))
>  # Nothing to include if no BR2_EXTERNAL tree in use
>  include $(BR2_EXTERNAL_MK)
>  
> @@ -882,7 +896,13 @@ $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
>  	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
>  		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
>  		printf 'menu "User-provided options"\n\n'; \
> -		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> +		if [ -z "$(BR2_EXTERNAL_ID)" ]; then \
> +			printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> +		else \
> +			printf 'config BR2_EXTERNAL_%s\n' $(BR2_EXTERNAL_ID); \
> +			printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \

 Why is this config still needed? It was there before because that's the only
way to use it in the 'source' statement, but now that we generate the external
Config.in, I don't think it's needed anymore. Which also removes the need for
the unquoting.

 Regards,
 Arnout

> +			printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(BR2_EXTERNAL_ID); \
> +		fi; \
>  		printf 'endmenu # User-provided options\n'; \
>  	fi >$@
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 10/16 v3] core: introduce per br2-external ID

[Yann E. MORIN]

Arnout, All,

On 2016-08-27 23:49 +0200, Arnout Vandecappelle spake thusly:
>  I guess you're about to send a v3 so I'll try to still do a quick review of the
> remaining things.

Not really "about" to. But I will, yes.

Thanks! :-)

> On 17-07-16 12:34, Yann E. MORIN wrote:
> > This unique ID is used to construct a per br2-external tree variable,
> > BR2_EXTERNAL_$(ID), which contains the path to the br2-external tree.
> > 
> > This variable is available both from Kconfig (set in the Kconfig
> > snippet) and from the .mk files.
> > 
> > Also, display the BR2_EXTERNAL_$(ID) and its path as a comment in the
> > menuconfig.
> > 
> > This will ultimately allow us to support multiple br2-external trees at
> > once, with that ID (and thus BR2_EXTERNAL_$(ID)) uniquely defining which
> > br2-external tree is being used.
> > 
> > Note: since the variables in the Makefile and in Kconfig are named the
> > same, the one we computed early on (second hunk) will be overridden by
> > the one in .config when we have it. Thus, even though they are set to
> > the same raw value, the one from .config is quoted and, being included
> > later in the Makefile, will take precedence, so we must un-quote it
> > before we include the br2-external's makefile (third hunk). That's
> > unfortunate, but there is no easy way around that as we want the two
> > variables to be named the same in Makefile and Kconfig (and we can't
> > ask the user to un-quote that variable himself either), hence the
> > little dirty trick.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > Cc: Peter Korsgaard <jacmet@uclibc.org>
> > Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
> > ---
> >  Makefile | 22 +++++++++++++++++++++-
> >  1 file changed, 21 insertions(+), 1 deletion(-)
> > 
> > diff --git a/Makefile b/Makefile
> > index be2f586..72b55cb 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -148,6 +148,9 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
> >  #
> >  # When BR2_EXTERNAL is set to an empty value (e.g. explicitly in command
> >  # line), the .br-external file is removed.
> > +#
> > +# If the br2-external tree defines its ID, then export the path in the
> > +# BR2_EXTERNAL_$(ID) variable.
> 
>  I agree with Thomas that NAME is better than ID.
> 
>  The name can be anything, including, for example, MK - which would create a
> conflict with BR2_EXTERNAL_MK. Therefore I think there should be an additional
> prefix, e.g. BR2_EXTERNAL_DIR_$(NAME).

OK, will see to it.

> >  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
> >  -include $(BR2_EXTERNAL_FILE)
> > @@ -160,6 +163,13 @@ else
> >    endif
> >    override BR2_EXTERNAL := $(_BR2_EXTERNAL)
> >    $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
> > +  ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
> > +    BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
> 
>  BR2_EXTERNAL_ID is an internal variable, so it should be EXTERNAL_NAME I think.
> Admittedly many of the currently existing BR2_EXTERNAL variables wrongly have
> the BR2 prefix, but let's not make it worse :-)

OK, fair enough. Will see to it.

> > +    ifeq ($(BR2_EXTERNAL_ID),)
> > +      $(error $(BR2_EXTERNAL) has no ID (in file 'external.id'))
> > +    endif
> > +    BR2_EXTERNAL_$(BR2_EXTERNAL_ID) = $(BR2_EXTERNAL)
> > +  endif
> >    BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
> >  endif
> >  
> > @@ -457,6 +467,10 @@ include boot/common.mk
> >  include linux/linux.mk
> >  include fs/common.mk
> >  
> > +# If the br2-external tree defines its ID, then the BR2_EXTERNAL_$(ID)
> > +# variable is also present in .config, so it is quoted. We must unquote
> > +# it before feeding it to the br2-external makefile.
> > +BR2_EXTERNAL_$(BR2_EXTERNAL_ID) := $(call qstrip,$(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)))
> >  # Nothing to include if no BR2_EXTERNAL tree in use
> >  include $(BR2_EXTERNAL_MK)
> >  
> > @@ -882,7 +896,13 @@ $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
> >  	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
> >  		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
> >  		printf 'menu "User-provided options"\n\n'; \
> > -		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> > +		if [ -z "$(BR2_EXTERNAL_ID)" ]; then \
> > +			printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> > +		else \
> > +			printf 'config BR2_EXTERNAL_%s\n' $(BR2_EXTERNAL_ID); \
> > +			printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \
> 
>  Why is this config still needed? It was there before because that's the only
> way to use it in the 'source' statement, but now that we generate the external
> Config.in, I don't think it's needed anymore. Which also removes the need for
> the unquoting.

Hmmm.. As far as I can see, it is still needed, because the Config.in
files in the br2-external tree may want to include other Config.in
files, no?

Like for example (with the variable renamed as per above):

    .../ext-tree/Config.in
        source "$BR2_EXTERNAL_DIR_FOO/package/pkg-1/Config.in"
        source "$BR2_EXTERNAL_DIR_FOO/package/pkg-2/Config.in"

So it is stil needed to have the path to each br2-external paths from
Kconfig.

Unless I missed something?

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 05/16 v3] core: move rule to create basic directories

[Yann E. MORIN]

Thomas, All,

On 2016-08-27 21:41 +0200, Thomas Petazzoni spake thusly:
> On Sun, 17 Jul 2016 12:34:25 +0200, Yann E. MORIN wrote:
> > Some of those directories will be needed even during configuration, like
> > BUILD_DIR, where we'll store the generated kconfig snippet.
> > 
> > So, move the rule to create them outside the HAVE_CONFIG block.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > Cc: Peter Korsgaard <jacmet@uclibc.org>
> 
> I've applied this patch to next, after fixing the typo in the commit
> log noticed by Romain Naour. One question below, though.

Thanks! :-)

> > +################################################################################
> > +#
> > +# staging and target directories do NOT list these as
> > +# dependencies anywhere else
> > +#
> > +################################################################################
> 
> Do you understand what this comment means? It seems to date back from
> the "major rework" commit 08782ae7d8812933ad3f932132955ea1c0fb1e52 in
> 2002, and I don't really understand the meaning of this comment.

Neither do I... And as a once-colleague-of-mine said:

    When you don't know, don't touch!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 08/16 v3] core: do not hard-code inclusion of br2-external in Kconfig

[Yann E. MORIN]

Thomas, All,

On 2016-08-27 22:00 +0200, Thomas Petazzoni spake thusly:
> On Sun, 17 Jul 2016 12:34:28 +0200, Yann E. MORIN wrote:
> >  .PHONY: $(BUILD_DIR)/.br2-external.in
> >  $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
> > -	@touch $@
> > +	$(Q)( \
> > +		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
> > +		printf 'menu "User-provided options"\n'; \
> > +		printf '\tdepends on BR2_EXTERNAL != "support/dummy-external"\n\n'; \
> > +		printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> > +		printf 'endmenu # User-provided options\n'; \
> > +	) >$@
> 
> Shouldn't the logic that generates the .br2-external.in be moved to a
> shell script?
> 
> I'm generally in favor of having most of the logic in make, but here,
> it is really ugly. Did you try having this in a shell script?

Well, for the above, it's OK to move to a shell script, indeed.

But once we have multiple br2-external trees, we need, for each of them:
  - the path to the br2-external tree,
  - the name of that tree.

So, if offloading this to a script, we'd need to either:

  - construct a list of tuples of {name,path} to the script, maybe
    something like:

        MY_NAME:/path/to/my-name-ext/ FOO:/path/to/foo-ext/ ...
or
  - rescann and re-extract the names from each br2-external tree.

Not very nice in either cases... :-/

> Also, what about using readlink or realpath to convert the path to an
> absolute path?

You meant the "cd blabla && pwd" trick?

I kept it because that's the way it is done today...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 10/16 v3] core: introduce per br2-external ID

[Yann E. MORIN]

Thomas, All,

On 2016-08-27 22:10 +0200, Thomas Petazzoni spake thusly:
> On Sun, 17 Jul 2016 12:34:30 +0200, Yann E. MORIN wrote:
> > This unique ID is used to construct a per br2-external tree variable,
> > BR2_EXTERNAL_$(ID), which contains the path to the br2-external tree.
> 
> I would suggest to use the terminology "name" rather than "ID". I
> think it's much easier to understand when using "name" that using "ID".
> It's a bit longer, but not much, and will make things clearer.

I'm OK with that. I will change.

Cordialement,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 11/16 v3] docs/manual: document the br2-external ID

[Yann E. MORIN]

Thomas, All,

On 2016-08-27 22:13 +0200, Thomas Petazzoni spake thusly:
> On Sun, 17 Jul 2016 12:34:31 +0200, Yann E. MORIN wrote:
> 
> > diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
> > index 9ad177d..839d1b1 100644
> > --- a/docs/manual/customize-outside-br.txt
> > +++ b/docs/manual/customize-outside-br.txt
> > @@ -57,19 +57,50 @@ Or disable the usage of external definitions:
> >  buildroot/ $ make BR2_EXTERNAL= xconfig
> >  -----
> >  
> > -+BR2_EXTERNAL+ allows three different things:
> > +A +BR2_EXTERNAL+ tree must contain at least those three files:
> > +
> > ++external.id+::
> 
> I thought this file was optional when there was a single external tree,
> but you're saying it's mandatory here. Is it because the "no
> external.id file" case is only supported for backward compat, and we
> expect that new users write an external.id, regardless of whether they
> are using a single or multiple external trees ?

Exactly, as explained in the commit log:

    Even though it is still possible to write a br2-external tree that
    is not multi-aware, we only document it with an ID.

> > +   That file should contain the ID for the +BR2_EXTERNAL+ tree.
> > +   That ID is used to construct the +BR2_EXTERNAL_$(ID)+ variable,
> > +   available in +Config.in+ and +external.mk+ (see below), so that ID
> > +   must be a valid make and Kconfig variable. Buildroot sets
> > +   +BR2_EXTERNAL_$(ID)+ to the path of the +BR2_EXTERNAL+ tree.
> > +   The format for this file is a single line with the ID.
> > ++
> > +Examples of an +external.mk+ file that declares the ID +FOO+:
> 
> Huh? Is it really external.mk that contains the ID ?

Good catch!

> > ++
> > +----
> > +$ cat external.mk
> 
> Ditto.

Ditto. ;-)

> > +FOO
> > +----
> > ++
> > +Examples of IDs and the corresponding +BR2_EXTERNAL_$(ID)+ variable:
> > ++
> > +  * +FOO+ -> +BR2_EXTERNAL_FOO+
> > +  * +BAR_42+ -> +BR2_EXTERNAL_BAR_42+
> > +  * ...
> 
> Maybe mention which characters are allowed and which ones are not
> allowed?

I thought I did address this, a bit above, when describing the
external.id file:

    [...] ID must be a valid make and Kconfig variable.

Isn't that enough?

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 12/16 v3] core: handle .br-external in a make rule

[Yann E. MORIN]

Thomas, All,

On 2016-08-27 22:16 +0200, Thomas Petazzoni spake thusly:
> On Sun, 17 Jul 2016 12:34:32 +0200, Yann E. MORIN wrote:
> 
> >  BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
> >  -include $(BR2_EXTERNAL_FILE)

Here ^^^^

> > -ifeq ($(BR2_EXTERNAL),)
> > -  $(shell rm -f $(BR2_EXTERNAL_FILE))
> > -else
> > +ifneq ($(BR2_EXTERNAL),)
> >    _BR2_EXTERNAL = $(shell cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd)
> >    ifeq ($(_BR2_EXTERNAL),)
> >      $(error BR2_EXTERNAL='$(BR2_EXTERNAL)' does not exist, relative to $(TOPDIR))
> >    endif
> >    override BR2_EXTERNAL := $(_BR2_EXTERNAL)
> > -  $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
> >    ifneq ($(wildcard $(BR2_EXTERNAL)/external.id),)
> >      BR2_EXTERNAL_ID := $(shell cat $(BR2_EXTERNAL)/external.id 2>/dev/null)
> >      ifeq ($(BR2_EXTERNAL_ID),)
> > @@ -173,6 +167,11 @@ else
> >    BR2_EXTERNAL_MK = $(BR2_EXTERNAL)/external.mk
> >  endif
> >  
> > +# This needs to be *after* we compute BR_EXTERNAL, above.
> > +.PHONY: $(BR2_EXTERNAL_FILE)
> > +$(BR2_EXTERNAL_FILE):
> > +	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@
> > +
> 
> I'm probably missing something, but where is this make target being
> "triggered" ?

When make encounters a -include directive, and the file to be included
is the target of a rule, then make will trigger the rule before including
the file, even if the rule is defined after the inclusion (make parses
makefiles in multiple passes).

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 10/16 v3] core: introduce per br2-external ID

[Thomas Petazzoni]

Hello,

On Sat, 27 Aug 2016 23:59:51 +0200, Yann E. MORIN wrote:

> > > +			printf 'config BR2_EXTERNAL_%s\n' $(BR2_EXTERNAL_ID); \
> > > +			printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \  
> > 
> >  Why is this config still needed? It was there before because that's the only
> > way to use it in the 'source' statement, but now that we generate the external
> > Config.in, I don't think it's needed anymore. Which also removes the need for
> > the unquoting.  
> 
> Hmmm.. As far as I can see, it is still needed, because the Config.in
> files in the br2-external tree may want to include other Config.in
> files, no?
> 
> Like for example (with the variable renamed as per above):
> 
>     .../ext-tree/Config.in
>         source "$BR2_EXTERNAL_DIR_FOO/package/pkg-1/Config.in"
>         source "$BR2_EXTERNAL_DIR_FOO/package/pkg-2/Config.in"
> 
> So it is stil needed to have the path to each br2-external paths from
> Kconfig.
> 
> Unless I missed something?

No, I think you're right. I don't see how we can do without this config
variable, which is needed to communicate the BR2_EXTERNAL locations
from the make logic down to the Kconfig logic, in order to allow
Config.in files to include other Config.in files located in
BR2_EXTERNAL.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 12/16 v3] core: handle .br-external in a make rule

[Thomas Petazzoni]

Hello,

On Sun, 28 Aug 2016 00:22:35 +0200, Yann E. MORIN wrote:

> > I'm probably missing something, but where is this make target being
> > "triggered" ?  
> 
> When make encounters a -include directive, and the file to be included
> is the target of a rule, then make will trigger the rule before including
> the file, even if the rule is defined after the inclusion (make parses
> makefiles in multiple passes).

Ah, OK. Didn't know that arguments of the include directive where
handled as make targets. Make sense, thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Hello,
 > On Sun, 17 Jul 2016 12:34:26 +0200, Yann E. MORIN wrote:

 >> +.PHONY: $(BUILD_DIR)/.br2-external.in

 > Why is a real file that exists on the filesystem a phony target?

 >> +# Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
 >> +MANUAL_DEPENDENCIES += prepare-kconfig

 > It's a bit crazy the complexity that we have just to allow our manual
 > to have the list of all packages in Buildroot. I'm not even sure this
 > section of the manual is that useful, to be honest. But oh well, that's
 > supported today, so we have to continue support it.

Why do we _HAVE_ to continue to support it?

If we don't find the complexity worth the added value (I personally
don't, I think we could just ask people to look under package/ for the
list of packages), then we should just remove it.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Thomas Petazzoni]

Hello,

On Sun, 28 Aug 2016 22:50:22 +0200, Peter Korsgaard wrote:

>  > It's a bit crazy the complexity that we have just to allow our manual
>  > to have the list of all packages in Buildroot. I'm not even sure this
>  > section of the manual is that useful, to be honest. But oh well, that's
>  > supported today, so we have to continue support it.  
> 
> Why do we _HAVE_ to continue to support it?
> 
> If we don't find the complexity worth the added value (I personally
> don't, I think we could just ask people to look under package/ for the
> list of packages), then we should just remove it.

I wouldn't mind if we were to remove this. I think the original intent
was to give some "exposure" as to what is available in Buildroot, so
that people just skimming through our documentation can see the wide
variety of packages we have.

The other thing that I find quite complex for no super benefit is the
whole "doc-asciidoc" infrastructure, but that's another story.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 >> Why do we _HAVE_ to continue to support it?
 >> 
 >> If we don't find the complexity worth the added value (I personally
 >> don't, I think we could just ask people to look under package/ for the
 >> list of packages), then we should just remove it.

 > I wouldn't mind if we were to remove this. I think the original intent
 > was to give some "exposure" as to what is available in Buildroot, so
 > that people just skimming through our documentation can see the wide
 > variety of packages we have.

Yes, but the list is really long and overwhelming. As it stands, the
list of packages takes up more than half the size of the manual. I think
just asking people to browse packages/ (or make menuconfig) is more
sane.

 > The other thing that I find quite complex for no super benefit is the
 > whole "doc-asciidoc" infrastructure, but that's another story.

Yeah, having this infrastructure just for the manual is maybe a bit
overkill.

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Thomas Petazzoni]

Hello,

On Mon, 29 Aug 2016 09:50:13 +0200, Peter Korsgaard wrote:

> Yes, but the list is really long and overwhelming. As it stands, the
> list of packages takes up more than half the size of the manual. I think
> just asking people to browse packages/ (or make menuconfig) is more
> sane.

Right.

>  > The other thing that I find quite complex for no super benefit is the
>  > whole "doc-asciidoc" infrastructure, but that's another story.  
> 
> Yeah, having this infrastructure just for the manual is maybe a bit
> overkill.

I agree, though some people wanted to build several manuals, if I
remember correctly.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 13/16 v3] core: add support for multiple br2-external trees

[Yann E. MORIN]

Thomas, All,

On 2016-08-27 22:20 +0200, Thomas Petazzoni spake thusly:
> On Sun, 17 Jul 2016 12:34:33 +0200, Yann E. MORIN wrote:
> 
> > What we do is to treat BR2_EXTERNAL as a space-separated list of paths,
> > which we iterate to construct:
> 
> As discussed on IRC, is the space-separated list the best choice? Most
> environment variables that contain paths in Unix are colon-separated
> instead (PATH, LD_LIBRARY_PATH, PKG_CONFIG_PATH, etc, etc.)
> 
> Making it space-separated makes it a bit annoying IMO.

Already changed here; will be part of the next spin.

> > -# This needs to be *after* we compute BR_EXTERNAL, above.
> >  .PHONY: $(BR2_EXTERNAL_FILE)
> >  $(BR2_EXTERNAL_FILE):
> > -	@echo BR2_EXTERNAL ?= $(BR_EXTERNAL) >$@
> > +	@echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) >$@
> > +
> > +# Those two variables need to be defined as simply-expanded variables, they
> > +# can't be recursively-expanded, because the values they are assigned change
> > +# with each iteration of the foreach, below.
> > +BR_EXTERNAL_IDS :=
> > +EXTRA_ENV :=
> > +
> > +# If there is no or one br2-external trees used, then we don't require (but
> > +# accept) an ID; otherwise (i.e. there are two or more br2-external trees)
> > +# we require they do all define their ID.
> > +ifeq ($(filter-out 0 1,$(words $(BR2_EXTERNAL))),)
> > +BR2_EXTERNAL_NEED_ID := loose
> > +else
> > +BR2_EXTERNAL_NEED_ID := strict
> > +endif
> > +
> > +# Validate the br2-external tree passed as $(1):
> > +# - check the directory actually exists
> > +# - check if we need and have a non-empty ID
> > +# - check the ID is not a duplicate
> > +# - set variables for later use
> > +define BR2_EXTERNAL_VALIDATE
> > +  _BR_EXT_DIR := $$(shell cd $(1) >/dev/null 2>&1 && pwd)
> > +  ifeq ($$(_BR_EXT_DIR),)
> > +    $$(error BR2_EXTERNAL='$(1)' does not exist, relative to $$(TOPDIR))
> > +  endif
> > +  _BR_EXT_ID := $$(shell cat $$(_BR_EXT_DIR)/external.id 2>/dev/null)
> > +  ifeq ($$(_BR_EXT_ID),)
> > +    ifeq ($(BR2_EXTERNAL_NEED_ID),strict)
> > +      $$(error BR2_EXTERNAL='$(1)' has no ID (in file 'external.id'),\
> > +               mandatory to use more than one br2-external tree at once)
> > +    endif # BR2_EXTERNAL_NEED_ID strict
> > +  endif # No ID
> > +  ifneq ($$(filter $$(_BR_EXT_ID),$$(BR_EXTERNAL_IDS)),)
> > +    $$(error Duplicate ID '$$(_BR_EXT_ID)' in '$(1)', previously defined in '$$(BR2_EXTERNAL_$$(_BR_EXT_ID))')
> > +  endif
> > +  ifneq ($$(_BR_EXT_ID),)
> > +    BR2_EXTERNAL_$$(_BR_EXT_ID) := $$(_BR_EXT_DIR)
> > +    BR_EXTERNAL_IDS += $$(_BR_EXT_ID)
> > +    EXTRA_ENV += BR2_EXTERNAL_$$(_BR_EXT_ID)=$$(_BR_EXT_DIR)
> > +  endif # _BR_EXT_ID not empty
> > +endef # BR2_EXTERNAL_VALIDATE
> 
> This is really long and not so pretty to look at. Perhaps all the
> "validation" aspects can be handled in the shell script that generates
> the .br2-external.in Config.in snippet?

Well, at first I thought that mught be a good idea.

However, we also need to generate makefile-level variables, so that each
br2-external tree can access its files:

    BR2_EXTERNAL_$$(_BR_EXT_ID) := $$(_BR_EXT_DIR)

is something we can not offload to a shell script; it has to be done in
the Makefile.

One solution to offload that to a script, would be to generate another
makefile fragment, and I'm pretty sure you would not like it much.

However, there might be a solution that is not too complex: we already
generate $(BASE_DIR)/.br-external, today with a simple echo. But we
could turn it into:

    BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
    -include $(BR2_EXTERNAL_FILE)

    $(BR2_EXTERNAL_FILE):
        support/scripts/mk-br-external -x "$(BR2_EXTERNAL)" -m -o $(@)

    BR2_KCONFIG_SNIPPET = $(BUILD_DIR)/.br2-external.in
    $(BR2_KCONFIG_SNIPPET):
        support/scripts/mk-br-external -x "$(BR2_EXTERNAL)" -k -o $(@)

And then have support/scripts/mk-br-external bail out if BR2_EXTERNAL is
incorrect, which would have make error out when generating .br-external.

Otherwise, it would:

  - for BR2_EXTERNAL_FILE, store the value of BR2_EXTERNAL like we do
    for now, plus generate the needed variables BR2_EXTERNAL_$(ID) ;

  - for BR2_KCONFIG_SNIPPET, generate the Kconfig fragment which is
    currently done in the makefile code (your comment below).

> >  # Now we are sure we have all the packages scanned and defined. We now
> >  # check for each package in the list of enabled packages, that all its
> > @@ -787,7 +813,7 @@ COMMON_CONFIG_ENV = \
> >  	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
> >  	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
> >  	BR2_CONFIG=$(BR2_CONFIG) \
> > -	BR2_EXTERNAL=$(BR2_EXTERNAL) \
> > +	BR2_EXTERNAL="$(BR2_EXTERNAL)" \
> >  	HOST_GCC_VERSION="$(HOSTCC_VERSION)" \
> >  	BUILD_DIR=$(BUILD_DIR) \
> >  	SKIP_LEGACY=
> > @@ -895,12 +921,25 @@BR2_EXTERNAL_FILE $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
> >  	$(Q)if [ -n '$(BR2_EXTERNAL)' ]; then \
> >  		printf "#\n# Automatically generated file; DO NOT EDIT.\n#\n\n"; \
> >  		printf 'menu "User-provided options"\n\n'; \
> > -		if [ -z "$(BR2_EXTERNAL_ID)" ]; then \
> > +		if [ -z "$(call strip,$(BR_EXTERNAL_IDS))" ]; then \
> > +			printf 'comment "%s"\n\n' $(BR2_EXTERNAL); \
> >  			printf 'source "%s/Config.in"\n\n' $$(cd $(BR2_EXTERNAL) >/dev/null 2>&1 && pwd); \
> >  		else \
> > -			printf 'config BR2_EXTERNAL_%s\n' $(BR2_EXTERNAL_ID); \
> > -			printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(BR2_EXTERNAL_ID)); \
> > -			printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(BR2_EXTERNAL_ID); \
> > +			$(foreach id,$(BR_EXTERNAL_IDS),\
> > +				for i in $$(seq 1 80); do printf '#'; done; printf '\n\n'; \
> > +				if [ $(words $(call strip,$(BR_EXTERNAL_IDS))) -gt 1 ]; then \
> > +					printf 'menu "BR2_EXTERNAL_%s"\n\n' $(id); \
> > +				else \
> > +					printf 'comment "BR2_EXTERNAL_%s"\n\n' $(id); \
> > +				fi; \
> > +				printf 'comment "%s"\n\n' $(BR2_EXTERNAL_$(id)); \
> > +				printf 'config BR2_EXTERNAL_%s\n' $(id); \
> > +				printf '\tstring\n\tdefault "%s"\n\n' $(BR2_EXTERNAL_$(id)); \
> > +				printf 'source "$$BR2_EXTERNAL_%s/Config.in"\n\n' $(id); \
> > +				if [ $(words $(call strip,$(BR_EXTERNAL_IDS))) -gt 1 ]; then \
> > +					printf 'endmenu # BR2_EXTERNAL_%s\n\n' $(id); \
> > +				fi; ) \
> > +			for i in $$(seq 1 80); do printf '#'; done; printf '\n\n'; \
> >  		fi; \
> >  		printf 'endmenu # User-provided options\n'; \
> 
> This is really where you want to start having a helper shell script IMO.
> 
> > -ifneq ($(wildcard $(BR2_EXTERNAL)/configs/*_defconfig),)
> > +ifneq ($(wildcard $(patsubst %,%/configs/*_defconfig,$(BR2_EXTERNAL))),)
> 
> Everywhere you use a patsubst like this, a foreach would be more
> appropriate I believe:
> 
> ifneq ($(wildcard $(foreach d,$(BR2_EXTERNAL),$(d)/configs/*_defconfig))

Ok, can do.

Thanks! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Arnout Vandecappelle]

On 29-08-16 12:02, Thomas Petazzoni wrote:
> Hello,
> 
> On Mon, 29 Aug 2016 09:50:13 +0200, Peter Korsgaard wrote:
> 
>> Yes, but the list is really long and overwhelming. As it stands, the
>> list of packages takes up more than half the size of the manual. I think
>> just asking people to browse packages/ (or make menuconfig) is more
>> sane.
> 
> Right.

 We could replace it with a reference to http://autobuild.buildroot.net/stats/


>>  > The other thing that I find quite complex for no super benefit is the
>>  > whole "doc-asciidoc" infrastructure, but that's another story.  
>>
>> Yeah, having this infrastructure just for the manual is maybe a bit
>> overkill.

 I introduced it back in the day to reduce the size of manual.mk a little. Back
then I still believed that all code duplication is evil. Now I think that the
manual generation logic would be a lot easier to understand if it was all
unrolled...

> I agree, though some people wanted to build several manuals, if I
> remember correctly.

 I never saw the use of that, TBH. But I guess it's out there now so we should
keep it?

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Yann E. MORIN]

Peter, Thomas, All,

On 2016-08-28 22:50 +0200, Peter Korsgaard spake thusly:
> >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
>  > On Sun, 17 Jul 2016 12:34:26 +0200, Yann E. MORIN wrote:
>  >> +# Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
>  >> +MANUAL_DEPENDENCIES += prepare-kconfig
> 
>  > It's a bit crazy the complexity that we have just to allow our manual
>  > to have the list of all packages in Buildroot. I'm not even sure this
>  > section of the manual is that useful, to be honest. But oh well, that's
>  > supported today, so we have to continue support it.
> 
> Why do we _HAVE_ to continue to support it?
> 
> If we don't find the complexity worth the added value (I personally
> don't, I think we could just ask people to look under package/ for the
> list of packages), then we should just remove it.

I'm not entirely against removing the list of out packages from our
manual.

Yet, for some users of br2-external trees, they may still want to have
the list of their packages in their manuals, build with the asciidoc
infrastructure.

It is entirely imaginable that such users of br2-external have packages
annotated with some documentation for those packages, and want to
include that doc in their manual, so need to generate that list from
Kconfig.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Yann E. MORIN]

Arnout, All,

On 2016-08-30 00:14 +0200, Arnout Vandecappelle spake thusly:
> On 29-08-16 12:02, Thomas Petazzoni wrote:
> > Hello,
> > 
> > On Mon, 29 Aug 2016 09:50:13 +0200, Peter Korsgaard wrote:
> > 
> >> Yes, but the list is really long and overwhelming. As it stands, the
> >> list of packages takes up more than half the size of the manual. I think
> >> just asking people to browse packages/ (or make menuconfig) is more
> >> sane.
> > Right.
>  We could replace it with a reference to http://autobuild.buildroot.net/stats/

I'm OK with that.

> >>  > The other thing that I find quite complex for no super benefit is the
> >>  > whole "doc-asciidoc" infrastructure, but that's another story.  
> >>
> >> Yeah, having this infrastructure just for the manual is maybe a bit
> >> overkill.
> 
>  I introduced it back in the day to reduce the size of manual.mk a little. Back
> then I still believed that all code duplication is evil. Now I think that the
> manual generation logic would be a lot easier to understand if it was all
> unrolled...
> 
> > I agree, though some people wanted to build several manuals, if I
> > remember correctly.
> 
>  I never saw the use of that, TBH. But I guess it's out there now so we should
> keep it?

I've seen it. And I use it as well:
    http://ymorin.is-a-geek.org/git/buildroot.config/
    http://ymorin.is-a-geek.org/download/tmp/br.config/manual/

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Arnout Vandecappelle]

On 30-08-16 00:16, Yann E. MORIN wrote:
> Peter, Thomas, All,
> 
> On 2016-08-28 22:50 +0200, Peter Korsgaard spake thusly:
>>>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
>>  > On Sun, 17 Jul 2016 12:34:26 +0200, Yann E. MORIN wrote:
>>  >> +# Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
>>  >> +MANUAL_DEPENDENCIES += prepare-kconfig
>>
>>  > It's a bit crazy the complexity that we have just to allow our manual
>>  > to have the list of all packages in Buildroot. I'm not even sure this
>>  > section of the manual is that useful, to be honest. But oh well, that's
>>  > supported today, so we have to continue support it.
>>
>> Why do we _HAVE_ to continue to support it?
>>
>> If we don't find the complexity worth the added value (I personally
>> don't, I think we could just ask people to look under package/ for the
>> list of packages), then we should just remove it.
> 
> I'm not entirely against removing the list of out packages from our
> manual.
> 
> Yet, for some users of br2-external trees, they may still want to have
> the list of their packages in their manuals, build with the asciidoc
> infrastructure.
> 
> It is entirely imaginable that such users of br2-external have packages
> annotated with some documentation for those packages, and want to
> include that doc in their manual, so need to generate that list from
> Kconfig.

 I'm not really enthousiastic about keeping complexity in our Makefiles for
imaginary users that may want to have such and such feature :-P

 I'm very enthousiastic when people contribute some feature that I'm never going
to use myself - but in that case, we know that there is at least one user (the
contributor). But here, we really have added a feature with no known users. At
the time, it didn't add (much) complexity, but more and more it's starting to
weigh down I think.

 For sure, the asciidoc support is even more difficult to understand and
maintain than the normal package infras.


 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Yann E. MORIN]

Arnout, All,

On 2016-08-30 00:46 +0200, Arnout Vandecappelle spake thusly:
> On 30-08-16 00:16, Yann E. MORIN wrote:
> > Peter, Thomas, All,
> > 
> > On 2016-08-28 22:50 +0200, Peter Korsgaard spake thusly:
> >>>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> >>  > On Sun, 17 Jul 2016 12:34:26 +0200, Yann E. MORIN wrote:
> >>  >> +# Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
> >>  >> +MANUAL_DEPENDENCIES += prepare-kconfig
> >>
> >>  > It's a bit crazy the complexity that we have just to allow our manual
> >>  > to have the list of all packages in Buildroot. I'm not even sure this
> >>  > section of the manual is that useful, to be honest. But oh well, that's
> >>  > supported today, so we have to continue support it.
> >>
> >> Why do we _HAVE_ to continue to support it?
> >>
> >> If we don't find the complexity worth the added value (I personally
> >> don't, I think we could just ask people to look under package/ for the
> >> list of packages), then we should just remove it.
> > 
> > I'm not entirely against removing the list of out packages from our
> > manual.
> > 
> > Yet, for some users of br2-external trees, they may still want to have
> > the list of their packages in their manuals, build with the asciidoc
> > infrastructure.
> > 
> > It is entirely imaginable that such users of br2-external have packages
> > annotated with some documentation for those packages, and want to
> > include that doc in their manual, so need to generate that list from
> > Kconfig.
> 
>  I'm not really enthousiastic about keeping complexity in our Makefiles for
> imaginary users that may want to have such and such feature :-P
> 
>  I'm very enthousiastic when people contribute some feature that I'm never going
> to use myself - but in that case, we know that there is at least one user (the
> contributor). But here, we really have added a feature with no known users.

Have you read what I wrote: I *am* using it.

    http://lists.busybox.net/pipermail/buildroot/2016-August/171055.html

> At
> the time, it didn't add (much) complexity, but more and more it's starting to
> weigh down I think.
> 
>  For sure, the asciidoc support is even more difficult to understand and
> maintain than the normal package infras.

And I seem to have a good track-record at looking at it:

    $ git log --follow --pretty=%an package/doc-asciidoc.mk |sort |uniq -c  |sort -k1nr,2
         22 Yann E. MORIN
          8 Samuel Martin
          7 Thomas De Schampheleire
          2 Thomas Petazzoni
          1 Atul Singh
          1 J?r?me Pouiller

So, please, please. let's keep this asciidoc infra in place.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

Hi,

>> For sure, the asciidoc support is even more difficult to understand and
 >> maintain than the normal package infras.

 > And I seem to have a good track-record at looking at it:

 >     $ git log --follow --pretty=%an package/doc-asciidoc.mk |sort |uniq -c  |sort -k1nr,2
 >          22 Yann E. MORIN
 >           8 Samuel Martin
 >           7 Thomas De Schampheleire
 >           2 Thomas Petazzoni
 >           1 Atul Singh
 >           1 J?r?me Pouiller

 > So, please, please. let's keep this asciidoc infra in place.

Yes, as long as it doesn't cause any problems I don't see any reason to
get rid of it.

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Arnout Vandecappelle]

On 30-08-16 11:00, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2016-08-30 00:46 +0200, Arnout Vandecappelle spake thusly:
>> On 30-08-16 00:16, Yann E. MORIN wrote:
[snip]
>>> It is entirely imaginable that such users of br2-external have packages
>>> annotated with some documentation for those packages, and want to
>>> include that doc in their manual, so need to generate that list from
>>> Kconfig.
>>
>>  I'm not really enthousiastic about keeping complexity in our Makefiles for
>> imaginary users that may want to have such and such feature :-P
>>
>>  I'm very enthousiastic when people contribute some feature that I'm never going
>> to use myself - but in that case, we know that there is at least one user (the
>> contributor). But here, we really have added a feature with no known users.
> 
> Have you read what I wrote: I *am* using it.
> 
>     http://lists.busybox.net/pipermail/buildroot/2016-August/171055.html

 Er, I haven't read that message because I didn't receive it... Disappearing
mail, that's something that scares the hell out of me... It seems to be the only
one that got lost however. Weird.

> 
>> At
>> the time, it didn't add (much) complexity, but more and more it's starting to
>> weigh down I think.
>>
>>  For sure, the asciidoc support is even more difficult to understand and
>> maintain than the normal package infras.
> 
> And I seem to have a good track-record at looking at it:
> 
>     $ git log --follow --pretty=%an package/doc-asciidoc.mk |sort |uniq -c  |sort -k1nr,2
>          22 Yann E. MORIN
>           8 Samuel Martin
>           7 Thomas De Schampheleire
>           2 Thomas Petazzoni
>           1 Atul Singh
>           1 J?r?me Pouiller
> 
> So, please, please. let's keep this asciidoc infra in place.

 As I wrote, if there is an identified user, let's certainly keep it. Though
honestly I don't think the list of projects is very useful in the brsh manual
either :-)

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 11/16 v3] docs/manual: document the br2-external ID

[Arnout Vandecappelle]

On 17-07-16 12:34, Yann E. MORIN wrote:
> Update the manual with the new external.id mandatory file.
> 
> Take the opportunity to add a section listing all mandatory files,
> Config.in, external.mk and the new external.id, instead of just hinting
> about them in the external package recipes section.
> 
> Change the example to use the ID-suffixed variable instead of the raw
> BR2_EXTERNAL variable. Even though it is still possible to write a
> br2-external tree that is not multi-aware, we only document it with an
> ID.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Jeremy Rosen <jeremy.rosen@openwide.fr>
> 
> ---
> Changes v2 -> v3:
>   - better document the content of the ID file  (Jeremy)
> ---
>  docs/manual/customize-outside-br.txt | 59 ++++++++++++++++++++++++++----------
>  1 file changed, 43 insertions(+), 16 deletions(-)
> 
> diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
> index 9ad177d..839d1b1 100644
> --- a/docs/manual/customize-outside-br.txt
> +++ b/docs/manual/customize-outside-br.txt
> @@ -57,19 +57,50 @@ Or disable the usage of external definitions:
>  buildroot/ $ make BR2_EXTERNAL= xconfig
>  -----
>  
> -+BR2_EXTERNAL+ allows three different things:
> +A +BR2_EXTERNAL+ tree must contain at least those three files:
                                               ^^^^^the following

 Particularly in the context of multiple external trees, but also because we now
have BR2_EXTERNAL on the one hand and BR2_EXTERNAL_DIR_FOO on the other hand, I
would not use +BR2_EXTERNAL+ anymore in the normal text, but rather 'an external
tree'. So:

An external tree must contain at least the following three files:

 But also higher up:

 * outside of the Buildroot tree, using an external tree. An external
   tree allows to keep package recipes, board support and configuration
   files outside of the Buildroot tree, while still having them nicely
   integrated in the build logic. This section explains how to use
   external trees.


 So use +BR2_EXTERNAL+ only to refer to the environment variable.

 This is just a proposal, of course :-). Below I'll also indicate where I think
it should change.


> +
> ++external.id+::

 I guess if we change ID -> name, this should be external.name? Actually,
external.id sounds better to me.

> +   That file should contain the ID for the +BR2_EXTERNAL+ tree.
                                      ^^^of   ^^^^^^^^^^^^^^external

> +   That ID is used to construct the +BR2_EXTERNAL_$(ID)+ variable,
> +   available in +Config.in+ and +external.mk+ (see below), so that ID
> +   must be a valid make and Kconfig variable. Buildroot sets
> +   +BR2_EXTERNAL_$(ID)+ to the path of the +BR2_EXTERNAL+ tree.
                                              ^^^^^^^^^^^^^^external
> +   The format for this file is a single line with the ID.
> ++
> +Examples of an +external.mk+ file that declares the ID +FOO+:
> ++
> +----
> +$ cat external.mk
> +FOO
> +----
> ++
> +Examples of IDs and the corresponding +BR2_EXTERNAL_$(ID)+ variable:
> ++
> +  * +FOO+ -> +BR2_EXTERNAL_FOO+
> +  * +BAR_42+ -> +BR2_EXTERNAL_BAR_42+
> +  * ...
> +
> +
> ++Config.in+::
> ++external.mk+::
> +   Those files (which may each be empty) can be used to define package
> +   recipes, like for packages bundled in Buildroot itself, or other
> +   custom configuration options.
> +
> +Using +BR2_EXTERNAL+ then allows three different things:

Using an external tree allows three different things:

>  
>   * One can store all the board-specific configuration files there,
>     such as the kernel configuration, the root filesystem overlay, or
>     any other configuration file for which Buildroot allows to set its
> -   location. The +BR2_EXTERNAL+ value is available within the
> -   Buildroot configuration using +$(BR2_EXTERNAL)+. As an example, one
> -   could set the +BR2_ROOTFS_OVERLAY+ Buildroot option to
> -   +$(BR2_EXTERNAL)/board/<boardname>/overlay/+ (to specify a root
> +   location. The location of the +BR2_EXTERNAL+ tree is available in the
                                    ^^^^^^^^^^^^^^external

> +   Buildroot configuration using the +BR2_EXTERNAL_$(ID)+ variable. As
> +   an example, (considering the ID +BAR_42+), one could set the
> +   +BR2_ROOTFS_OVERLAY+ Buildroot option to
> +   +$(BR2_EXTERNAL_BAR_42)/board/<boardname>/overlay/+ (to specify a root

 Hm, with the additional _DIR bit, this is becoming quite long:
BR2_EXTERNAL_DIR_BAR_42. Perhaps it should be abbreviated, e.g. BR2_EXT_DIR_*?
But that's definitely less expressive... I'm not sure :-(

>     filesystem overlay), or the +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
>     Buildroot option to
> -   +$(BR2_EXTERNAL)/board/<boardname>/kernel.config+ (to specify the
> -   location of the kernel configuration file).
> +   +$(BR2_EXTERNAL_BAR_42)/board/<boardname>/kernel.config+ (to specify
> +   the location of the kernel configuration file).
>  
>   * One can store package recipes (i.e. +Config.in+ and
>     +<packagename>.mk+), or even custom configuration options and make
> @@ -77,27 +108,23 @@ buildroot/ $ make BR2_EXTERNAL= xconfig

 There are uses of +BR2_EXTERNAL+ hidden here :-)


>     make it appear in the top-level configuration menu, and includes
>     +$(BR2_EXTERNAL)/external.mk+ with the rest of the makefile logic.
>  +
> -.Note
> -Providing +Config.in+ and +external.mk+ is mandatory, but they can be
> -   empty.
> -+
>  The main usage of this is to store package recipes. The recommended

 This is not related to your patch of course, but I find it clearer if the
paragraph after the + is indented in the same way as above it.


 Regards,
 Arnout

>     way to do this is to write a +$(BR2_EXTERNAL)/Config.in+ file that
>     looks like:
>  +
>  ------
> -source "$BR2_EXTERNAL/package/package1/Config.in"
> -source "$BR2_EXTERNAL/package/package2/Config.in"
> +source "$BR2_EXTERNAL_BAR_42/package/package1/Config.in"
> +source "$BR2_EXTERNAL_BAR_42/package/package2/Config.in"
>  ------
>  +
>  Then, have a +$(BR2_EXTERNAL)/external.mk+ file that looks like:
>  +
>  ------
> -include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))
> +include $(sort $(wildcard $(BR2_EXTERNAL_BAR_42)/package/*/*.mk))
>  ------
>  +
> -And then in +$(BR2_EXTERNAL)/package/package1+ and
> -   +$(BR2_EXTERNAL)/package/package2+ create normal Buildroot
> +And then in +$(BR2_EXTERNAL_FOO_42)/package/package1+ and
> +   +$(BR2_EXTERNAL_FOO_42)/package/package2+ create normal Buildroot
>     package recipes, as explained in xref:adding-packages[].
>     If you prefer, you can also group the packages in subdirectories
>     called <boardname> and adapt the above paths accordingly.
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 00/16 v3] br2-external: support multiple trees at once (branch yem/multi-br2-external-ID-7)

[Arnout Vandecappelle]

On 17-07-16 12:34, Yann E. MORIN wrote:
> Therefore, br2-external trees must now use $(BR2_EXTERNAL_$(MY_ID)) to
> refer to their own files. It also prevent them from (accidentally)
> including files from other br2-external trees.
> 
> Note: IDs are optional, so that existing br2-external trees continue to
> work as before. However, if a br2-external tree does not provide an ID,
> it will not be useable in conjunction with other br2-external trees.
> Using multiple br2-external trees mandates that all of them provide an
> ID.

 A general remark on the series: I don't think it's worthwhile to keep
supporting the existing single BR2_EXTERNAL without external.name.

 First of all, an external tree is a somewhat advanced option. It exposes an
important amount of "internal API". So it is to be expected that you need to do
_something_ with it when you upgrade buildroot.

 And indeed, when you have external packages, you do have to deal with changes
in buildroot internals, e.g. the removal of host package dependency
auto-derivation. We don't have legacy handling for these internals.

 And to top it off, it's really easy to update:
- If you don't update, you get a nice error message
  '/path/to/foo' has no NAME (in file 'external.name')
- You only have to do two things:
  * Create external.name.
  * Run 'find . -type f | xargs sed -i s/BR2_EXTERNAL/BR2_EXTERNAL_FOO_PATH/g'

 Supporting the old single-external without name in parallel with the new named
approach takes a significant amount of code - I estimate about 40 lines, out of
285 lines affected by this patch set...

 So, I repeat: I think the backward compatibility should be dropped here.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 00/16 v3] br2-external: support multiple trees at once (branch yem/multi-br2-external-ID-7)

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

Hi,

 >  Supporting the old single-external without name in parallel with the new named
 > approach takes a significant amount of code - I estimate about 40 lines, out of
 > 285 lines affected by this patch set...

 >  So, I repeat: I think the backward compatibility should be dropped here.

That also sounds sensible to me.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 00/16 v3] br2-external: support multiple trees at once (branch yem/multi-br2-external-ID-7)

[Thomas Petazzoni]

Hello,

On Wed, 31 Aug 2016 08:15:57 +0200, Peter Korsgaard wrote:

>  >  Supporting the old single-external without name in parallel with the new named
>  > approach takes a significant amount of code - I estimate about 40 lines, out of
>  > 285 lines affected by this patch set...  
> 
>  >  So, I repeat: I think the backward compatibility should be dropped here.  
> 
> That also sounds sensible to me.

I originally insisted to keep the backward compatibility, but Arnout's
argumentation convinced me that it is not really necessary. And if it
can help reduce the implementation complexity, I'm all for it.

So: I'm fine with dropping backward compatibility on this.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 00/16 v3] br2-external: support multiple trees at once (branch yem/multi-br2-external-ID-7)

[Yann E. MORIN]

Thomas, Arnout,All,

On 2016-08-31 09:14 +0200, Thomas Petazzoni spake thusly:
> On Wed, 31 Aug 2016 08:15:57 +0200, Peter Korsgaard wrote:
> 
> >  >  Supporting the old single-external without name in parallel with the new named
> >  > approach takes a significant amount of code - I estimate about 40 lines, out of
> >  > 285 lines affected by this patch set...  
> > 
> >  >  So, I repeat: I think the backward compatibility should be dropped here.  
> > 
> > That also sounds sensible to me.
> 
> I originally insisted to keep the backward compatibility, but Arnout's
> argumentation convinced me that it is not really necessary. And if it
> can help reduce the implementation complexity, I'm all for it.
> 
> So: I'm fine with dropping backward compatibility on this.

OK, I'll drop.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Yann E. MORIN]

Peter, Thomas, All,

On 2016-08-28 22:50 +0200, Peter Korsgaard spake thusly:
> >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> 
>  > Hello,
>  > On Sun, 17 Jul 2016 12:34:26 +0200, Yann E. MORIN wrote:
> 
>  >> +.PHONY: $(BUILD_DIR)/.br2-external.in
> 
>  > Why is a real file that exists on the filesystem a phony target?
> 
>  >> +# Ensure the kconfig snippet is generated (for MANUAL_GEN_LISTS, below):
>  >> +MANUAL_DEPENDENCIES += prepare-kconfig
> 
>  > It's a bit crazy the complexity that we have just to allow our manual
>  > to have the list of all packages in Buildroot. I'm not even sure this
>  > section of the manual is that useful, to be honest. But oh well, that's
>  > supported today, so we have to continue support it.
> 
> Why do we _HAVE_ to continue to support it?

I was going to prepare a patchset to remove the list of packages from
the manual, when I stumbled on *one* reason parts of it is interesting
to have in the manual: the list of virtual packages, their symboles and
their providers.

I think this is worth having in the manual, if at least for packagers of
new packages that would need or provide such a virtual package.

However, I think the normal and deperacted packages should be removed,
and our script made much simpler (if at all possible).

So, what's the idea:
  - remove all packages (easy)
  - keep as-is  (even easier)
  - only keep virtual packages (not easy at all)

Insights? ;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

Hi,

 >> Why do we _HAVE_ to continue to support it?

 > I was going to prepare a patchset to remove the list of packages from
 > the manual, when I stumbled on *one* reason parts of it is interesting
 > to have in the manual: the list of virtual packages, their symboles and
 > their providers.

 > I think this is worth having in the manual, if at least for packagers of
 > new packages that would need or provide such a virtual package.

Perhaps.

 > However, I think the normal and deperacted packages should be removed,
 > and our script made much simpler (if at all possible).

 > So, what's the idea:
 >   - remove all packages (easy)
 >   - keep as-is  (even easier)
 >   - only keep virtual packages (not easy at all)

Personally I would go for 1, but if 3 is doable without too much work /
complexity (and you feel up for it), then that is also OK with me.

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Arnout Vandecappelle]

On 01-09-16 20:57, Peter Korsgaard wrote:
>>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
> 
> Hi,
> 
>  >> Why do we _HAVE_ to continue to support it?
> 
>  > I was going to prepare a patchset to remove the list of packages from
>  > the manual, when I stumbled on *one* reason parts of it is interesting
>  > to have in the manual: the list of virtual packages, their symboles and
>  > their providers.
> 
>  > I think this is worth having in the manual, if at least for packagers of
>  > new packages that would need or provide such a virtual package.
> 
> Perhaps.

 If it's just for a list of virtual packages, isn't sorting
http://autobuild.buildroot.org/stats/ by package type easier?

> 
>  > However, I think the normal and deperacted packages should be removed,
>  > and our script made much simpler (if at all possible).
> 
>  > So, what's the idea:
>  >   - remove all packages (easy)
>  >   - keep as-is  (even easier)
>  >   - only keep virtual packages (not easy at all)
> 
> Personally I would go for 1, but if 3 is doable without too much work /
> complexity (and you feel up for it), then that is also OK with me.

 Well, option 2 is not the end of the world either. The reason we consider it
now is that some complexity (1 line!) has to be added in the manual.mk to
continue supporting it. It is starting to violate the KISS principle, but not
overly so.

 It's not like the BR2_EXTERNAL backward compatibility stuff which was *really*
complicating things.


 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 00/16 v3] br2-external: support multiple trees at once (branch yem/multi-br2-external-ID-7)

[Arnout Vandecappelle]

On 31-08-16 11:26, Yann E. MORIN wrote:
> Thomas, Arnout,All,
> 
> On 2016-08-31 09:14 +0200, Thomas Petazzoni spake thusly:
>> On Wed, 31 Aug 2016 08:15:57 +0200, Peter Korsgaard wrote:
>>
>>>  >  Supporting the old single-external without name in parallel with the new named
>>>  > approach takes a significant amount of code - I estimate about 40 lines, out of
>>>  > 285 lines affected by this patch set...  
>>>
>>>  >  So, I repeat: I think the backward compatibility should be dropped here.  
>>>
>>> That also sounds sensible to me.
>>
>> I originally insisted to keep the backward compatibility, but Arnout's
>> argumentation convinced me that it is not really necessary. And if it
>> can help reduce the implementation complexity, I'm all for it.
>>
>> So: I'm fine with dropping backward compatibility on this.
> 
> OK, I'll drop.

 One remark though: we should really document such backward compatibility
issues. I don't think we're currently doing that well.

 Where should this be done?

- In CHANGES? At the moment, however, the CHANGES file is not really inviting to
look for that kind of thing. Especially because users will often upgrade in
jumps of 4 or more releases - that makes up 900 lines of CHANGES text...

- In the manual? There isn't really an appropriate place for it at the moment.
And how long do we keep it there?

- In some new location? Where?


 My personal preference would be in the manual, and keep it for a couple of years.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 00/16 v3] br2-external: support multiple trees at once (branch yem/multi-br2-external-ID-7)

[Yann E. MORIN]

Arnout, All,

On 2016-09-01 23:43 +0200, Arnout Vandecappelle spake thusly:
> On 31-08-16 11:26, Yann E. MORIN wrote:
> > Thomas, Arnout,All,
> > 
> > On 2016-08-31 09:14 +0200, Thomas Petazzoni spake thusly:
> >> On Wed, 31 Aug 2016 08:15:57 +0200, Peter Korsgaard wrote:
> >>
> >>>  >  Supporting the old single-external without name in parallel with the new named
> >>>  > approach takes a significant amount of code - I estimate about 40 lines, out of
> >>>  > 285 lines affected by this patch set...  
> >>>
> >>>  >  So, I repeat: I think the backward compatibility should be dropped here.  
> >>>
> >>> That also sounds sensible to me.
> >>
> >> I originally insisted to keep the backward compatibility, but Arnout's
> >> argumentation convinced me that it is not really necessary. And if it
> >> can help reduce the implementation complexity, I'm all for it.
> >>
> >> So: I'm fine with dropping backward compatibility on this.
> > 
> > OK, I'll drop.
> 
>  One remark though: we should really document such backward compatibility
> issues. I don't think we're currently doing that well.
> 
>  Where should this be done?
> 
> - In CHANGES? At the moment, however, the CHANGES file is not really inviting to
> look for that kind of thing. Especially because users will often upgrade in
> jumps of 4 or more releases - that makes up 900 lines of CHANGES text...
> 
> - In the manual? There isn't really an appropriate place for it at the moment.
> And how long do we keep it there?
> 
> - In some new location? Where?
> 
>  My personal preference would be in the manual, and keep it for a couple of years.

And since we can detect this situation, we can point the user to the
manual and/or emit the brief suggestion you previously posted.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 06/16 v3] core: introduce a generated kconfig snippet

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

Hi,

 >> > So, what's the idea:
 >> >   - remove all packages (easy)
 >> >   - keep as-is  (even easier)
 >> >   - only keep virtual packages (not easy at all)
 >> 
 >> Personally I would go for 1, but if 3 is doable without too much work /
 >> complexity (and you feel up for it), then that is also OK with me.

 >  Well, option 2 is not the end of the world either. The reason we consider it
 > now is that some complexity (1 line!) has to be added in the manual.mk to
 > continue supporting it. It is starting to violate the KISS principle, but not
 > overly so.

That and the fact that more than half the user manual content is this
list of packages, even though it doesn't have much value.

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] [PATCH 00/16 v3] br2-external: support multiple trees at once (branch yem/multi-br2-external-ID-7)

[Arnout Vandecappelle]

On 01-09-16 23:46, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2016-09-01 23:43 +0200, Arnout Vandecappelle spake thusly:
[snip]
>>  One remark though: we should really document such backward compatibility
>> issues. I don't think we're currently doing that well.
>>
>>  Where should this be done?
>>
>> - In CHANGES? At the moment, however, the CHANGES file is not really inviting to
>> look for that kind of thing. Especially because users will often upgrade in
>> jumps of 4 or more releases - that makes up 900 lines of CHANGES text...
>>
>> - In the manual? There isn't really an appropriate place for it at the moment.
>> And how long do we keep it there?
>>
>> - In some new location? Where?
>>
>>  My personal preference would be in the manual, and keep it for a couple of years.
> 
> And since we can detect this situation, we can point the user to the
> manual and/or emit the brief suggestion you previously posted.

 Right, that's true. In that case, having it documented in the manual or
somewhere else isn't really needed - documenting it where it hits you right in
the face is sufficient :-)

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 00/16 v3] br2-external: support multiple trees at once (branch yem/multi-br2-external-ID-7)

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

Hi,

 >  One remark though: we should really document such backward compatibility
 > issues. I don't think we're currently doing that well.

 >  Where should this be done?

 > - In CHANGES? At the moment, however, the CHANGES file is not really inviting to
 > look for that kind of thing. Especially because users will often upgrade in
 > jumps of 4 or more releases - that makes up 900 lines of CHANGES text...

Do you still think this is an issue? It is natually still big, but I do
find that readability has improved quite a bit since I removed the
Updated/fixed packages list for -rc1. I currently still add it for later
rc's, but we could drop that as well if people prefer.

 > - In the manual? There isn't really an appropriate place for it at the moment.
 > And how long do we keep it there?

 > - In some new location? Where?

 >  My personal preference would be in the manual, and keep it for a couple of years.

I think we should add it to CHANGES, and add a short notice in the
manual that earlier BR versions used a different br2-external setup.

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] [PATCH 00/16 v3] br2-external: support multiple trees at once (branch yem/multi-br2-external-ID-7)

[Yann E. MORIN]

Peter, All,

On 2016-09-04 20:42 +0200, Peter Korsgaard spake thusly:
> >>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:
>  >  One remark though: we should really document such backward compatibility
>  > issues. I don't think we're currently doing that well.
> 
>  >  Where should this be done?
> 
>  > - In CHANGES? At the moment, however, the CHANGES file is not really inviting to
>  > look for that kind of thing. Especially because users will often upgrade in
>  > jumps of 4 or more releases - that makes up 900 lines of CHANGES text...
> 
> Do you still think this is an issue? It is natually still big, but I do
> find that readability has improved quite a bit since I removed the
> Updated/fixed packages list for -rc1. I currently still add it for later
> rc's, but we could drop that as well if people prefer.
> 
>  > - In the manual? There isn't really an appropriate place for it at the moment.
>  > And how long do we keep it there?
> 
>  > - In some new location? Where?
> 
>  >  My personal preference would be in the manual, and keep it for a couple of years.
> 
> I think we should add it to CHANGES, and add a short notice in the
> manual that earlier BR versions used a different br2-external setup.

Already done, and will be part of the next iteration:
    https://git.buildroot.org/~ymorin/git/buildroot/commit/?h=yem/multi-br2-external-10&id=1eb85fca3b3363057a5e6662fd5af7c529593af4

This adds three things:
  - an appendix on how to convert, with Arnout's suggestion,
  - an xref in the br2-external section to point to that appendix,
  - a pointer to that appendix in the error message.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 00/16 v3] br2-external: support multiple trees at once (branch yem/multi-br2-external-ID-7)

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

Hi,

 >> I think we should add it to CHANGES, and add a short notice in the
 >> manual that earlier BR versions used a different br2-external setup.

 > Already done, and will be part of the next iteration:
 >     https://git.buildroot.org/~ymorin/git/buildroot/commit/?h=yem/multi-br2-external-10&id=1eb85fca3b3363057a5e6662fd5af7c529593af4

 > This adds three things:
 >   - an appendix on how to convert, with Arnout's suggestion,
 >   - an xref in the br2-external section to point to that appendix,
 >   - a pointer to that appendix in the error message.

Great, thanks!

-- 
Venlig hilsen,
Peter Korsgaard