[Buildroot] [PATCH 01/12] support/scripts: drop help for internal helper script

[Buildroot] [PATCH 01/12] support/scripts: drop help for internal helper script

[Yann E. MORIN]

We do not usually provide help or --help options for our internal
scripts. Besides, such help has a tendency to bitrot pretty quickly
anyway.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/scripts/br2-external | 33 +--------------------------------
 1 file changed, 1 insertion(+), 32 deletions(-)

diff --git a/support/scripts/br2-external b/support/scripts/br2-external
index 00cb57d1ed..dfb3cc8bcc 100755
--- a/support/scripts/br2-external
+++ b/support/scripts/br2-external
@@ -16,9 +16,8 @@ main() {
     local OPT OPTARG
     local br2_ext ofile ofmt
 
-    while getopts :hkmo: OPT; do
+    while getopts :kmo: OPT; do
         case "${OPT}" in
-        h)  help; exit 0;;
         o)  ofile="${OPTARG}";;
         k)  ofmt="kconfig";;
         m)  ofmt="mk";;
@@ -188,36 +187,6 @@ do_kconfig() {
     printf "endmenu # User-provided options\n"
 }
 
-help() {
-    cat <<-_EOF_
-	Usage:
-	    ${my_name} <-m|-k> -o FILE PATH
-
-	With -m, ${my_name} generates the makefile fragment that defines
-	variables related to the br2-external trees passed as positional
-	arguments.
-
-	With -k, ${my_name} generates the kconfig snippet to include the
-	configuration options specified in the br2-external trees passed
-	as positional arguments.
-
-	Using -k and -m together is not possible. The last one wins.
-
-	Options:
-	    -m  Generate the makefile fragment.
-
-	    -k  Generate the kconfig snippet.
-
-	    -o FILE
-	        FILE in which to generate the kconfig snippet or makefile
-	        fragment.
-
-	Returns:
-	    0   If no error
-	    !0  If any error
-	_EOF_
-}
-
 error() { local fmt="${1}"; shift; printf "BR2_EXTERNAL_ERROR = ${fmt}" "${@}"; exit 1; }
 
 my_name="${0##*/}"
-- 
2.14.1

[Buildroot] [PATCH 02/12] core: move generated .br2-external kconfig file to $(BASE_DIR)

[Yann E. MORIN]

Currently, that file is generated rather late in the configuration
process, so BUILD_DIR is known (and exists) by then.

We're soon to generate that file much earlier, at a point where
BUILD_DIR is not yet known, so we have two options:
 1- declare BUILD_DIR earlier;
 2- generate the file in an already-known location.

We go with the second solution, as we're already generating a
br2-external related file in BASE_DIR, so we can as well generate all
br2-external files in the same place.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 Config.in |  6 +++---
 Makefile  | 10 +++++-----
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/Config.in b/Config.in
index 757ad1ca40..c611bb7dec 100644
--- a/Config.in
+++ b/Config.in
@@ -14,9 +14,9 @@ config BR2_HOSTARCH
 	string
 	option env="HOSTARCH"
 
-config BR2_BUILD_DIR
+config BR2_BASE_DIR
 	string
-	option env="BUILD_DIR"
+	option env="BASE_DIR"
 
 # Hidden config symbols for packages to check system gcc version
 config BR2_HOST_GCC_VERSION
@@ -865,4 +865,4 @@ source "package/Config.in.host"
 
 source "Config.in.legacy"
 
-source "$BR2_BUILD_DIR/.br2-external.in"
+source "$BR2_BASE_DIR/.br2-external.in"
diff --git a/Makefile b/Makefile
index 86e444da98..0498b28def 100644
--- a/Makefile
+++ b/Makefile
@@ -939,7 +939,7 @@ HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
 .PHONY: prepare-kconfig
-prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in
+prepare-kconfig: outputmakefile $(BASE_DIR)/.br2-external.in
 
 $(BUILD_DIR)/buildroot-config/%onf:
 	mkdir -p $(@D)/lxdialog
@@ -957,7 +957,7 @@ COMMON_CONFIG_ENV = \
 	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
 	BR2_CONFIG=$(BR2_CONFIG) \
 	HOST_GCC_VERSION="$(HOSTCC_VERSION)" \
-	BUILD_DIR=$(BUILD_DIR) \
+	BASE_DIR=$(BASE_DIR) \
 	SKIP_LEGACY=
 
 xconfig: $(BUILD_DIR)/buildroot-config/qconf prepare-kconfig
@@ -1039,8 +1039,8 @@ endif
 # Even though the target is a real file, we mark it as PHONY as we
 # want it to be re-generated each time make is invoked, in case the
 # value of BR2_EXTERNAL is changed.
-.PHONY: $(BUILD_DIR)/.br2-external.in
-$(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
+.PHONY: $(BASE_DIR)/.br2-external.in
+$(BASE_DIR)/.br2-external.in: $(BUILD_DIR)
 	$(Q)support/scripts/br2-external -k -o "$(@)" $(BR2_EXTERNAL)
 
 # printvars prints all the variables currently defined in our
@@ -1071,7 +1071,7 @@ ifeq ($(O),$(CURDIR)/output)
 	rm -rf $(O)
 endif
 	rm -rf $(TOPDIR)/dl $(BR2_CONFIG) $(CONFIG_DIR)/.config.old $(CONFIG_DIR)/..config.tmp \
-		$(CONFIG_DIR)/.auto.deps $(BR2_EXTERNAL_FILE)
+		$(CONFIG_DIR)/.auto.deps $(BR2_EXTERNAL_FILE) $(BASE_DIR)/.br2-external.in
 
 .PHONY: help
 help:
-- 
2.14.1

[Buildroot] [PATCH 03/12] core: rename generated .br2-external.mk file

[Yann E. MORIN]

Now that the two (all of them!) br2-external related files are generated
in the same location, it makes sense they are named after the same
pattern.

When initial support for (then single) br2-external trees was added back
in a4239f7fd1 (core: introduce the BR2_EXTERNAL variable), it was not
clear-cut why that file was not named with a br2 prefix.

So rename it now.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

---
The reason for the original naming is probably because of the long
undocumented (except buried in an old dev-days report) and mostly
never obeyed rule that internal variables should be prefixed with
BR_ not BR2_, the latter being reserved for user-facing variables,
and thus the same was thought for the generated file. Or maybe not...
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 0498b28def..8b8f5b2690 100644
--- a/Makefile
+++ b/Makefile
@@ -179,7 +179,7 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 # still be overridden on the command line, therefore the file is re-created
 # every time make is run.
 
-BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external.mk
+BR2_EXTERNAL_FILE = $(BASE_DIR)/.br2-external.mk
 -include $(BR2_EXTERNAL_FILE)
 $(shell support/scripts/br2-external \
 	-m -o '$(BR2_EXTERNAL_FILE)' $(BR2_EXTERNAL))
-- 
2.14.1

[Buildroot] [PATCH 04/12] core: simplify removal of generated br2-external files

[Yann E. MORIN]

Now that all the br2-external generated files are named after the same
pattern, it gets easier to remove them all using a glob.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 8b8f5b2690..196bb3c958 100644
--- a/Makefile
+++ b/Makefile
@@ -1071,7 +1071,7 @@ ifeq ($(O),$(CURDIR)/output)
 	rm -rf $(O)
 endif
 	rm -rf $(TOPDIR)/dl $(BR2_CONFIG) $(CONFIG_DIR)/.config.old $(CONFIG_DIR)/..config.tmp \
-		$(CONFIG_DIR)/.auto.deps $(BR2_EXTERNAL_FILE) $(BASE_DIR)/.br2-external.in
+		$(CONFIG_DIR)/.auto.deps $(BASE_DIR)/.br2-external.*
 
 .PHONY: help
 help:
-- 
2.14.1

[Buildroot] [PATCH 05/12] supports/scripts: declare missing local variable

[Yann E. MORIN]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/scripts/br2-external | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/scripts/br2-external b/support/scripts/br2-external
index dfb3cc8bcc..a70e0159af 100755
--- a/support/scripts/br2-external
+++ b/support/scripts/br2-external
@@ -154,7 +154,7 @@ do_mk() {
 
 # Generate the kconfig snippet for the br2-external tree.
 do_kconfig() {
-    local br2_name br2_ext
+    local br2_name br2_desc br2_ext
 
     printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
     printf '\n'
-- 
2.14.1

[Buildroot] [PATCH 06/12] core: generate all br2-external files in one go

[Yann E. MORIN]

When we introduced support for multiple br2-external trees, we
introduced two files, one on the Makefile side, needed very early,
and one on the kconfig side, needed later in the configuration
process. We naturally introduced a two-step generation, as it looked
like the simplest and most obvious way.

But now, we are on the verge of generating more files on the kconfig
side, and it does not make sense to add even more steps to generate
them.

And even better yet, we can generate both the Makefile-side and
kconfig-side files at the same time, in fact.

Make it so.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 Makefile                     | 12 ++----------
 support/scripts/br2-external | 23 ++++++++---------------
 2 files changed, 10 insertions(+), 25 deletions(-)

diff --git a/Makefile b/Makefile
index 196bb3c958..d611eb9c02 100644
--- a/Makefile
+++ b/Makefile
@@ -181,8 +181,7 @@ $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 
 BR2_EXTERNAL_FILE = $(BASE_DIR)/.br2-external.mk
 -include $(BR2_EXTERNAL_FILE)
-$(shell support/scripts/br2-external \
-	-m -o '$(BR2_EXTERNAL_FILE)' $(BR2_EXTERNAL))
+$(shell support/scripts/br2-external -d '$(BASE_DIR)' $(BR2_EXTERNAL))
 BR2_EXTERNAL_ERROR =
 include $(BR2_EXTERNAL_FILE)
 ifneq ($(BR2_EXTERNAL_ERROR),)
@@ -939,7 +938,7 @@ HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
 .PHONY: prepare-kconfig
-prepare-kconfig: outputmakefile $(BASE_DIR)/.br2-external.in
+prepare-kconfig: outputmakefile
 
 $(BUILD_DIR)/buildroot-config/%onf:
 	mkdir -p $(@D)/lxdialog
@@ -1036,13 +1035,6 @@ ifeq ($(NEED_WRAPPER),y)
 	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
 endif
 
-# Even though the target is a real file, we mark it as PHONY as we
-# want it to be re-generated each time make is invoked, in case the
-# value of BR2_EXTERNAL is changed.
-.PHONY: $(BASE_DIR)/.br2-external.in
-$(BASE_DIR)/.br2-external.in: $(BUILD_DIR)
-	$(Q)support/scripts/br2-external -k -o "$(@)" $(BR2_EXTERNAL)
-
 # printvars prints all the variables currently defined in our
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
 # only the variables matching the make pattern passed in VARS are
diff --git a/support/scripts/br2-external b/support/scripts/br2-external
index a70e0159af..91ce1801a9 100755
--- a/support/scripts/br2-external
+++ b/support/scripts/br2-external
@@ -14,13 +14,11 @@ MANUAL_URL='https://buildroot.org/manual.html\#br2-external-converting'
 
 main() {
     local OPT OPTARG
-    local br2_ext ofile ofmt
+    local br2_ext outputdir
 
-    while getopts :kmo: OPT; do
+    while getopts :d: OPT; do
         case "${OPT}" in
-        o)  ofile="${OPTARG}";;
-        k)  ofmt="kconfig";;
-        m)  ofmt="mk";;
+        d)  outputdir="${OPTARG}";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
         esac
@@ -28,23 +26,18 @@ main() {
     # Forget options; keep only positional args
     shift $((OPTIND-1))
 
-    case "${ofmt}" in
-    mk|kconfig)
-        ;;
-    *)  error "no output format specified (-m/-k)\n";;
-    esac
-    if [ -z "${ofile}" ]; then
-        error "no output file specified (-o)\n"
+    if [ -z "${outputdir}" ]; then
+        error "no output directory specified (-d)\n"
     fi
 
-    exec >"${ofile}"
-
     # Trap any unexpected error to generate a meaningful error message
     trap "error 'unexpected error while generating ${ofile}\n'" ERR
 
     do_validate ${@//:/ }
 
-    do_${ofmt}
+    mkdir -p "${outputdir}"
+    do_mk >"${outputdir}/.br2-external.mk"
+    do_kconfig >"${outputdir}/.br2-external.in"
 }
 
 # Validates the br2-external trees passed as arguments. Makes each of
-- 
2.14.1

[Buildroot] [PATCH 07/12] core: drop now-useless prepare-kconfig rule

[Yann E. MORIN]

This rule was added back in 9429e7b698 (core: introduce an intermediate
rule before the configurators) when the kconfig-side br2-external file
was generated separately from the Makefile-side one.

Now that they are generated together very early in the Makefile, we no
longer need this intermediate rule. Drop it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 Makefile | 25 +++++++++++--------------
 1 file changed, 11 insertions(+), 14 deletions(-)

diff --git a/Makefile b/Makefile
index d611eb9c02..34c0dd75be 100644
--- a/Makefile
+++ b/Makefile
@@ -937,9 +937,6 @@ endif # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
-.PHONY: prepare-kconfig
-prepare-kconfig: outputmakefile
-
 $(BUILD_DIR)/buildroot-config/%onf:
 	mkdir -p $(@D)/lxdialog
 	PKG_CONFIG_PATH="$(HOST_PKG_CONFIG_PATH)" $(MAKE) CC="$(HOSTCC_NOCCACHE)" HOSTCC="$(HOSTCC_NOCCACHE)" \
@@ -959,19 +956,19 @@ COMMON_CONFIG_ENV = \
 	BASE_DIR=$(BASE_DIR) \
 	SKIP_LEGACY=
 
-xconfig: $(BUILD_DIR)/buildroot-config/qconf prepare-kconfig
+xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
-gconfig: $(BUILD_DIR)/buildroot-config/gconf prepare-kconfig
+gconfig: $(BUILD_DIR)/buildroot-config/gconf outputmakefile
 	@$(COMMON_CONFIG_ENV) srctree=$(TOPDIR) $< $(CONFIG_CONFIG_IN)
 
-menuconfig: $(BUILD_DIR)/buildroot-config/mconf prepare-kconfig
+menuconfig: $(BUILD_DIR)/buildroot-config/mconf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
-nconfig: $(BUILD_DIR)/buildroot-config/nconf prepare-kconfig
+nconfig: $(BUILD_DIR)/buildroot-config/nconf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
-config: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
+config: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< $(CONFIG_CONFIG_IN)
 
 # For the config targets that automatically select options, we pass
@@ -979,11 +976,11 @@ config: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
 # no values are set for the legacy options so a subsequent oldconfig
 # will query them. Therefore, run an additional olddefconfig.
 
-randconfig allyesconfig alldefconfig allnoconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
+randconfig allyesconfig alldefconfig allnoconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y $< --$@ $(CONFIG_CONFIG_IN)
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-randpackageconfig allyespackageconfig allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
+randpackageconfig allyespackageconfig allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
 		KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
@@ -991,15 +988,15 @@ randpackageconfig allyespackageconfig allnopackageconfig: $(BUILD_DIR)/buildroot
 	@rm -f $(CONFIG_DIR)/.config.nopkg
 	@$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
 
-oldconfig syncconfig olddefconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
+oldconfig syncconfig olddefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< --$@ $(CONFIG_CONFIG_IN)
 
-defconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
+defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< --defconfig$(if $(DEFCONFIG),=$(DEFCONFIG)) $(CONFIG_CONFIG_IN)
 
 define percent_defconfig
 # Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
-%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig prepare-kconfig
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile
 	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \
 		$$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN)
 endef
@@ -1007,7 +1004,7 @@ $(eval $(foreach d,$(call reverse,$(TOPDIR) $(BR2_EXTERNAL_DIRS)),$(call percent
 
 update-defconfig: savedefconfig
 
-savedefconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig
+savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< \
 		--savedefconfig=$(if $(DEFCONFIG),$(DEFCONFIG),$(CONFIG_DIR)/defconfig) \
 		$(CONFIG_CONFIG_IN)
-- 
2.14.1

[Buildroot] [PATCH 00/12] core: allow br2-external trees to provide resources usually in choices (branch yem/br2-ext-providers)

[Yann E. MORIN]

Hello All!

Some packages, like jpeg, openssl or the external toolchains, are
presented to the user in a kconfig choice, which precludes br2-external
trees to easily provide alternatives to those.

This series prepares Buildroot to accept br2-external trees to provide
such alternatives, that are seamlessly integrated in the existing
choices.

Before we can allow br2-external trees to provide them we must allow for
the kconfig variables that hold the paths to br2-external trees, to be
available through-out kconfig, not only at the end. It usually does not
matter in kconfig when a variable gets assigned a value, except when the
variable is used to source a file, at which point its value must be
known.

So, the few preparatory patches make it so that the paths are defined
early, so they are available everywhere.

Finally, the series introduces the possibility for br2-external trees to
provide alternatives to the three meaningful packages: external
toolchains, jpeg and openssl.

Eventually, this is documented in the manual.

Notes: we decided to go with a single do_kconfig() function in the helper
script, so the redirection is a bit disconcerting at first, especially in
the do_mk() function. The alternative would have been to provide one
funciton for each feature, and redirect the output of each functions,
like so:
    do_mk >$output_mk
    do_kconfig_paths >$output_paths_in
    do_kconfig_menus >$output_menus_in
    do_kconfig_jpeg >$output_jpeg_in

but that would have meant that each of those functions would have had to
repeat all the boilerplate:
    - the comment "this is a generated file; don't edit"
    - testing if there was no br2-external tree and returning if so
    - the loop over all the br2_names
    - the evaluation of br2_desc and br2_ext (the path)

In the end, at the expense of a bit more complexity in the redirections,
a single function was deemed nicer overall.

For reference, the alternative (unfinished) implementation can be seen
at:
    https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/br2-ext-toolchain


Regards,
Yann E. MORIN.


The following changes since commit f243f57abcdb415b57b8b887e9cc35bf69163224

  package/jemalloc: fix build on or1k with gcc < 6 (2019-04-21 22:32:21 +0200)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to d788f64db79f5ccefe552a7ff201005a3c045240

  docs/manual: document providers from br2-external (2019-04-22 21:18:05 +0200)


----------------------------------------------------------------
Yann E. MORIN (12):
      support/scripts: drop help for internal helper script
      core: move generated .br2-external kconfig file to $(BASE_DIR)
      core: rename generated .br2-external.mk file
      core: simplify removal of generated br2-external files
      supports/scripts: declare missing local variable
      core: generate all br2-external files in one go
      core: drop now-useless prepare-kconfig rule
      core: split generated kconfig file
      core: allow br2-external trees to provide pre-configured toolchains
      core: allow br2-external trees to provide libjpeg
      core: allow br2-external trees to provide opensl
      docs/manual: document providers from br2-external

 Config.in                              |  10 +-
 Makefile                               |  41 +++-----
 docs/manual/customize-outside-br.txt   |  81 +++++++++++++-
 package/jpeg/Config.in                 |   3 +
 package/openssl/Config.in              |   3 +
 support/scripts/br2-external           | 187 +++++++++++++++++----------------
 toolchain/toolchain-external/Config.in |   5 +-
 7 files changed, 206 insertions(+), 124 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 08/12] core: split generated kconfig file

[Yann E. MORIN]

Currently, the kconfig part contains two things: the kconfig option
with the paths to br2-external trees, and the kconfig menus for the
br2-external trees.

When we want to include more kconfig files from the br2-external tree
(e.g. to get definitions for pre-built toolchains), we will need to
have the paths defined earlier, so they can be used from the br2-external
tree to include files earlier than the existing menus.

Split the generated kconfig file in two: one to define the paths, which
gets included early in our main Config.in, and one to actually define
the existing menus, which still gets included at the same place.

For consistency, we also change the way the .mk file is generated, by
passing the output directory down to the function, rather than
redirecting the output of the macro.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 Config.in                    |   6 ++-
 support/scripts/br2-external | 113 ++++++++++++++++++++++++-------------------
 2 files changed, 69 insertions(+), 50 deletions(-)

diff --git a/Config.in b/Config.in
index c611bb7dec..e0c54a4bf1 100644
--- a/Config.in
+++ b/Config.in
@@ -18,6 +18,9 @@ config BR2_BASE_DIR
 	string
 	option env="BASE_DIR"
 
+# br2-external paths definitions
+source "$BR2_BASE_DIR/.br2-external.paths.in"
+
 # Hidden config symbols for packages to check system gcc version
 config BR2_HOST_GCC_VERSION
 	string
@@ -865,4 +868,5 @@ source "package/Config.in.host"
 
 source "Config.in.legacy"
 
-source "$BR2_BASE_DIR/.br2-external.in"
+# br2-external menus definitions
+source "$BR2_BASE_DIR/.br2-external.menus.in"
diff --git a/support/scripts/br2-external b/support/scripts/br2-external
index 91ce1801a9..60ae67c967 100755
--- a/support/scripts/br2-external
+++ b/support/scripts/br2-external
@@ -36,8 +36,8 @@ main() {
     do_validate ${@//:/ }
 
     mkdir -p "${outputdir}"
-    do_mk >"${outputdir}/.br2-external.mk"
-    do_kconfig >"${outputdir}/.br2-external.in"
+    do_mk "${outputdir}"
+    do_kconfig "${outputdir}"
 }
 
 # Validates the br2-external trees passed as arguments. Makes each of
@@ -111,73 +111,88 @@ do_validate_one() {
 # Generate the .mk snippet that defines makefile variables
 # for the br2-external tree
 do_mk() {
+    local outputdir="${1}"
     local br2_name br2_ext
 
-    printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
-    printf '\n'
+    {
+        printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
+        printf '\n'
 
-    printf 'BR2_EXTERNAL ?='
-    for br2_name in "${BR2_EXT_NAMES[@]}"; do
-        eval br2_ext="\"\${BR2_EXT_PATHS_${br2_name}}\""
-        printf ' %s' "${br2_ext}"
-    done
-    printf '\n'
+        printf 'BR2_EXTERNAL ?='
+        for br2_name in "${BR2_EXT_NAMES[@]}"; do
+            eval br2_ext="\"\${BR2_EXT_PATHS_${br2_name}}\""
+            printf ' %s' "${br2_ext}"
+        done
+        printf '\n'
 
-    printf 'BR2_EXTERNAL_NAMES = \n'
-    printf 'BR2_EXTERNAL_DIRS = \n'
-    printf 'BR2_EXTERNAL_MKS = \n'
+        printf 'BR2_EXTERNAL_NAMES = \n'
+        printf 'BR2_EXTERNAL_DIRS = \n'
+        printf 'BR2_EXTERNAL_MKS = \n'
 
-    if [ ${#BR2_EXT_NAMES[@]} -eq 0 ]; then
-        printf '\n'
-        printf '# No br2-external tree defined.\n'
-        return
-    fi
+        if [ ${#BR2_EXT_NAMES[@]} -eq 0 ]; then
+            printf '\n'
+            printf '# No br2-external tree defined.\n'
+            return
+        fi
 
-    for br2_name in "${BR2_EXT_NAMES[@]}"; do
-        eval br2_desc="\"\${BR2_EXT_DESCS_${br2_name}}\""
-        eval br2_ext="\"\${BR2_EXT_PATHS_${br2_name}}\""
-        printf '\n'
-        printf 'BR2_EXTERNAL_NAMES += %s\n' "${br2_name}"
-        printf 'BR2_EXTERNAL_DIRS += %s\n' "${br2_ext}"
-        printf 'BR2_EXTERNAL_MKS += %s/external.mk\n' "${br2_ext}"
-        printf 'export BR2_EXTERNAL_%s_PATH = %s\n' "${br2_name}" "${br2_ext}"
-        printf 'export BR2_EXTERNAL_%s_DESC = %s\n' "${br2_name}" "${br2_desc}"
-    done
+        for br2_name in "${BR2_EXT_NAMES[@]}"; do
+            eval br2_desc="\"\${BR2_EXT_DESCS_${br2_name}}\""
+            eval br2_ext="\"\${BR2_EXT_PATHS_${br2_name}}\""
+            printf '\n'
+            printf 'BR2_EXTERNAL_NAMES += %s\n' "${br2_name}"
+            printf 'BR2_EXTERNAL_DIRS += %s\n' "${br2_ext}"
+            printf 'BR2_EXTERNAL_MKS += %s/external.mk\n' "${br2_ext}"
+            printf 'export BR2_EXTERNAL_%s_PATH = %s\n' "${br2_name}" "${br2_ext}"
+            printf 'export BR2_EXTERNAL_%s_DESC = %s\n' "${br2_name}" "${br2_desc}"
+        done
+    } >"${outputdir}/.br2-external.mk"
 }
 
-# Generate the kconfig snippet for the br2-external tree.
+# Generate the kconfig snippets for the br2-external tree.
 do_kconfig() {
-    local br2_name br2_desc br2_ext
-
-    printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
-    printf '\n'
+    local outputdir="${1}"
+    local br2_name br2_ext br2
 
+    for br2 in paths menus; do
+        {
+            printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
+            printf '\n'
+            if [ ${#BR2_EXT_NAMES[@]} -eq 0 ]; then
+                printf '# No br2-external tree defined.\n'
+            fi
+        } >"${outputdir}/.br2-external.${br2}.in"
+    done
     if [ ${#BR2_EXT_NAMES[@]} -eq 0 ]; then
-        printf '# No br2-external tree defined.\n'
         return
     fi
 
-    printf 'menu "External options"\n'
-    printf '\n'
+    printf 'menu "External options"\n\n' >>"${outputdir}/.br2-external.menus.in"
 
     for br2_name in "${BR2_EXT_NAMES[@]}"; do
         eval br2_desc="\"\${BR2_EXT_DESCS_${br2_name}}\""
         eval br2_ext="\"\${BR2_EXT_PATHS_${br2_name}}\""
-        if [ ${#BR2_EXT_NAMES[@]} -gt 1 ]; then
-            printf 'menu "%s"\n' "${br2_desc}"
-        fi
-        printf 'comment "%s (in %s)"\n' "${br2_desc}" "${br2_ext}"
-        printf 'config BR2_EXTERNAL_%s_PATH\n' "${br2_name}"
-        printf '\tstring\n'
-        printf '\tdefault "%s"\n' "${br2_ext}"
-        printf 'source "%s/Config.in"\n' "${br2_ext}"
-        if [ ${#BR2_EXT_NAMES[@]} -gt 1 ]; then
-            printf 'endmenu # %s\n' "${br2_name}"
-        fi
-        printf '\n'
+
+        {
+            printf 'config BR2_EXTERNAL_%s_PATH\n' "${br2_name}"
+            printf '\tstring\n'
+            printf '\tdefault "%s"\n' "${br2_ext}"
+            printf '\n'
+        } >>"${outputdir}/.br2-external.paths.in"
+
+        {
+            if [ ${#BR2_EXT_NAMES[@]} -gt 1 ]; then
+                printf 'menu "%s"\n' "${br2_desc}"
+            fi
+            printf 'comment "%s (in %s)"\n' "${br2_desc}" "${br2_ext}"
+            printf 'source "%s/Config.in"\n' "${br2_ext}"
+            if [ ${#BR2_EXT_NAMES[@]} -gt 1 ]; then
+                printf 'endmenu # %s\n' "${br2_name}"
+            fi
+            printf '\n'
+        } >>"${outputdir}/.br2-external.menus.in"
     done
 
-    printf "endmenu # User-provided options\n"
+    printf 'endmenu\n' >>"${outputdir}/.br2-external.menus.in"
 }
 
 error() { local fmt="${1}"; shift; printf "BR2_EXTERNAL_ERROR = ${fmt}" "${@}"; exit 1; }
-- 
2.14.1

[Buildroot] [PATCH 09/12] core: allow br2-external trees to provide pre-configured toolchains

[Yann E. MORIN]

Since we have a choice for the pre-configured pre-built toolchains,
there is no p[ossbility for a br2-external to provide its own. The
only solution so far for defconfigs in br2-external trees is to use
BR2_TOOLCHAIN_EXTERNAL_CUSTOM and define all the bits by itself...

This is not so convemient, so offer a way for br2-external trees to
provide such pre-configure toolchains.

To allow for this, we now scan each br2-external tree and look for a
specific file, provides.toiolchains.in. We generate a kconfig file that
sources each such file, and that generated file is sourced from within
the toolchain choice, thus making the toolchains from a br2-external
tree possible and available in the same location as the ones known to
Buildroot.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/scripts/br2-external           | 10 +++++++++-
 toolchain/toolchain-external/Config.in |  5 ++++-
 2 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/support/scripts/br2-external b/support/scripts/br2-external
index 60ae67c967..9f4f261c3d 100755
--- a/support/scripts/br2-external
+++ b/support/scripts/br2-external
@@ -153,7 +153,7 @@ do_kconfig() {
     local outputdir="${1}"
     local br2_name br2_ext br2
 
-    for br2 in paths menus; do
+    for br2 in paths menus toolchains; do
         {
             printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
             printf '\n'
@@ -190,6 +190,14 @@ do_kconfig() {
             fi
             printf '\n'
         } >>"${outputdir}/.br2-external.menus.in"
+
+        if [ -f "${br2_ext}/provides/toolchains.in" ]; then
+            printf 'comment "Toolchains from: %s"\n' "${br2_desc}"
+            printf 'source "%s/provides/toolchains.in"\n' "${br2_ext}"
+            printf '\n'
+        else
+            printf '# No toolchain from %s\n\n' "${br2_desc}"
+        fi >>"${outputdir}/.br2-external.toolchains.in"
     done
 
     printf 'endmenu\n' >>"${outputdir}/.br2-external.menus.in"
diff --git a/toolchain/toolchain-external/Config.in b/toolchain/toolchain-external/Config.in
index d234c1c552..1cf4fc850d 100644
--- a/toolchain/toolchain-external/Config.in
+++ b/toolchain/toolchain-external/Config.in
@@ -47,9 +47,12 @@ source "toolchain/toolchain-external/toolchain-external-codesourcery-amd64/Confi
 
 # Kept last, so it remains the non-default choice, unless there isn't
 # any available toolchain profile for the currently selected
-# architecture.
+# architecture, but before toolchains from br2-external trees.
 source "toolchain/toolchain-external/toolchain-external-custom/Config.in"
 
+# Toolchains from br2-external trees, if any
+source "$BR2_BASE_DIR/.br2-external.toolchains.in"
+
 endchoice
 
 choice
-- 
2.14.1

[Buildroot] [PATCH 10/12] core: allow br2-external trees to provide libjpeg

[Yann E. MORIN]

Similarly to toolchains, we now offer a way for br2-external trees to
provide their libjpeg implementation, which gets included in the jpeg
choice.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/jpeg/Config.in       |  3 +++
 support/scripts/br2-external | 10 +++++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/package/jpeg/Config.in b/package/jpeg/Config.in
index a4a63cc7dc..6940d4dfe1 100644
--- a/package/jpeg/Config.in
+++ b/package/jpeg/Config.in
@@ -38,6 +38,9 @@ config BR2_PACKAGE_JPEG_TURBO
 
 	  http://www.libjpeg-turbo.org
 
+# libjpeg from br2-external trees, if any
+source "$BR2_BASE_DIR/.br2-external.jpeg.in"
+
 endchoice
 
 config BR2_PACKAGE_HAS_JPEG
diff --git a/support/scripts/br2-external b/support/scripts/br2-external
index 9f4f261c3d..573e706502 100755
--- a/support/scripts/br2-external
+++ b/support/scripts/br2-external
@@ -153,7 +153,7 @@ do_kconfig() {
     local outputdir="${1}"
     local br2_name br2_ext br2
 
-    for br2 in paths menus toolchains; do
+    for br2 in paths menus toolchains jpeg; do
         {
             printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
             printf '\n'
@@ -198,6 +198,14 @@ do_kconfig() {
         else
             printf '# No toolchain from %s\n\n' "${br2_desc}"
         fi >>"${outputdir}/.br2-external.toolchains.in"
+
+        if [ -f "${br2_ext}/provides/jpeg.in" ]; then
+            printf 'comment "jpeg from: %s"\n' "${br2_desc}"
+            printf 'source "%s/provides/jpeg.in"\n' "${br2_ext}"
+            printf '\n'
+        else
+            printf '# No jpeg from %s\n\n' "${br2_desc}"
+        fi >>"${outputdir}/.br2-external.jpeg.in"
     done
 
     printf 'endmenu\n' >>"${outputdir}/.br2-external.menus.in"
-- 
2.14.1

[Buildroot] [PATCH 11/12] core: allow br2-external trees to provide opensl

[Yann E. MORIN]

Similarly to toolchains and jpeg, we now offer a way for br2-external
trees to provide their openssl implementation, which gets included in
the openssl choice.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/openssl/Config.in    |  3 +++
 support/scripts/br2-external | 10 +++++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/package/openssl/Config.in b/package/openssl/Config.in
index d84033141f..43468be43c 100644
--- a/package/openssl/Config.in
+++ b/package/openssl/Config.in
@@ -68,6 +68,9 @@ config BR2_PACKAGE_LIBRESSL_BIN
 
 endif
 
+# openssl from br2-external trees, if any
+source "$BR2_BASE_DIR/.br2-external.openssl.in"
+
 endchoice
 
 config BR2_PACKAGE_HAS_OPENSSL
diff --git a/support/scripts/br2-external b/support/scripts/br2-external
index 573e706502..a9ee602da5 100755
--- a/support/scripts/br2-external
+++ b/support/scripts/br2-external
@@ -153,7 +153,7 @@ do_kconfig() {
     local outputdir="${1}"
     local br2_name br2_ext br2
 
-    for br2 in paths menus toolchains jpeg; do
+    for br2 in paths menus toolchains jpeg openssl; do
         {
             printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
             printf '\n'
@@ -206,6 +206,14 @@ do_kconfig() {
         else
             printf '# No jpeg from %s\n\n' "${br2_desc}"
         fi >>"${outputdir}/.br2-external.jpeg.in"
+
+        if [ -f "${br2_ext}/provides/openssl.in" ]; then
+            printf 'comment "openssl from: %s"\n' "${br2_desc}"
+            printf 'source "%s/provides/openssl.in"\n' "${br2_ext}"
+            printf '\n'
+        else
+            printf '# No openssl from %s\n\n' "${br2_desc}"
+        fi >>"${outputdir}/.br2-external.openssl.in"
     done
 
     printf 'endmenu\n' >>"${outputdir}/.br2-external.menus.in"
-- 
2.14.1

[Buildroot] [PATCH 12/12] docs/manual: document providers from br2-external

[Yann E. MORIN]

Add documentation about how a br2-external tree can provide an external
toolchain or a libjpeg alternative implementation.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

---
Note: the documentation about external toolchains is inexistent so far,
but this is unrelated to this change, which is only about documenting
how to provide one from a br2-external.
---
 docs/manual/customize-outside-br.txt | 81 +++++++++++++++++++++++++++++++++++-
 1 file changed, 80 insertions(+), 1 deletion(-)

diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
index b1eed326aa..ceee1c33f3 100644
--- a/docs/manual/customize-outside-br.txt
+++ b/docs/manual/customize-outside-br.txt
@@ -81,7 +81,8 @@ in the following chapters:
 
 Apart from those mandatory files, there may be additional and optional
 content that may be present in a br2-external tree, like the +configs/+
-directory. They are described in the following chapters as well.
+or +provides/+ directories. They are described in the following chapters
+as well.
 
 A complete example br2-external tree layout is also described later.
 
@@ -186,6 +187,22 @@ the one from the last br2-external tree is used. It is thus possible
 to override a defconfig bundled in Buildroot or another br2-external
 tree.
 
+===== The +provides/+ directory
+
+For some packages, Buildroot provides a choice between two (or more)
+implementations of API-compatible such packages. For example, there is
+a choice to choose either libjpeg ot jpeg-turbo, and another choice
+between openssl or libressl. Finally, there is a choice to select one
+of the known, pre-configured toolchains.
+
+It is possible for a br2-external to extend those choices, by providing
+a set of files that define those alternatives:
+
+* +provides/toolchains.in+ defines the pre-configured toolchains, which
+  will then be listed in the toolchain selection;
+* +provides/jpeg.in+ defines the alternative libjpeg implementations;
+* +provides/openssl.in+ defines the alternative openssl implementations.
+
 ===== Free-form content
 
 One can store all the board-specific configuration files there, such
@@ -217,8 +234,10 @@ illustration, of course):
   |     `----
   |
   |- Config.in
+  |     |source "$BR2_EXTERNAL_BAR_42_PATH/toolchain/toolchain-external-mine/Config.in.options"
   |     |source "$BR2_EXTERNAL_BAR_42_PATH/package/pkg-1/Config.in"
   |     |source "$BR2_EXTERNAL_BAR_42_PATH/package/pkg-2/Config.in"
+  |     |source "$BR2_EXTERNAL_BAR_42_PATH/package/my-jpeg/Config.in"
   |     |
   |     |config BAR_42_FLASH_ADDR
   |     |    hex "my-board flash address"
@@ -259,6 +278,40 @@ illustration, of course):
   |- package/pkg-2/pkg-2.hash
   |- package/pkg-2/pkg-2.mk
   |
+  |- provides/jpeg.in
+  |     |config BR2_PACKAGE_MY_JPEG
+  |     |    bool "my-jpeg"
+  |     `----
+  |- package/my-jpeg/Config.in
+  |     |config BR2_PACKAGE_PROVIDES_JPEG
+  |     |    default "my-jpeg" if BR2_PACKAGE_MY_JPEG
+  |     `----
+  |- package/my-jpeg/my-jpeg.mk
+  |     |# This is a normal package .mk file
+  |     |MY_JPEG_VERSION = 1.2.3
+  |     |MY_JPEG_SITE = https://example.net/some/place
+  |     |$(eval $(autotools-package))
+  |     `----
+  |
+  |- provides/toolchains.in
+  |     |config BR2_TOOLCHAIN_EXTERNAL_MINE
+  |     |    bool "my custom toolchain"
+  |     |    depends on BR2_some_arch
+  |     |    select BR2_INSTALL_LIBSTDCPP
+  |     `----
+  |- toolchain/toolchain-external-mine/Config.in.options
+  |     |if BR2_TOOLCHAIN_EXTERNAL_MINE
+  |     |config BR2_TOOLCHAIN_EXTERNAL_PREFIX
+  |     |    default "arch-mine-linux-gnu"
+  |     |config BR2_PACKAGE_PROVIDES_TOOLCHAIN_EXTERNAL
+  |     |    default "toolchain-external-mine"
+  |     |endif
+  |     `----
+  |- toolchain/toolchain-external-mine/toolchain-external-mine,mk
+  |     |TOOLCHAIN_EXTERNAL_MINE_SITE = https://example.net/some/place
+  |     |$(eval $(toolchain-external-package))
+  |     `----
+  |
   |- configs/my-board_defconfig
   |     |BR2_GLOBAL_PATCH_DIR="$(BR2_EXTERNAL_BAR_42_PATH)/patches/"
   |     |BR2_ROOTFS_OVERLAY="$(BR2_EXTERNAL_BAR_42_PATH)/board/my-board/overlay/"
@@ -311,3 +364,29 @@ External options  --->
         [ ] foo
         [ ] bar
 ----
+
+Additionally, the toolchains will be visible in the toolchain choice:
+
+----
+Toolchain  --->
+    Toolchain ()  --->
+        ( ) Custom toolchain
+            *** Toolchains from: Example br2-external tree ***
+        (X) my custom toolchain
+----
+
+And similarly for jpeg:
+
+----
+Target packages  --->
+    Libraries  --->
+        Graphics  --->
+            [*] jpeg support
+                jpeg variant ()  --->
+                    ( ) jpeg
+                    ( ) jpeg-turbo
+                        *** jpeg from: Example br2-external tree ***
+                    (X) my-jpeg
+                        *** jpeg from: FOO_27 ***
+                    ( ) another-jpeg
+----
-- 
2.14.1