[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Thomas Petazzoni]

Hello,

Here is the second version of the BR2_EXTERNAL patch series.

Changes since v1
================

 * Store BR2_EXTERNAL in the Makefile generated in the output
   directory, so that BR2_EXTERNAL doesn't have to be passed over and
   over again. Suggested by Ryan Barnett.

 * Patch set splitted in more fine-grained changes, as suggested by
   Arnout.

 * Instead of having the top-level Config.in file generated at
   run-time, use instead a trick proposed by Arnout: make BR2_EXTERNAL
   point to support/dummy-external/, which contains an empty
   Config.in. This way, regardless of whether BR2_EXTERNAL is
   specified by the user or not, kconfig is happy as 'source
   "$BR2_EXTERNAL/Config.in"' will always point to an existing
   file. If BR2_EXTERNAL is not used, it will point to the dummy file,
   if BR2_EXTERNAL is used, it will point to Config.in file in the
   provided BR2_EXTERNAL directory.

   With this, BR2_EXTERNAL always has a value, so testing whether
   BR2_EXTERNAL is empty no longer tells us if the user has provided a
   value or not. Therefore, we have another variable,
   BR2_EXTERNAL_USED, which tells if the mechanism is used by the user
   or not. This variable avoids trying to use defconfigs from the
   dummy directory, or to encode the BR2_EXTERNAL value in the
   Makefile wrapper.

 * As a consequence of the previous change, it is no longer needed to
   refactor the *config dependencies using a variable, so I've dropped
   what was previously the first patch of the series.

 * BR2_EXTERNAL is turned into an absolute path, in order to avoid any
   problem. Suggested by Arnout.

 * Fixed the usage of <some board> vs. <boardname> in the
   documentation, as noted by Arnout.

Remaning questions
==================

 * Interaction with the 'local.mk' mechanism. Should 'external.mk'
   replace the 'local.mk' mechanism? Should they co-exist?

Introduction to the patch series
================================

Following the e-mail from Ryan Barnett entitled "[RFC] New feature to
handle a custom board folder", I finally took a bit of time to
implement how I think the support for externally stored Buildroot
customization should be done.

I really think the approach of having board-related configuration
options within Buildroot is wrong: it is the Buildroot configuration
as a whole that defines what the configuration of the user system is:
a combination of a kernel, a root filesystem with various packages, a
selection of a root filesystem image format, some system configuration
tuning, a bootloader, etc.

In Ryan e-mail, I believe the most important thing is the wish to
store the board-specific details outside of the Buildroot tree. In
fact, this is already possible: nothing prevents anyone from using
$(TOPDIR)/../company/kernel.config as the kernel configuration file
for example, where $(TOPDIR)/../company/ is outside of the Buildroot
tree, and versioned in a completely different way.

However, it seems like this possibility is not 'visible' enough, and
admittedly only usable for some Buildroot configuration elements,
while an user may also be interested in storing Buildroot package
recipes and Buildroot deconfig files outside of the main Buildroot
tree.

So, this patch set adds the BR2_EXTERNAL mechanism. It is very
lightweight, and doesn't change anything to the Buildroot overall
logic. Basically, BR2_EXTERNAL is an environment variable that one can
point to a directory that will contain Buildroot customizations. It is
then usable for three things:

 * It is available as $(BR2_EXTERNAL) within all of the Buildroot
   configuration options, to define the location of the kernel
   configuration file, the root filesystem overlay, etc.

 * It allows to store additional Config.in options and makefile logic
   in $(BR2_EXTERNAL), since Buildroot automatically includes
   $(BR2_EXTERNAL)/Config.in into the top-level menu, and
   $(BR2_EXTERNAL)/external.mk in the makefile logic. The main purpose
   of this is to be able to store custom packages outside of the
   Buildroot tree.

 * It allows to store defconfigs in $(BR2_EXTERNAL)/configs/. They
   appear automatically in 'make help' and can be loaded like any
   other defconfig using 'make <something>_defconfig'.

Best regards,

Thomas

Thomas Petazzoni (4):
  core: introduce the BR2_EXTERNAL variable
  core: allow external Config.in/makefile code to be integrated
  core: allow external defconfigs to be used
  docs/manual: add explanations about BR2_EXTERNAL

 Config.in                            |   6 ++
 Makefile                             |  25 +++++++-
 docs/manual/customize-outside-br.txt | 112 +++++++++++++++++++++++++++++++++++
 docs/manual/customize.txt            |   2 +
 support/dummy-external/Config.in     |   0
 support/scripts/mkmakefile           |   3 +-
 6 files changed, 144 insertions(+), 4 deletions(-)
 create mode 100644 docs/manual/customize-outside-br.txt
 create mode 100644 support/dummy-external/Config.in

-- 
1.8.1.2

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Thomas Petazzoni]

This commit introduces the BR2_EXTERNAL environment variable, which
will allow to keep Buildroot customization (board-specific
configuration files or root filesystem overlays, package Config.in and
makefiles, as well as defconfigs) outside of the Buildroot tree.

This commit only introduces the variable itself, and ensures that it
is available within Config.in options, so that string options used to
specify paths to directories or files can use $BR2_EXTERNAL as a
reference. For example, one can use
$BR2_EXTERNAL/board/<someboard>/kernel.config as the
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE value.

Following patches extend the usage of BR2_EXTERNAL to other areas
(packages and defconfigs).

In details, this commit:

 * Introduces the BR2_EXTERNAL Kconfig option. This option has no
   prompt, and is therefore not visible to the user and also not
   stored in the .config file. It is automatically set to the value of
   the BR2_EXTERNAL environment variable. The only purpose of this
   BR2_EXTERNAL Kconfig option is to allow $BR2_EXTERNAL to be
   properly expanded when used inside Kconfig option values.

 * Ensures that the path given in BR2_EXTERNAL is an absolute path, or
   if not path is given, ensures that BR2_EXTERNAL points to a dummy
   implementation in $(TOPDIR)/support/dummy-external/. This last part
   is not directly useful in this commit, but is needed to properly
   support Config.in options declared in BR2_EXTERNAL (next commit).

 * Passes the BR2_EXTERNAL into the *config environment, so that its
   value is found when parsing/evaluating Config.in files and .config
   values.

 * Encodes the BR2_EXTERNAL value into the Makefile wrapper (only if
   BR2_EXTERNAL is specified by the user, not if the dummy internal
   value is used).

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Config.in                  |  4 ++++
 Makefile                   | 13 +++++++++++--
 support/scripts/mkmakefile |  3 ++-
 3 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/Config.in b/Config.in
index 65c29b4..522eaf8 100644
--- a/Config.in
+++ b/Config.in
@@ -14,6 +14,10 @@ config BR2_HOSTARCH
 	string
 	option env="HOSTARCH"
 
+config BR2_EXTERNAL
+	string
+	option env="BR2_EXTERNAL"
+
 # Hidden boolean selected by pre-built packages for x86, when they
 # need to run on x86-64 machines (example: pre-built external
 # toolchains, binary tools like SAM-BA, etc.).
diff --git a/Makefile b/Makefile
index 8610592..13ad342 100644
--- a/Makefile
+++ b/Makefile
@@ -51,6 +51,14 @@ CONFIG_CONFIG_IN=Config.in
 CONFIG=support/kconfig
 DATE:=$(shell date +%Y%m%d)
 
+# Turn BR2_EXTERNAL into an absolute path
+ifneq ($(BR2_EXTERNAL),)
+override BR2_EXTERNAL := $(realpath $(BR2_EXTERNAL))
+BR2_EXTERNAL_USED = y
+else
+BR2_EXTERNAL = $(TOPDIR)/support/dummy-external/
+endif
+
 # Compute the full local version string so packages can use it as-is
 # Need to export it, so it can be got from environment in children (eg. mconf)
 export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)
@@ -632,7 +640,8 @@ COMMON_CONFIG_ENV = \
 	KCONFIG_AUTOCONFIG=$(BUILD_DIR)/buildroot-config/auto.conf \
 	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
 	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
-	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG)
+	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG) \
+	BR2_EXTERNAL=$(BR2_EXTERNAL)
 
 xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
 	@mkdir -p $(BUILD_DIR)/buildroot-config
@@ -739,7 +748,7 @@ source-check:
 # output directory.
 outputmakefile:
 ifeq ($(NEED_WRAPPER),y)
-	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
+	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O) $(if $(BR2_EXTERNAL_USED),$(BR2_EXTERNAL))
 endif
 
 # printvars prints all the variables currently defined in our Makefiles
diff --git a/support/scripts/mkmakefile b/support/scripts/mkmakefile
index cef2ec7..e7c1bd0 100755
--- a/support/scripts/mkmakefile
+++ b/support/scripts/mkmakefile
@@ -6,7 +6,7 @@
 # Usage
 # $1 - Kernel src directory
 # $2 - Output directory
-
+# $3 - BR2_EXTERNAL directory
 
 test ! -r $2/Makefile -o -O $2/Makefile || exit 0
 # Only overwrite automatically generated Makefiles
@@ -27,6 +27,7 @@ makedir := \$(dir \$(call lastword,\$(MAKEFILE_LIST)))
 
 MAKEARGS := -C $1
 MAKEARGS += O=\$(if \$(patsubst /%,,\$(makedir)),\$(CURDIR)/)\$(patsubst %/,%,\$(makedir))
+MAKEARGS += BR2_EXTERNAL=$3
 
 MAKEFLAGS += --no-print-directory
 
-- 
1.8.1.2

[Buildroot] [PATCHv2 2/4] core: allow external Config.in/makefile code to be integrated

[Thomas Petazzoni]

This commit allows the BR2_EXTERNAL directory to contain Config.in and
Makefile code, which gets integrated into the Buildroot build logic:

 - Buildroot automatically includes the $BR2_EXTERNAL/Config.in in the
   top-level configuration menu.

 - Buildroot automatically includes the BR2_EXTERNAL/external.mk in
   the build logic, so it can for example be used to include other .mk
   files that define package recipes.

We also add a dummy Config.in file in support/dummy-external/ to
ensure that the source "$BR2_EXTERNAL/Config.in" line will point to an
existing file even when BR2_EXTERNAL is not used by the user.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Config.in                        | 2 ++
 Makefile                         | 4 ++++
 support/dummy-external/Config.in | 0
 3 files changed, 6 insertions(+)
 create mode 100644 support/dummy-external/Config.in

diff --git a/Config.in b/Config.in
index 522eaf8..9ef5466 100644
--- a/Config.in
+++ b/Config.in
@@ -465,3 +465,5 @@ source "boot/Config.in"
 source "package/Config.in.host"
 
 source "Config.in.legacy"
+
+source "$BR2_EXTERNAL/Config.in"
diff --git a/Makefile b/Makefile
index 13ad342..bea5ad5 100644
--- a/Makefile
+++ b/Makefile
@@ -348,6 +348,10 @@ include boot/common.mk
 include linux/linux.mk
 include system/system.mk
 
+ifeq ($(BR2_EXTERNAL_USED),y)
+include $(BR2_EXTERNAL)/external.mk
+endif
+
 TARGETS+=target-finalize
 
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
diff --git a/support/dummy-external/Config.in b/support/dummy-external/Config.in
new file mode 100644
index 0000000..e69de29
-- 
1.8.1.2

[Buildroot] [PATCHv2 3/4] core: allow external defconfigs to be used

[Thomas Petazzoni]

This commit allows the user to store defconfigs in
$BR2_EXTERNAL/configs/. To achieve this:

 * It adds a new %_defconfig that looks in $BR2_EXTERNAL/configs/ for
   the corresponding defconfig file.

 * Updates the help target to also list external defconfigs.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index bea5ad5..799c529 100644
--- a/Makefile
+++ b/Makefile
@@ -729,6 +729,12 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@mkdir -p $(BUILD_DIR)/buildroot-config
 	@$(COMMON_CONFIG_ENV) $< --defconfig=$(TOPDIR)/configs/$@ $(CONFIG_CONFIG_IN)
 
+ifeq ($(BR2_EXTERNAL_USED),y)
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig outputmakefile
+	@mkdir -p $(BUILD_DIR)/buildroot-config
+	@$(COMMON_CONFIG_ENV) $< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
+endif
+
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@mkdir -p $(BUILD_DIR)/buildroot-config
 	@$(COMMON_CONFIG_ENV) $< \
@@ -840,7 +846,7 @@ endif
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
 	@echo '  make O=dir             - Locate all output files in "dir", including .config'
 	@echo
-	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
+	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig $(if $(BR2_EXTERNAL_USED),$(BR2_EXTERNAL)/configs/*_defconfig)))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 	@echo
 	@echo 'See docs/README, or generate the Buildroot manual for further details'
-- 
1.8.1.2

[Buildroot] [PATCHv2 4/4] docs/manual: add explanations about BR2_EXTERNAL

[Thomas Petazzoni]

This commit updates the manual to add details on how to use the
BR2_EXTERNAL feature.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 docs/manual/customize-outside-br.txt | 112 +++++++++++++++++++++++++++++++++++
 docs/manual/customize.txt            |   2 +
 2 files changed, 114 insertions(+)
 create mode 100644 docs/manual/customize-outside-br.txt

diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
new file mode 100644
index 0000000..74899af
--- /dev/null
+++ b/docs/manual/customize-outside-br.txt
@@ -0,0 +1,112 @@
+// -*- mode:doc -*- ;
+
+[[outside-br-custom]]
+Keeping customization outside Buildroot
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+While the Buildroot community recommends and encourages upstreaming to
+the official Buildroot version the packages and boards support that
+are written by developers, it is sometimes not possible or desirable,
+due to these packages or boards being highly specific or proprietary.
+
+In this case, Buildroot users are offered two choices:
+
+ * They can add their packages, board support and configuration files
+   directly within the Buildroot tree, and maintain them by using
+   branches in a version control system.
+
+ * They can use the +BR2_EXTERNAL+ mechanism, which allows to keep
+   package recipes, board support and configuration files outside of
+   the Buildroot tree, while still having them nicely integrated in
+   the build logic. The following paragraphs give details on how to
+   use +BR2_EXTERNAL+.
+
++BR2_EXTERNAL+ is an environment variable that one can use to point to
+a directory that contains Buildroot customizations. This environment
+variable must be passed everytime a Buildroot +make+ command is
+invoked, be it at configuration time (+make menuconfig+, +make
+xconfig+, etc.), at build time (+make+) or any other Buildroot command
+(+make source+, +make legal-info+, etc.). As an exception, when
+building out-of-tree with +O=+, the +BR2_EXTERNAL+ value is encoded
+into the Makefile wrapper generated by Buildroot, so that passing
++BR2_EXTERNAL+ is no longer needed.
+
+Examples:
+
+-----
+ buildroot/ $ make BR2_EXTERNAL=../company menuconfig
+ buildroot/ $ make BR2_EXTERNAL=../company
+-----
+
+This +BR2_EXTERNAL+ then allows three different things:
+
+ * One can store all the board-specific configuration files here, such
+   as the kernel configuration, the root filesystem overlay, or any
+   other configuration file for which Buildroot allows to set its
+   location. The +BR2_EXTERNAL+ value is available within the
+   Buildroot configuration using +$(BR2_EXTERNAL)+. As an example, one
+   could set the +BR2_ROOTFS_OVERLAY+ Buildroot option to
+   +$(BR2_EXTERNAL)/board/<boardname>/overlay/+ (to specify a root
+   filesystem overlay), or the +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
+   Buildroot option to +$(BR2_EXTERNAL)/board/<boardname>/kernel.config+
+   (to specify the location of the kernel configuration file).  + To
+   achieve this, it is recommended but not mandatory, to store those
+   details in directories called +board/<boardname>/+ under
+   +BR2_EXTERNAL+.
+
+ * One can store package recipes (i.e +Config.in+ and
+   +<packagename>.mk+), or even custom configuration options and make
+   logic. Buildroot automatically includes +BR2_EXTERNAL/Config.in+ to
+   make it appear in the top-level configuration menu, and includes
+   +BR2_EXTERNAL/external.mk+ with the rest of the makefile logic.
++
+The main usage of this is to store package recipes. The recommended
+   way to do this is to write a +BR2_EXTERNAL/Config.in+ that looks
+   like:
++
+------
+menu "<somecompany> packages"
+
+source "$BR2_EXTERNAL/package/package1/Config.in"
+source "$BR2_EXTERNAL/package/package2/Config.in"
+
+endmenu
+------
++
+Then, have a +BR2_EXTERNAL/external.mk' file that looks like:
++
+------
+include $(BR2_EXTERNAL)/package/*/*.mk
+------
++
+And then in +BR2_EXTERNAL/package/package1+ and
+   +BR2_EXTERNAL/package/package2+ create normal Buildroot package
+   recipes, as explained in xref:adding-packages[].
+
+ * One can store Buildroot defconfigs in the +configs+ subdirectory of
+   +BR2_EXTERNAL+. Buildroot will automatically show them in the
+   output of +make help+ and allow them to be loaded with the normal
+   +make <name>_defconfig+ command.
+
+In the end, a typical +BR2_EXTERNAL+ directory organization would
+generally be:
+
+-----
+??? Config.in
+??? external.mk
+??? board/
+?   ??? <boardname>/
+?       ??? overlay/
+?           ??? etc/
+?               ??? <some file>
+??? configs/
+?   ??? <boardname>_defconfig
+??? package/
+    ??? Config.in
+    ??? package1/
+        ??? Config.in
+        ??? package1.mk
+    ??? package2/
+        ??? Config.in
+        ??? package2.mk
+------
diff --git a/docs/manual/customize.txt b/docs/manual/customize.txt
index 0456ef1..7e46fd8 100644
--- a/docs/manual/customize.txt
+++ b/docs/manual/customize.txt
@@ -17,3 +17,5 @@ include::customize-toolchain.txt[]
 include::customize-store.txt[]
 
 include::customize-packages.txt[]
+
+include::customize-outside-br.txt[]
-- 
1.8.1.2

[Buildroot] [PATCHv2 4/4] docs/manual: add explanations about BR2_EXTERNAL

[Simon Dawson]

On 14 September 2013 20:03, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> +This +BR2_EXTERNAL+ then allows three different things:
> +
> + * One can store all the board-specific configuration files here, such
> +   as the kernel configuration, the root filesystem overlay, or any
> +   other configuration file for which Buildroot allows to set its
> +   location. The +BR2_EXTERNAL+ value is available within the
> +   Buildroot configuration using +$(BR2_EXTERNAL)+. As an example, one
> +   could set the +BR2_ROOTFS_OVERLAY+ Buildroot option to
> +   +$(BR2_EXTERNAL)/board/<boardname>/overlay/+ (to specify a root
> +   filesystem overlay), or the +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
> +   Buildroot option to +$(BR2_EXTERNAL)/board/<boardname>/kernel.config+
> +   (to specify the location of the kernel configuration file).  + To
> +   achieve this, it is recommended but not mandatory, to store those
> +   details in directories called +board/<boardname>/+ under
> +   +BR2_EXTERNAL+.

A minor point: this is slightly inconsistent with the existing
Buildroot manual (see customize-store.txt), where the root filesystem
overlay subdirectory is named "fs-overlay", and the kernel
configuration is named "linux.config".

> +The main usage of this is to store package recipes. The recommended
> +   way to do this is to write a +BR2_EXTERNAL/Config.in+ that looks
> +   like:
> ++
> +------
> +menu "<somecompany> packages"
> +
> +source "$BR2_EXTERNAL/package/package1/Config.in"
> +source "$BR2_EXTERNAL/package/package2/Config.in"
> +
> +endmenu
> +------

Shouldn't this rather be in the $BR2_EXTERNAL/package/Config.in file
(which appears in your example overlay directory layout tree diagram),
which should itself be sourced from the $BR2_EXTERNAL/Config.in file?

Simon.

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Simon Dawson]

On 14 September 2013 20:03, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> Here is the second version of the BR2_EXTERNAL patch series.

This is looking really good, and fits my typical workflow very nicely.

> Remaning questions
> ==================
>
>  * Interaction with the 'local.mk' mechanism. Should 'external.mk'
>    replace the 'local.mk' mechanism? Should they co-exist?

This is my only slight concern. I would personally prefer to see the
local.mk mechanism replaced by the new external.mk mechanism; it just
feels wrong to have both mechanisms in there.

Simon.

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Thomas Petazzoni]

Dear Simon Dawson,

On Sat, 14 Sep 2013 20:39:32 +0100, Simon Dawson wrote:

> > Remaning questions
> > ==================
> >
> >  * Interaction with the 'local.mk' mechanism. Should 'external.mk'
> >    replace the 'local.mk' mechanism? Should they co-exist?
> 
> This is my only slight concern. I would personally prefer to see the
> local.mk mechanism replaced by the new external.mk mechanism; it just
> feels wrong to have both mechanisms in there.

I tend to agree, but there are some differences between the two
mechanisms:

 (1) the location/name of local.mk is defined within the .config, while
 the BR2_EXTERNAL location must be passed as environment variable.

 (2) with the BR2_EXTERNAL mechanism, you _have_ to create a
 BR2_EXTERNAL/Config.in file (even empty), which is not needed with
 local.mk when all what you want is to use the OVERRIDE_SRCDIR
 mechanism.

 (3) The local.mk is included *before* the package .mk files from
 Buildroot (so that the <foo>_OVERRIDE_SRCDIR is defined before
 'foo.mk' is included and calls its generic-package/autotools-package
 macro), while the external.mk file is included *after* all
 Buildroot package .mk files.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Simon Dawson]

On 14 September 2013 20:55, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> On Sat, 14 Sep 2013 20:39:32 +0100, Simon Dawson wrote:
>> This is my only slight concern. I would personally prefer to see the
>> local.mk mechanism replaced by the new external.mk mechanism; it just
>> feels wrong to have both mechanisms in there.
>
> I tend to agree, but there are some differences between the two
> mechanisms:
>
>  (1) the location/name of local.mk is defined within the .config, while
>  the BR2_EXTERNAL location must be passed as environment variable.
>
>  (2) with the BR2_EXTERNAL mechanism, you _have_ to create a
>  BR2_EXTERNAL/Config.in file (even empty), which is not needed with
>  local.mk when all what you want is to use the OVERRIDE_SRCDIR
>  mechanism.

Hmm, yes. That does complicate the issue somewhat. That suggests to me
that we should either:

* keep both local.mk and external.mk, and just live with the slight
aesthetic niggle
* do away with the external.mk

>  (3) The local.mk is included *before* the package .mk files from
>  Buildroot (so that the <foo>_OVERRIDE_SRCDIR is defined before
>  'foo.mk' is included and calls its generic-package/autotools-package
>  macro), while the external.mk file is included *after* all
>  Buildroot package .mk files.

Is there any particular reason to include external.mk after the
internal Buildroot package .mk files? (Sorry if this has been
explained already somewhere.)

Simon.

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Yann E. MORIN]

Thomas, All,

On 2013-09-14 21:03 +0200, Thomas Petazzoni spake thusly:
> Here is the second version of the BR2_EXTERNAL patch series.
> 
> Changes since v1
[--SNIP--]
>  * BR2_EXTERNAL is turned into an absolute path, in order to avoid any
>    problem. Suggested by Arnout.

I dsiagree on that one: just use whatever the user passed in
BR2_EXTERNAL.

I see at least one case where it will help. Let's suppose I decide to
distribute my Buildroot tree *and* my external tree. I decide to do so
as thus:
    $ tar tf project.tar
    project/
    project/buildroot/
    project/buildroot/all-buildroot-files-and-dirs
    project/buildroot/.config
    project/external/
    project/external/all-external-files-and-dirs

and project/buildroot/.config would contain BR2_EXTERNAL="../external"
so the recipient would just have to run make withour hurdles.

If .config would contain absolute paths, .config would be broken as
soon as it gets distribute (chances the recipient uses the same
directory layout as my build machine is near zero).

I think we should instead try to fix the cases were relative paths are
broken, and use exactly what the user passes us, as-is, without mangling.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sun, 15 Sep 2013 00:30:44 +0200, Yann E. MORIN wrote:

> I see at least one case where it will help. Let's suppose I decide to
> distribute my Buildroot tree *and* my external tree. I decide to do so
> as thus:
>     $ tar tf project.tar
>     project/
>     project/buildroot/
>     project/buildroot/all-buildroot-files-and-dirs
>     project/buildroot/.config
>     project/external/
>     project/external/all-external-files-and-dirs
> 
> and project/buildroot/.config would contain BR2_EXTERNAL="../external"
> so the recipient would just have to run make withour hurdles.

The BR2_EXTERNAL value is *NOT* stored in .config. It must be passed as
an environment variable, so in the case you describe above, you would
have to distribute project.tar, with a small document that says:

""
To build the stuff:

  1/ cd project/buildroot/
  2/ make BR2_EXTERNAL=../external
""

The fact that BR2_EXTERNAL is *not* part of the .config has already
been discussed in the thread of the previous version. The "config
BR2_EXTERNAL" option is prompt-less and therefore the value of this
variable is not stored in .config.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Ryan Barnett]

Thomas,

Per our chat on IRC, putting BR2_EXTERNAL into the generated makefile
isn't quite working out like I expected. I'm losing the BR2_EXTERNAL
variable definition.

Here are the steps I'm taking after applying your v2 of this patch to
a git clone of the mainline of buildroot. The buildroot clone is not
in the directory of ~/tmp but elsewhere.

make O=~/tmp/build_xm BR2_EXTERNAL=~/tmp/buildroot 
rc_beagle_xm_min_defconfig
cd ~/tmp/build_xm
make

Here is the output from the make:

make -C /home/rjbarnet/projects/buildroot/buildroot-git 
O=/home/rjbarnet/projects/muos/build_xm/. 
BR2_EXTERNAL=/home/rjbarnet/projects/muos/muos-svn/buildroot
/usr/bin/make -j8 O=/home/rjbarnet/projects/muos/build_xm/. 
HOSTCC="/usr/bin/gcc" HOSTCXX="/usr/bin/g++" silentoldconfig
  GEN     /home/rjbarnet/projects/muos/build_xm/./Makefile
BR2_DEFCONFIG='' 
KCONFIG_AUTOCONFIG=/home/rjbarnet/projects/muos/build_xm/build/buildroot-config/auto.conf 
KCONFIG_AUTOHEADER=/home/rjbarnet/projects/muos/build_xm/build/buildroot-config/autoconf.h 
KCONFIG_TRISTATE=/home/rjbarnet/projects/muos/build_xm/build/buildroot-config/tristate.config 
BUILDROOT_CONFIG=/home/rjbarnet/projects/muos/build_xm/./.config 
BR2_EXTERNAL=/home/rjbarnet/projects/buildroot/buildroot-git/support/dummy-external/ 
/home/rjbarnet/projects/muos/build_xm/build/buildroot-config/conf 
--silentoldconfig Config.in
mkdir -p /home/rjbarnet/projects/muos/build_xm/target
rsync -a \
                --exclude .empty --exclude .svn --exclude .git \
                --exclude .hg --exclude=CVS --exclude '*~' \
 /home/rjbarnet/projects/buildroot/buildroot-git/system/skeleton/ 
/home/rjbarnet/projects/muos/build_xm/target/
cp support/misc/target-dir-warning.txt 
/home/rjbarnet/projects/muos/build_xm/target/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM
touch /home/rjbarnet/projects/muos/build_xm/build/.root
--2013-09-16 10:50:52-- 
<Stuff saying unable to downloading toolchain...>
--2013-09-16 10:50:55--  
http://sources.buildroot.net//gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux.tar.xz
<Stuff saying can't download toolcahin....>
make[1]: *** 
[/home/apps/opt/buildroot-2011.05/dl/gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux.tar.xz] 
Error 1
make: *** [all] Error 2

Note: I know the toolchain can't be downloaded, but the point is that 
I'm losing BR2_EXTERNAL path.

Here is the config that I'm using:

BR2_arm=y
BR2_cortex_a9=y
BR2_ARM_EABIHF=y
BR2_DL_DIR="/home/apps/opt/buildroot-2011.05/dl"
BR2_JLEVEL=8
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_LINARO_2013_04=y
BR2_TOOLCHAIN_EXTERNAL_GDB_SERVER_COPY=y
BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_DEVTMPFS=y
BR2_TARGET_GENERIC_GETTY_PORT="ttyO2"
BR2_ROOTFS_OVERLAY="$(BR2_EXTERNAL)/board/common-omap/local-skeleton"
BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_CUSTOM_VERSION=y
BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="3.8.13"
BR2_LINUX_KERNEL_PATCH="$(BR2_EXTERNAL)/board/common-omap/kernel-patches-3.8.13"
BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="$(BR2_EXTERNAL)/board/common-omap/linux-3.8.13.config"
BR2_LINUX_KERNEL_ZIMAGE=y
BR2_TARGET_ROOTFS_TAR_BZIP2=y
BR2_TARGET_UBOOT=y
BR2_TARGET_UBOOT_BOARDNAME="omap3_beagle"
BR2_TARGET_UBOOT_CUSTOM_PATCH_DIR="$(BR2_EXTERNAL)/board/common-omap/uboot-patches-2013.07"
BR2_TARGET_UBOOT_FORMAT_IMG=y
BR2_TARGET_UBOOT_SPL=y
BR2_TARGET_UBOOT_SPL_NAME="MLO"
BR2_PACKAGE_HOST_E2FSPROGS=y
BR2_PACKAGE_HOST_GENEXT2FS=y

Thanks,
-Ryan

-----------------------------------------------------------------
Ryan J Barnett / Software Engineer / Platform SW
MS 137-157, 855 35th St NE, Cedar Rapids, IA, 52498-3161, US
Phone: 319-263-3880 / VPN: 263-3880 
rjbarnet at rockwellcollins.com
www.rockwellcollins.com 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130916/398d2c22/attachment.html>

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Thomas Petazzoni]

Dear Ryan Barnett,

On Mon, 16 Sep 2013 11:34:39 -0500, Ryan Barnett wrote:

> Per our chat on IRC, putting BR2_EXTERNAL into the generated makefile
> isn't quite working out like I expected. I'm losing the BR2_EXTERNAL
> variable definition.
> 
> Here are the steps I'm taking after applying your v2 of this patch to
> a git clone of the mainline of buildroot. The buildroot clone is not
> in the directory of ~/tmp but elsewhere.
> 
> make O=~/tmp/build_xm BR2_EXTERNAL=~/tmp/buildroot 
> rc_beagle_xm_min_defconfig
> cd ~/tmp/build_xm
> make

Ah, right. The below patch fixes that:

diff --git a/Makefile b/Makefile
index 799c529..5d942bd 100644
--- a/Makefile
+++ b/Makefile
@@ -59,6 +59,8 @@ else
 BR2_EXTERNAL = $(TOPDIR)/support/dummy-external/
 endif
 
+EXTRAMAKEARGS += BR2_EXTERNAL="$(BR2_EXTERNAL)"
+
 # Compute the full local version string so packages can use it as-is
 # Need to export it, so it can be got from environment in children (eg. mconf)
 export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)
@@ -95,7 +97,7 @@ MAKEOVERRIDES =
 override O:=$(O)
 CONFIG_DIR:=$(O)
 # we need to pass O= everywhere we call back into the toplevel makefile
-EXTRAMAKEARGS = O=$(O)
+EXTRAMAKEARGS += O=$(O)
 NEED_WRAPPER=y
 endif
 



-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Arnout Vandecappelle]

  Hi all,

On 14/09/13 21:03, Thomas Petazzoni wrote:
> Here is the second version of the BR2_EXTERNAL patch series.

  I've taken a few days to let the dust settle on this patch series (and 
it seems it still hasn't settled...). After reading the entire series 
this time (instead of commenting before I understood what was happening 
exactly), I have two overall comments still. More comments will follow in 
individual patches.


1. Peter nacked all previous attempts of having this type of feature, so 
I think it's best if he gives an indication if this series has a chance 
of survival before we all spend more time on it.


2. I really have a huge problem with the fact that the .config is no 
longer complete, and that instead you have to explicitly provide extra 
command-line arguments to be able to reproduce the build. And it doesn't 
help that it behaves differently when you have an out-of-tree output 
directory (where the BR2_EXTERNAL is stored) than when you're using the 
default output directory. I'm actually very surprised that nobody else 
seems to have a problem with this. So for me, this gets a big Nack unless 
the BR2_EXTERNAL is stored in the .config.



  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Arnout Vandecappelle]

On 14/09/13 21:03, Thomas Petazzoni wrote:
> This commit introduces the BR2_EXTERNAL environment variable, which
> will allow to keep Buildroot customization (board-specific
> configuration files or root filesystem overlays, package Config.in and
> makefiles, as well as defconfigs) outside of the Buildroot tree.
>
> This commit only introduces the variable itself, and ensures that it
> is available within Config.in options, so that string options used to
> specify paths to directories or files can use $BR2_EXTERNAL as a

  $(BR2_EXTERNAL). The $ is expanded by make. Same comment applies to 
other places in the commit message as well. You did put it correctly in 
the manual.

> reference. For example, one can use
> $BR2_EXTERNAL/board/<someboard>/kernel.config as the
> BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE value.
>
> Following patches extend the usage of BR2_EXTERNAL to other areas
> (packages and defconfigs).
>
> In details, this commit:
>
>   * Introduces the BR2_EXTERNAL Kconfig option. This option has no
>     prompt, and is therefore not visible to the user and also not
>     stored in the .config file. It is automatically set to the value of
>     the BR2_EXTERNAL environment variable. The only purpose of this
>     BR2_EXTERNAL Kconfig option is to allow $BR2_EXTERNAL to be
>     properly expanded when used inside Kconfig option values.
>
>   * Ensures that the path given in BR2_EXTERNAL is an absolute path, or
>     if not path is given, ensures that BR2_EXTERNAL points to a dummy
>     implementation in $(TOPDIR)/support/dummy-external/. This last part
>     is not directly useful in this commit, but is needed to properly
>     support Config.in options declared in BR2_EXTERNAL (next commit).

  It would make more sense to do it in that commit then, but that's just 
nitpicking.

>
>   * Passes the BR2_EXTERNAL into the *config environment, so that its
>     value is found when parsing/evaluating Config.in files and .config
>     values.

  That's pretty useless in this commit, since it anyway doesn't end up in 
.config and isn't used in Config.in.

>
>   * Encodes the BR2_EXTERNAL value into the Makefile wrapper (only if
>     BR2_EXTERNAL is specified by the user, not if the dummy internal
>     value is used).
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>   Config.in                  |  4 ++++
>   Makefile                   | 13 +++++++++++--
>   support/scripts/mkmakefile |  3 ++-
>   3 files changed, 17 insertions(+), 3 deletions(-)
>
> diff --git a/Config.in b/Config.in
> index 65c29b4..522eaf8 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -14,6 +14,10 @@ config BR2_HOSTARCH
>   	string
>   	option env="HOSTARCH"
>
> +config BR2_EXTERNAL
> +	string
> +	option env="BR2_EXTERNAL"
> +

  If you agree to store the value in the .config file, it could look 
something like this.

-------

config BR2_EXTERNAL_FROM_ENV
         string
         option env="BR2_EXTERNAL"

# This condition forces the user to set BR2_EXTERNAL from the
# command line the first time.
if BR2_EXTERNAL_FROM_ENV != ""
config BR2_EXTERNAL
         string "External buildroot add-ons"
         default BR2_EXTERNAL_FROM_ENV
         help
           List of directories with buildroot add-ons. ...
endif

if BR2_EXTERNAL != BR2_EXTERNAL_FROM_ENV
comment "You need to re-run the config to see the new packages"
endif


-------

  It doesn't really work, though, because currently the .config is not 
read in when doing 'make menuconfig', so that BR2_EXTERNAL_FROM_ENV will 
always be "". But I'm sure we'll be able to find something that works 
with a little more effort.


>   # Hidden boolean selected by pre-built packages for x86, when they
>   # need to run on x86-64 machines (example: pre-built external
>   # toolchains, binary tools like SAM-BA, etc.).
> diff --git a/Makefile b/Makefile
> index 8610592..13ad342 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -51,6 +51,14 @@ CONFIG_CONFIG_IN=Config.in
>   CONFIG=support/kconfig
>   DATE:=$(shell date +%Y%m%d)
>
> +# Turn BR2_EXTERNAL into an absolute path
> +ifneq ($(BR2_EXTERNAL),)
> +override BR2_EXTERNAL := $(realpath $(BR2_EXTERNAL))
> +BR2_EXTERNAL_USED = y
> +else
> +BR2_EXTERNAL = $(TOPDIR)/support/dummy-external/
> +endif
> +
>   # Compute the full local version string so packages can use it as-is
>   # Need to export it, so it can be got from environment in children (eg. mconf)
>   export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setlocalversion)
> @@ -632,7 +640,8 @@ COMMON_CONFIG_ENV = \
>   	KCONFIG_AUTOCONFIG=$(BUILD_DIR)/buildroot-config/auto.conf \
>   	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
>   	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
> -	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG)
> +	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG) \
> +	BR2_EXTERNAL=$(BR2_EXTERNAL)
>
>   xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
>   	@mkdir -p $(BUILD_DIR)/buildroot-config
> @@ -739,7 +748,7 @@ source-check:
>   # output directory.
>   outputmakefile:
>   ifeq ($(NEED_WRAPPER),y)
> -	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
> +	$(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O) $(if $(BR2_EXTERNAL_USED),$(BR2_EXTERNAL))
>   endif
>
>   # printvars prints all the variables currently defined in our Makefiles
> diff --git a/support/scripts/mkmakefile b/support/scripts/mkmakefile
> index cef2ec7..e7c1bd0 100755
> --- a/support/scripts/mkmakefile
> +++ b/support/scripts/mkmakefile
> @@ -6,7 +6,7 @@
>   # Usage
>   # $1 - Kernel src directory
>   # $2 - Output directory
> -
> +# $3 - BR2_EXTERNAL directory
>
>   test ! -r $2/Makefile -o -O $2/Makefile || exit 0
>   # Only overwrite automatically generated Makefiles
> @@ -27,6 +27,7 @@ makedir := \$(dir \$(call lastword,\$(MAKEFILE_LIST)))
>
>   MAKEARGS := -C $1
>   MAKEARGS += O=\$(if \$(patsubst /%,,\$(makedir)),\$(CURDIR)/)\$(patsubst %/,%,\$(makedir))
> +MAKEARGS += BR2_EXTERNAL=$3

  Obviously, this won't be needed anymore if the EXTERNAL is stored in 
the .config.


  Regards,
  Arnout

>
>   MAKEFLAGS += --no-print-directory
>
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCHv2 2/4] core: allow external Config.in/makefile code to be integrated

[Arnout Vandecappelle]

On 14/09/13 21:03, Thomas Petazzoni wrote:
> This commit allows the BR2_EXTERNAL directory to contain Config.in and
> Makefile code, which gets integrated into the Buildroot build logic:
>
>   - Buildroot automatically includes the $BR2_EXTERNAL/Config.in in the
>     top-level configuration menu.
>
>   - Buildroot automatically includes the BR2_EXTERNAL/external.mk in
>     the build logic, so it can for example be used to include other .mk
>     files that define package recipes.
>
> We also add a dummy Config.in file in support/dummy-external/ to
> ensure that the source "$BR2_EXTERNAL/Config.in" line will point to an
> existing file even when BR2_EXTERNAL is not used by the user.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>   Config.in                        | 2 ++
>   Makefile                         | 4 ++++
>   support/dummy-external/Config.in | 0
>   3 files changed, 6 insertions(+)
>   create mode 100644 support/dummy-external/Config.in
>
> diff --git a/Config.in b/Config.in
> index 522eaf8..9ef5466 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -465,3 +465,5 @@ source "boot/Config.in"
>   source "package/Config.in.host"
>
>   source "Config.in.legacy"
> +
> +source "$BR2_EXTERNAL/Config.in"

  I would prefer this to be in package/Config.in, and source 
$BR2_EXTERNAL/package/Config.in. I think 95% of the use cases will be for 
packages, and the other 5% are bootloaders which are the same as packages 
anyway. And even if it is not a proper generic package (e.g. it's a 
filesystem), putting it in the packages menu doesn't hurt that much. If 
you don't put it in the package subdirectory, then you either force users 
to create a oneline Config.in, or you end up with a messy looking 
directory mixing packages with the configs/ and board/ directories.


  Also it is perhaps better to add to the Makefile:

ifeq ($(wildcard $(BR2_EXTERNAL)/package/Config.in,)
EXTERNAL_PACKAGE_CONFIG_IN = support/dummy-external/Config.in
else
EXTERNAL_PACKAGE_CONFIG_IN = $(BR2_EXTERNAL)/package/Config.in
endif

(and pass that to the config env as well).

  That way, you're not obliged to have a Config.in in your external tree, 
and it is still possible to use it for nothing more than the defconfigs etc.

> diff --git a/Makefile b/Makefile
> index 13ad342..bea5ad5 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -348,6 +348,10 @@ include boot/common.mk
>   include linux/linux.mk
>   include system/system.mk
>
> +ifeq ($(BR2_EXTERNAL_USED),y)
> +include $(BR2_EXTERNAL)/external.mk

  Here I'd

-include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))


  Regards,
  Arnout

> +endif
> +
>   TARGETS+=target-finalize
>
>   ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
> diff --git a/support/dummy-external/Config.in b/support/dummy-external/Config.in
> new file mode 100644
> index 0000000..e69de29
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCHv2 3/4] core: allow external defconfigs to be used

[Arnout Vandecappelle]

On 14/09/13 21:03, Thomas Petazzoni wrote:
> This commit allows the user to store defconfigs in
> $BR2_EXTERNAL/configs/. To achieve this:
>
>   * It adds a new %_defconfig that looks in $BR2_EXTERNAL/configs/ for
>     the corresponding defconfig file.
>
>   * Updates the help target to also list external defconfigs.
>
> Signed-off-by: Thomas Petazzoni<thomas.petazzoni@free-electrons.com>

  Looks good!

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>


  Regards,
  Arnout


[snip]
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Mon, 16 Sep 2013 23:30:49 +0200, Arnout Vandecappelle wrote:
> On 14/09/13 21:03, Thomas Petazzoni wrote:
> > This commit introduces the BR2_EXTERNAL environment variable, which
> > will allow to keep Buildroot customization (board-specific
> > configuration files or root filesystem overlays, package Config.in
> > and makefiles, as well as defconfigs) outside of the Buildroot tree.
> >
> > This commit only introduces the variable itself, and ensures that it
> > is available within Config.in options, so that string options used
> > to specify paths to directories or files can use $BR2_EXTERNAL as a
> 
>   $(BR2_EXTERNAL). The $ is expanded by make. Same comment applies to 
> other places in the commit message as well. You did put it correctly
> in the manual.

Right, true.

> >   * Ensures that the path given in BR2_EXTERNAL is an absolute
> > path, or if not path is given, ensures that BR2_EXTERNAL points to
> > a dummy implementation in $(TOPDIR)/support/dummy-external/. This
> > last part is not directly useful in this commit, but is needed to
> > properly support Config.in options declared in BR2_EXTERNAL (next
> > commit).
> 
>   It would make more sense to do it in that commit then, but that's
> just nitpicking.

Yeah, but since it was really part of the same chunk of code, it was
annoying to split further. But I can certainly split that up further.

> >   * Passes the BR2_EXTERNAL into the *config environment, so that
> > its value is found when parsing/evaluating Config.in files
> > and .config values.
> 
>   That's pretty useless in this commit, since it anyway doesn't end
> up in .config and isn't used in Config.in.

Right, could be moved to the next commit then (but to me the separation
of the commits is relatively artificial, since the patch set really
only makes sense if everything is applied).

>   If you agree to store the value in the .config file, it could look 
> something like this.
> 
> -------
> 
> config BR2_EXTERNAL_FROM_ENV
>          string
>          option env="BR2_EXTERNAL"
> 
> # This condition forces the user to set BR2_EXTERNAL from the
> # command line the first time.
> if BR2_EXTERNAL_FROM_ENV != ""
> config BR2_EXTERNAL
>          string "External buildroot add-ons"
>          default BR2_EXTERNAL_FROM_ENV
>          help
>            List of directories with buildroot add-ons. ...
> endif
> 
> if BR2_EXTERNAL != BR2_EXTERNAL_FROM_ENV
> comment "You need to re-run the config to see the new packages"
> endif
> 
> 
> -------

I don't know. Do we really want to do that? Isn't it simpler to always
pass it as an environment variable?

I mean, if an user changes the value of BR2_EXTERNAL in menuconfig, he
has to exit / re-run menuconfig to anyway see the updated set of
configuration options after the BR2_EXTERNAL value change. Does it
really makes sense to set in menuconfig a configuration option that by
itself changes the available configuration options, but which requires
exiting/re-runnning menuconfig?

>   It doesn't really work, though, because currently the .config is
> not read in when doing 'make menuconfig', so that
> BR2_EXTERNAL_FROM_ENV will always be "". But I'm sure we'll be able
> to find something that works with a little more effort.

Hum, the .config is not read when doing 'make menuconfig'? Then how can
'make menuconfig' show the current state of enabled options?

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCHv2 2/4] core: allow external Config.in/makefile code to be integrated

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Mon, 16 Sep 2013 23:39:34 +0200, Arnout Vandecappelle wrote:

>   I would prefer this to be in package/Config.in, and source 
> $BR2_EXTERNAL/package/Config.in. I think 95% of the use cases will be
> for packages, and the other 5% are bootloaders which are the same as
> packages anyway. And even if it is not a proper generic package (e.g.
> it's a filesystem), putting it in the packages menu doesn't hurt that
> much. If you don't put it in the package subdirectory, then you
> either force users to create a oneline Config.in, or you end up with
> a messy looking directory mixing packages with the configs/ and
> board/ directories.
> 
> 
>   Also it is perhaps better to add to the Makefile:
> 
> ifeq ($(wildcard $(BR2_EXTERNAL)/package/Config.in,)
> EXTERNAL_PACKAGE_CONFIG_IN = support/dummy-external/Config.in
> else
> EXTERNAL_PACKAGE_CONFIG_IN = $(BR2_EXTERNAL)/package/Config.in
> endif
> 
> (and pass that to the config env as well).
> 
>   That way, you're not obliged to have a Config.in in your external
> tree, and it is still possible to use it for nothing more than the
> defconfigs etc.
> 
> > diff --git a/Makefile b/Makefile
> > index 13ad342..bea5ad5 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -348,6 +348,10 @@ include boot/common.mk
> >   include linux/linux.mk
> >   include system/system.mk
> >
> > +ifeq ($(BR2_EXTERNAL_USED),y)
> > +include $(BR2_EXTERNAL)/external.mk
> 
>   Here I'd
> 
> -include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))

Right, it's a question of vision on what BR2_EXTERNAL should allow. As
I already stated, I originally had the same idea as you, but then
thought that including the external Config.in at the top-level, and
including an external.mk was offering more customization possibilities.

But I agree with you that 'enforcing' the usage of BR2_EXTERNAL only
for packages might be a good idea. It provides a clear guideline on how
BR2_EXTERNAL should be used/organized, which is probably good.

Unless there is opposition from others, I believe I'll move in the
direction you're proposing.

Thanks!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Mon, 16 Sep 2013 22:38:29 +0200, Arnout Vandecappelle wrote:

>   I've taken a few days to let the dust settle on this patch series
> (and it seems it still hasn't settled...). After reading the entire
> series this time (instead of commenting before I understood what was
> happening exactly), I have two overall comments still. More comments
> will follow in individual patches.
> 
> 
> 1. Peter nacked all previous attempts of having this type of feature,
> so I think it's best if he gives an indication if this series has a
> chance of survival before we all spend more time on it.

Adding Peter in Cc on this one. I know he is a bit behind in terms of
reading the backlog of Buildroot e-mails (which has been enormous since
a few days).

I have the pretension of thinking that the currently proposed
implementation is simple enough to remain in the KISS spirit of
Buildroot. I believe that what prevented earlier implementation from
being merged was not so much that they allowed 'external packages',
but more due to their complexity. But honestly, I don't really remember
the implementation details of the previous proposals, so I might be
wrong on that one.

> 2. I really have a huge problem with the fact that the .config is no 
> longer complete, and that instead you have to explicitly provide
> extra command-line arguments to be able to reproduce the build. And
> it doesn't help that it behaves differently when you have an
> out-of-tree output directory (where the BR2_EXTERNAL is stored) than
> when you're using the default output directory. I'm actually very
> surprised that nobody else seems to have a problem with this. So for
> me, this gets a big Nack unless the BR2_EXTERNAL is stored in
> the .config.

On the wrapper Makefile that contains BR2_EXTERNAL, I am not convinced
it is a good idea, since as you mention it creates a difference whether
you're building out-of-tree from the output directory, or out-of-tree
but from the source directory. I think this is not good, and therefore
that BR2_EXTERNAL shouldn't be kept in the wrapper Makefile.

However, I still do not understand how storing BR2_EXTERNAL in .config
will clarify things for the user. Changing BR2_EXTERNAL affects what
'menuconfig' shows. So there is no way for 'menuconfig' to update what
it displays when the user changes the value of BR2_EXTERNAL without
exiting/re-running menuconfig, which sounds really odd.

To me, BR2_EXTERNAL= is a bit like O=, it's a local build decision that
you make to add some external configurations/external packages. Most
likely, if you have external packages in BR2_EXTERNAL, the defconfig
you have to use is also in BR2_EXTERNAL, so there's no way you can
forget to specify BR2_EXTERNAL on the command, since otherwise you
won't be able to even see your defconfig.

So while I do agree that being able to store BR2_EXTERNAL would be
nice, I don't see how it can be achieved in a way that isn't confusing
for the user.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Arnout Vandecappelle]

On 17/09/13 06:37, Thomas Petazzoni wrote:
> Dear Arnout Vandecappelle,
>
> On Mon, 16 Sep 2013 22:38:29 +0200, Arnout Vandecappelle wrote:
>
>>    I've taken a few days to let the dust settle on this patch series
>> (and it seems it still hasn't settled...). After reading the entire
>> series this time (instead of commenting before I understood what was
>> happening exactly), I have two overall comments still. More comments
>> will follow in individual patches.
>>
>>
>> 1. Peter nacked all previous attempts of having this type of feature,
>> so I think it's best if he gives an indication if this series has a
>> chance of survival before we all spend more time on it.
>
> Adding Peter in Cc on this one. I know he is a bit behind in terms of
> reading the backlog of Buildroot e-mails (which has been enormous since
> a few days).
>
> I have the pretension of thinking that the currently proposed
> implementation is simple enough to remain in the KISS spirit of
> Buildroot. I believe that what prevented earlier implementation from
> being merged was not so much that they allowed 'external packages',
> but more due to their complexity. But honestly, I don't really remember
> the implementation details of the previous proposals, so I might be
> wrong on that one.
>
>> 2. I really have a huge problem with the fact that the .config is no
>> longer complete, and that instead you have to explicitly provide
>> extra command-line arguments to be able to reproduce the build. And
>> it doesn't help that it behaves differently when you have an
>> out-of-tree output directory (where the BR2_EXTERNAL is stored) than
>> when you're using the default output directory. I'm actually very
>> surprised that nobody else seems to have a problem with this. So for
>> me, this gets a big Nack unless the BR2_EXTERNAL is stored in
>> the .config.
>
> On the wrapper Makefile that contains BR2_EXTERNAL, I am not convinced
> it is a good idea, since as you mention it creates a difference whether
> you're building out-of-tree from the output directory, or out-of-tree
> but from the source directory. I think this is not good, and therefore
> that BR2_EXTERNAL shouldn't be kept in the wrapper Makefile.
>
> However, I still do not understand how storing BR2_EXTERNAL in .config
> will clarify things for the user. Changing BR2_EXTERNAL affects what
> 'menuconfig' shows. So there is no way for 'menuconfig' to update what
> it displays when the user changes the value of BR2_EXTERNAL without
> exiting/re-running menuconfig, which sounds really odd.
>
> To me, BR2_EXTERNAL= is a bit like O=, it's a local build decision that
> you make to add some external configurations/external packages. Most
> likely, if you have external packages in BR2_EXTERNAL, the defconfig
> you have to use is also in BR2_EXTERNAL, so there's no way you can
> forget to specify BR2_EXTERNAL on the command, since otherwise you
> won't be able to even see your defconfig.

  But you usually don't run "make foo_defconfig; make". You usually run 
either "make menuconfig; make" or "make foo-dirclean; make" or some other 
variant.

  I just think it's a really bad idea to split the buildroot 
configuration over the .config file and the environment. The concept just 
scares the hell out of me.

  The comparison with O= is a good one: O= makes pretty damn sure that 
things remain consistent, because the .config is stored in the O= 
directory rather than someplace else.



> So while I do agree that being able to store BR2_EXTERNAL would be
> nice, I don't see how it can be achieved in a way that isn't confusing
> for the user.

  Since the user probably is never going to change the value of 
BR2_EXTERNAL, I think that's a relatively small price to pay and that it 
can be solved with documentation.

  Regards,
  Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Arnout Vandecappelle]

On 17/09/13 06:26, Thomas Petazzoni wrote:
> Dear Arnout Vandecappelle,
>
> On Mon, 16 Sep 2013 23:30:49 +0200, Arnout Vandecappelle wrote:
>> On 14/09/13 21:03, Thomas Petazzoni wrote:
>>> This commit introduces the BR2_EXTERNAL environment variable, which
>>> will allow to keep Buildroot customization (board-specific
>>> configuration files or root filesystem overlays, package Config.in
>>> and makefiles, as well as defconfigs) outside of the Buildroot tree.
>>>
>>> This commit only introduces the variable itself, and ensures that it
>>> is available within Config.in options, so that string options used
>>> to specify paths to directories or files can use $BR2_EXTERNAL as a
>>
>>    $(BR2_EXTERNAL). The $ is expanded by make. Same comment applies to
>> other places in the commit message as well. You did put it correctly
>> in the manual.
>
> Right, true.
>
>>>    * Ensures that the path given in BR2_EXTERNAL is an absolute
>>> path, or if not path is given, ensures that BR2_EXTERNAL points to
>>> a dummy implementation in $(TOPDIR)/support/dummy-external/. This
>>> last part is not directly useful in this commit, but is needed to
>>> properly support Config.in options declared in BR2_EXTERNAL (next
>>> commit).
>>
>>    It would make more sense to do it in that commit then, but that's
>> just nitpicking.
>
> Yeah, but since it was really part of the same chunk of code, it was
> annoying to split further. But I can certainly split that up further.
>
>>>    * Passes the BR2_EXTERNAL into the *config environment, so that
>>> its value is found when parsing/evaluating Config.in files
>>> and .config values.
>>
>>    That's pretty useless in this commit, since it anyway doesn't end
>> up in .config and isn't used in Config.in.
>
> Right, could be moved to the next commit then (but to me the separation
> of the commits is relatively artificial, since the patch set really
> only makes sense if everything is applied).

  Fair enough.


>>    If you agree to store the value in the .config file, it could look
>> something like this.
>>
>> -------
>>
>> config BR2_EXTERNAL_FROM_ENV
>>           string
>>           option env="BR2_EXTERNAL"
>>
>> # This condition forces the user to set BR2_EXTERNAL from the
>> # command line the first time.
>> if BR2_EXTERNAL_FROM_ENV != ""
>> config BR2_EXTERNAL
>>           string "External buildroot add-ons"
>>           default BR2_EXTERNAL_FROM_ENV
>>           help
>>             List of directories with buildroot add-ons. ...
>> endif
>>
>> if BR2_EXTERNAL != BR2_EXTERNAL_FROM_ENV
>> comment "You need to re-run the config to see the new packages"
>> endif
>>
>>
>> -------
>
> I don't know. Do we really want to do that? Isn't it simpler to always
> pass it as an environment variable?
>
> I mean, if an user changes the value of BR2_EXTERNAL in menuconfig, he
> has to exit / re-run menuconfig to anyway see the updated set of
> configuration options after the BR2_EXTERNAL value change. Does it
> really makes sense to set in menuconfig a configuration option that by
> itself changes the available configuration options, but which requires
> exiting/re-runnning menuconfig?

  The primary way of setting BR2_EXTERNAL would still be through the 
environment - in fact, the code above forces you to set it through the 
environment the first time, otherwise the option won't be visible.


>>    It doesn't really work, though, because currently the .config is
>> not read in when doing 'make menuconfig', so that
>> BR2_EXTERNAL_FROM_ENV will always be "". But I'm sure we'll be able
>> to find something that works with a little more effort.
>
> Hum, the .config is not read when doing 'make menuconfig'? Then how can
> 'make menuconfig' show the current state of enabled options?

  I mena that it isn't read by the Makefile.

  Regards,
  Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[rjbarnet at rockwellcollins.com]

Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote on 
09/16/2013 11:37:17 PM:

> Dear Arnout Vandecappelle,
> 
> On Mon, 16 Sep 2013 22:38:29 +0200, Arnout Vandecappelle wrote:
[...]
> > 2. I really have a huge problem with the fact that the .config is no 
> > longer complete, and that instead you have to explicitly provide
> > extra command-line arguments to be able to reproduce the build. And
> > it doesn't help that it behaves differently when you have an
> > out-of-tree output directory (where the BR2_EXTERNAL is stored) than
> > when you're using the default output directory. I'm actually very
> > surprised that nobody else seems to have a problem with this. So for
> > me, this gets a big Nack unless the BR2_EXTERNAL is stored in
> > the .config.
> 
> On the wrapper Makefile that contains BR2_EXTERNAL, I am not convinced
> it is a good idea, since as you mention it creates a difference whether
> you're building out-of-tree from the output directory, or out-of-tree
> but from the source directory. I think this is not good, and therefore
> that BR2_EXTERNAL shouldn't be kept in the wrapper Makefile.

I do think that concerns those are valid concerns but from my use with 
BR2_EXTERNAL (with both v1 and v2) here is my take away with the 
experience. Note: I've been using this almost exclusively the last week
since I've had to put together some new board configuration.

Having to make sure that BR2_EXTERNAL is defined when you call make 
gets very annoying quickly. I understand that I can just export 
BR2_EXTERNAL 
to your environment to not have to declare BR2_EXTERNAL every time. 
So to make my life easier, I've did just that but have run into one 
major gotcha with it. Exporting BR2_EXTERNAL to the environment is 
just as unclear as being written to the wrapper makefile and if I 
come back to shell and forget to check what BR2_EXTERNAL is, I can
perform a build with BR2_EXTERNAL that isn't correct for that board.

Well beta testing this latest patchset, I've found that putting it the 
wrapper makefile has allowed to not always have to keep putting 
BR2_EXTERNAL
in but also not have BR2_EXTERNAL seek into other builds/configurations.
I know in my external build folder that I'm using X defconfig the Y 
BR2_EXTERNAL and I usually don't have any reason to change Y because 
that folder has been dedicated to a specific project that share common
BR2_EXTERNAL.

Maybe with BR2_EXTERNAL, we can encourage the use of O= since to quote 
Thomas from below: "it's a local build decision"? I don't know what is
best do here since the way BR2_EXTERNAL has been implement here I feel
is the best way and since BR2_EXTERNAL is completely optional I don't
know if encouraging the use O= is a bad thing.
 
> However, I still do not understand how storing BR2_EXTERNAL in .config
> will clarify things for the user. Changing BR2_EXTERNAL affects what
> 'menuconfig' shows. So there is no way for 'menuconfig' to update what
> it displays when the user changes the value of BR2_EXTERNAL without
> exiting/re-running menuconfig, which sounds really odd.
> 
> To me, BR2_EXTERNAL= is a bit like O=, it's a local build decision that
> you make to add some external configurations/external packages. Most
> likely, if you have external packages in BR2_EXTERNAL, the defconfig
> you have to use is also in BR2_EXTERNAL, so there's no way you can
> forget to specify BR2_EXTERNAL on the command, since otherwise you
> won't be able to even see your defconfig.

I completely agree with this as while using this patchset, I've forgotten
to specify BR2_EXTERNAL so I don't get see the defconfig I'm looking for.
I guess is a subtle reminder that I haven't specified BR2_EXTERNAL.

My usual workflow while using this patchset is follows:

make BR2_EXTERNAL=/some/path help
[find the defconfig I want to build for]
make BR2_EXTERNAL=/some/path O=/build/path defconfig
cd /build/path
[do buildroot menuconfig stuff if needed]]
make

Here is my workflow when doing a new BR2_EXTERNAL build that I've found
useful

make BR2_EXTERNAL=/some/path O=/build/path allnoconfig
cd /build/path
make menuconfig
[make my buildroot configuration selections]
make savedefconfig 
[NOTE: in buildroot menuconfig the save defconfig option has set to 
$(BR2_EXTERNAL)/configs/some_defconfig]
 
> So while I do agree that being able to store BR2_EXTERNAL would be
> nice, I don't see how it can be achieved in a way that isn't confusing
> for the user.

I agree too, maybe 

> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, Free Electrons
> Kernel, drivers, real-time and embedded Linux
> development, consulting, training and support.
> http://free-electrons.com
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130917/b019f883/attachment-0001.html>

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Tue, 17 Sep 2013 08:10:11 +0200, Arnout Vandecappelle wrote:

> > I mean, if an user changes the value of BR2_EXTERNAL in menuconfig, he
> > has to exit / re-run menuconfig to anyway see the updated set of
> > configuration options after the BR2_EXTERNAL value change. Does it
> > really makes sense to set in menuconfig a configuration option that by
> > itself changes the available configuration options, but which requires
> > exiting/re-runnning menuconfig?
> 
>   The primary way of setting BR2_EXTERNAL would still be through the 
> environment - in fact, the code above forces you to set it through the 
> environment the first time, otherwise the option won't be visible.
> 
> 
> >>    It doesn't really work, though, because currently the .config is
> >> not read in when doing 'make menuconfig', so that
> >> BR2_EXTERNAL_FROM_ENV will always be "". But I'm sure we'll be able
> >> to find something that works with a little more effort.
> >
> > Hum, the .config is not read when doing 'make menuconfig'? Then how can
> > 'make menuconfig' show the current state of enabled options?
> 
>   I mena that it isn't read by the Makefile.

Ok, so what do you suggest exactly? Because you proposed an
implementation, but concluded with 'it doesn't work'. So I'm a little
bit confused here :-)

Thanks!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Ryan Barnett]

Thomas,

I've been using this patch for around a week now and today I tired to
generate the manual with this patch applied. However, "make manual"
fails pretty bad with this patch set applied. Have you tired 
generating the manual at all?

Here is what I've done:
 * Applied all four patches for v2 of BR2_EXTERNAL
 * make clean
 * make manual

I'm not exactly sure the root cause of the problem but my guess is
the top-level Config.in. To fix the problem, I just revert the 4 
patches and I can build the manual fine. When I do make manual I 
get the following error output:

/home/rjbarnet/projects/buildroot/buildroot-git/Config.in:19:warning:
The symbol BR2_EXTERNAL references the non-existent environment variable 
BR2_EXTERNAL and will
get the empty string as its value.

If you're using kconfiglib via 'make (i)scriptconfig' it should have set 
up the
environment correctly for you. If you still got this message, that might 
be an
error, and you should e-mail kconfiglib at gmail.com.
.
Traceback (most recent call last):
  File 
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/gen-manual-lists.py", 
line 376, in <module>
    buildroot = Buildroot()
  File 
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/gen-manual-lists.py", 
line 217, in __init__
    self.root_config))
  File 
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py", 
line 214, in __init__
    self.top_block = self._parse_file(filename, None, None, None)
  File 
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py", 
line 919, in _parse_file
    return self._parse_block(line_feeder, None, parent, deps, 
visible_if_deps, res)
  File 
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py", 
line 1114, in _parse_block
    self.base_dir))
IOError: /home/rjbarnet/projects/buildroot/buildroot-git/Config.in:490: 
sourced file "$BR2_EXTERNAL/Config.in" (expands to
"2_EXTERNAL/Config.in") not found. Perhaps base_dir
(argument to Config.__init__(), currently
"/home/rjbarnet/projects/buildroot/buildroot-git") is set to the wrong 
value.
make: *** [manual-update-lists] Error 1


Thanks,
-Ryan

Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote on 
09/14/2013 02:03:50 PM:
 
> This commit introduces the BR2_EXTERNAL environment variable, which
> will allow to keep Buildroot customization (board-specific
> configuration files or root filesystem overlays, package Config.in and
> makefiles, as well as defconfigs) outside of the Buildroot tree.
> 
> This commit only introduces the variable itself, and ensures that it
> is available within Config.in options, so that string options used to
> specify paths to directories or files can use $BR2_EXTERNAL as a
> reference. For example, one can use
> $BR2_EXTERNAL/board/<someboard>/kernel.config as the
> BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE value.
> 
> Following patches extend the usage of BR2_EXTERNAL to other areas
> (packages and defconfigs).
> 
> In details, this commit:
> 
>  * Introduces the BR2_EXTERNAL Kconfig option. This option has no
>    prompt, and is therefore not visible to the user and also not
>    stored in the .config file. It is automatically set to the value of
>    the BR2_EXTERNAL environment variable. The only purpose of this
>    BR2_EXTERNAL Kconfig option is to allow $BR2_EXTERNAL to be
>    properly expanded when used inside Kconfig option values.
> 
>  * Ensures that the path given in BR2_EXTERNAL is an absolute path, or
>    if not path is given, ensures that BR2_EXTERNAL points to a dummy
>    implementation in $(TOPDIR)/support/dummy-external/. This last part
>    is not directly useful in this commit, but is needed to properly
>    support Config.in options declared in BR2_EXTERNAL (next commit).
> 
>  * Passes the BR2_EXTERNAL into the *config environment, so that its
>    value is found when parsing/evaluating Config.in files and .config
>    values.
> 
>  * Encodes the BR2_EXTERNAL value into the Makefile wrapper (only if
>    BR2_EXTERNAL is specified by the user, not if the dummy internal
>    value is used).
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  Config.in                  |  4 ++++
>  Makefile                   | 13 +++++++++++--
>  support/scripts/mkmakefile |  3 ++-
>  3 files changed, 17 insertions(+), 3 deletions(-)
> 
> diff --git a/Config.in b/Config.in
> index 65c29b4..522eaf8 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -14,6 +14,10 @@ config BR2_HOSTARCH
>     string
>     option env="HOSTARCH"
> 
> +config BR2_EXTERNAL
> +   string
> +   option env="BR2_EXTERNAL"
> +
>  # Hidden boolean selected by pre-built packages for x86, when they
>  # need to run on x86-64 machines (example: pre-built external
>  # toolchains, binary tools like SAM-BA, etc.).
> diff --git a/Makefile b/Makefile
> index 8610592..13ad342 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -51,6 +51,14 @@ CONFIG_CONFIG_IN=Config.in
>  CONFIG=support/kconfig
>  DATE:=$(shell date +%Y%m%d)
> 
> +# Turn BR2_EXTERNAL into an absolute path
> +ifneq ($(BR2_EXTERNAL),)
> +override BR2_EXTERNAL := $(realpath $(BR2_EXTERNAL))
> +BR2_EXTERNAL_USED = y
> +else
> +BR2_EXTERNAL = $(TOPDIR)/support/dummy-external/
> +endif
> +
>  # Compute the full local version string so packages can use it as-is
>  # Need to export it, so it can be got from environment in children (eg. 
mconf)
>  export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell $(TOPDIR)/support/
> scripts/setlocalversion)
> @@ -632,7 +640,8 @@ COMMON_CONFIG_ENV = \
>     KCONFIG_AUTOCONFIG=$(BUILD_DIR)/buildroot-config/auto.conf \
>     KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
>     KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
> -   BUILDROOT_CONFIG=$(BUILDROOT_CONFIG)
> +   BUILDROOT_CONFIG=$(BUILDROOT_CONFIG) \
> +   BR2_EXTERNAL=$(BR2_EXTERNAL)
> 
>  xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
>     @mkdir -p $(BUILD_DIR)/buildroot-config
> @@ -739,7 +748,7 @@ source-check:
>  # output directory.
>  outputmakefile:
>  ifeq ($(NEED_WRAPPER),y)
> -   $(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O)
> +   $(Q)$(TOPDIR)/support/scripts/mkmakefile $(TOPDIR) $(O) $(if $
> (BR2_EXTERNAL_USED),$(BR2_EXTERNAL))
>  endif
> 
>  # printvars prints all the variables currently defined in our Makefiles
> diff --git a/support/scripts/mkmakefile b/support/scripts/mkmakefile
> index cef2ec7..e7c1bd0 100755
> --- a/support/scripts/mkmakefile
> +++ b/support/scripts/mkmakefile
> @@ -6,7 +6,7 @@
>  # Usage
>  # $1 - Kernel src directory
>  # $2 - Output directory
> -
> +# $3 - BR2_EXTERNAL directory
> 
>  test ! -r $2/Makefile -o -O $2/Makefile || exit 0
>  # Only overwrite automatically generated Makefiles
> @@ -27,6 +27,7 @@ makedir := \$(dir \$(call lastword,\$(MAKEFILE_LIST)))
> 
>  MAKEARGS := -C $1
>  MAKEARGS += O=\$(if \$(patsubst /%,,\$(makedir)),\$(CURDIR)/)\$
> (patsubst %/,%,\$(makedir))
> +MAKEARGS += BR2_EXTERNAL=$3
> 
>  MAKEFLAGS += --no-print-directory
> 
> -- 
> 1.8.1.2
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130923/7857176c/attachment-0001.html>

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Samuel Martin]

Hi Ryan, Thomas,


2013/9/23 Ryan Barnett <rjbarnet@rockwellcollins.com>
>
> Thomas,
>
> I've been using this patch for around a week now and today I tired to
> generate the manual with this patch applied. However, "make manual"
> fails pretty bad with this patch set applied. Have you tired
> generating the manual at all?
>
> Here is what I've done:
>  * Applied all four patches for v2 of BR2_EXTERNAL
>  * make clean
>  * make manual
>
> I'm not exactly sure the root cause of the problem but my guess is
> the top-level Config.in. To fix the problem, I just revert the 4
> patches and I can build the manual fine. When I do make manual I
> get the following error output:
>
> /home/rjbarnet/projects/buildroot/buildroot-git/Config.in:19:warning:
> The symbol BR2_EXTERNAL references the non-existent environment variable
BR2_EXTERNAL and will
> get the empty string as its value.
>
> If you're using kconfiglib via 'make (i)scriptconfig' it should have set
up the
> environment correctly for you. If you still got this message, that might
be an
> error, and you should e-mail kconfiglib at gmail.com.
> .
> Traceback (most recent call last):
>   File
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/gen-manual-lists.py",
line 376, in <module>
>     buildroot = Buildroot()
>   File
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/gen-manual-lists.py",
line 217, in __init__
>     self.root_config))
>   File
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py",
line 214, in __init__
>     self.top_block = self._parse_file(filename, None, None, None)
>   File
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py",
line 919, in _parse_file
>     return self._parse_block(line_feeder, None, parent, deps,
visible_if_deps, res)
>   File
"/home/rjbarnet/projects/buildroot/buildroot-git/support/scripts/kconfiglib.py",
line 1114, in _parse_block
>     self.base_dir))
> IOError: /home/rjbarnet/projects/buildroot/buildroot-git/Config.in:490:
sourced file "$BR2_EXTERNAL/Config.in" (expands to
> "2_EXTERNAL/Config.in") not found. Perhaps base_dir
> (argument to Config.__init__(), currently
> "/home/rjbarnet/projects/buildroot/buildroot-git") is set to the wrong
value.
> make: *** [manual-update-lists] Error 1

Good catch!
I missed it...

Could you try this patch: http://code.bulix.org/y27qmv-84542

Regards,


--
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130924/b6c49e3c/attachment.html>

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Ryan Barnett]

Samuel,

Samuel Martin <s.martin49@gmail.com> wrote on 
09/23/2013 05:17:08 PM:

> Hi Ryan, Thomas,
> 

[...]

> 
> Good catch!
> I missed it...
> 
> Could you try this patch: http://code.bulix.org/y27qmv-84542

Could you send this in an email? I can't get to any pastebin type 
website from behind my work's firewall.

> 
> Regards,
> 
> 
> --
> Samuel

Thanks,
-Ryan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130923/2c2b9f31/attachment.html>

[Buildroot] [PATCH] core: introduce the BR2_EXTERNAL variable (additional patch to fix manual build)

[Samuel Martin]

---
 docs/manual/manual.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
index 157d13c..8cd732a 100644
--- a/docs/manual/manual.mk
+++ b/docs/manual/manual.mk
@@ -1,6 +1,7 @@
 manual-update-lists: manual-check-dependencies-lists
 	$(Q)$(call MESSAGE,"Updating the manual lists...")
 	$(Q)BR2_DEFCONFIG="" TOPDIR=$(TOPDIR) O=$(O)/docs/manual/.build \
+		BR2_EXTERNAL=$(TOPDIR)/support/dummy-external \
 		$(TOPDIR)/support/scripts/gen-manual-lists.py
 
 manual-check-dependencies:
-- 
1.8.4

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Thomas Petazzoni]

Dear Ryan Barnett,

It's a problem I had already fixed in earlier versions, where I was
passing the proper BR2_EXTERNAL value, but Arnout disagreed in that
external packages should not be listed in the manual (which is
certainly correct), so I removed the variable. But indeed, there needs
to be a value, or the Kconfig logic is broken. The solution (below)
from Samuel is correct.

On Mon, 23 Sep 2013 17:30:02 -0500, Ryan Barnett wrote:

> Could you send this in an email? I can't get to any pastebin type 
> website from behind my work's firewall.

commit f160920f50898ee7fd8d96a4613af2e7a893db13
Author: Samuel Martin <s.martin49@gmail.com>
Date:   Tue Sep 24 00:07:09 2013 +0200

    manual: fix BR2_EXTERNAL support when building the manual
    
    Signed-off-by: Samuel Martin <s.martin49@gmail.com>

diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
index 157d13c..8cd732a 100644
--- a/docs/manual/manual.mk
+++ b/docs/manual/manual.mk
@@ -1,6 +1,7 @@
 manual-update-lists: manual-check-dependencies-lists
 	$(Q)$(call MESSAGE,"Updating the manual lists...")
 	$(Q)BR2_DEFCONFIG="" TOPDIR=$(TOPDIR) O=$(O)/docs/manual/.build \
+		BR2_EXTERNAL=$(TOPDIR)/support/dummy-external \
 		$(TOPDIR)/support/scripts/gen-manual-lists.py
 
 manual-check-dependencies:

-- 
Thomas Petazzoni, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCHv2 1/4] core: introduce the BR2_EXTERNAL variable

[Samuel Martin]

Ryan, Thomas,


2013/9/24 Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

> Dear Ryan Barnett,
>
> It's a problem I had already fixed in earlier versions, where I was
> passing the proper BR2_EXTERNAL value, but Arnout disagreed in that
> external packages should not be listed in the manual (which is
> certainly correct), so I removed the variable. But indeed, there needs
> to be a value, or the Kconfig logic is broken. The solution (below)
> from Samuel is correct.
>
> On Mon, 23 Sep 2013 17:30:02 -0500, Ryan Barnett wrote:
>
> > Could you send this in an email? I can't get to any pastebin type
> > website from behind my work's firewall.
>
> commit f160920f50898ee7fd8d96a4613af2e7a893db13
> Author: Samuel Martin <s.martin49@gmail.com>
> Date:   Tue Sep 24 00:07:09 2013 +0200
>
>     manual: fix BR2_EXTERNAL support when building the manual
>
>     Signed-off-by: Samuel Martin <s.martin49@gmail.com>
>
> diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
> index 157d13c..8cd732a 100644
> --- a/docs/manual/manual.mk
> +++ b/docs/manual/manual.mk
> @@ -1,6 +1,7 @@
>  manual-update-lists: manual-check-dependencies-lists
>         $(Q)$(call MESSAGE,"Updating the manual lists...")
>         $(Q)BR2_DEFCONFIG="" TOPDIR=$(TOPDIR) O=$(O)/docs/manual/.build \
> +               BR2_EXTERNAL=$(TOPDIR)/support/dummy-external \
>                 $(TOPDIR)/support/scripts/gen-manual-lists.py
>
>  manual-check-dependencies:
>

This patch fixes the undefined symbol error triggered by kconfiglib
package, but not the wrong symbol expansion.
Renaming BR2_EXTERNAL to BR_EXTERNAL is enough to solve the later issue.
This sounds like a kconfiglib bug... :(
Anyway, it does not explain why this was not triggered before...

Next step: understand and fix the expansion problem in kconfiglib

Regards,

-- 
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130924/84a91c54/attachment.html>

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Ryan Barnett]

Thomas/Yann/All,

Today on the IRC channel there was a discussion about BR2_EXTERNAL
and the remaining issues with the patch. Thomas Petazonni said that he
is waiting for the following two things:

1. The opinion of Jacmet (Peter Korsgaard), the Buildroot maintainer

2. A consensus with Arnout on whether BR2_EXTERNAL should be stored
   or not in the .config

For #1 - Thomas P. said that he was confident that he could convince
Peter K. of including BR2_EXTERNAL into buildroot.

For #2 - there was some dicussion on how to store BR2_EXTERNAL between
Thomas P., Yann E MORIN and myself about how best to accomplish this.

The discussion started with why BR2_EXTERNAL can't be a part of the
".config" file. As Thomas P. has stated many times before, there exists
a chicken-and-egg problem with BR2_EXTERNAL in that we can't store it
in the .config because menuconfig needs BR2_EXTERNAL to display the
external packages in the menuconfig and there is no way to pull
BR2_EXTERNAL's definition into menuconfig if it is in the config file.

The discussion then progressed with myself suggesting that we could put
a check to search if BR2_EXTERNAL is in the ".config" and then prompt a
warning when BR2_EXTERNAL exists in the ".config" but isn't defined in
the environment. However, I quickly realized that this wouldn't cover
the case where packages have been added to the .config from BR2_EXTERNAL
and are now no longer defined.

The discussion then continued with Thomas P. and myself talking about my
suggestion of having the wrapper makefile (out-of-tree (OOT) builds) 
store the value of BR2_EXTERNAL. Thomas P. said that storing 
BR2_EXTERNAL in the wrapper makefile is not good because non-OOT builds
don't work the same OOT builds. I then made the point that having to 
ensure BR2_EXTERNAL is defined everytime you call make is annoying and 
that defining in your shell environment is dangerous. This is when you 
are managing multiple builds (OOT builds) with different BR2_EXTERNALs 
and using same buildroot source.

Yann then asked me to the sum up what this issue was with in-tree builds
and BR2_EXTERNAL. Thomas P. then responded with "you have to pass the 
BR2_EXTERNAL value on the command line (or environment) at each 
Buildroot command"

I then suggested that BR2_EXTERNAL have a pre-requisite of being used
with only out-of-tree builds. The documentation for BR2_EXTERNAL should
then be updated to reflect this requirement and say something about this
requirement. 

Yann saw two options moving forward with this issue - 1) always require
BR2_EXTERNAL (in env or on command line) or 2) only support BR2_EXTERNAL
in OOT, stored in the wrapper.

Yann put the discussion of BR2_EXTERNAL on the agenda for the DevDays
coming up at the end of October:

http://elinux.org/Buildroot:DeveloperDaysELCE2013

Thomas P. suggested that maybe the wrapper makefile could be
genreated in the default output directory (O=output). Which, if a user
wanted to have the BR2_EXTERNAL remembered, they would have to change
their working directory to the default output directory (output) to use
the wrapper.

Finally, Thomas P. suggested that make BR2_EXTERNAL should be contained
in a file. I responded with asking how it would work with having
different BR2_EXTERNAL for different builds while using the same
buildroot source. To this question there was no response and that is
where things stand today.

To expand on Thomas P.'s idea of having a file, I'm starting to warm to
this idea as we could have buildroot default to look for a BR2_EXTERNAL
in the top-level of the buildroot source. If present, it would use the
directory that is listed there. However, we could override the location
of this file by specifying a variable fo the file in the wrapper 
makefile.

I hope I did a decent job capturing this discussion becuase I rushed
to put this together before I left for the day. 

Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote on 
09/14/2013 02:03:49 PM:

[Thomas P.'s cover page]

Thanks,
-Ryan

------------------------------------------------------------------------------------------
Ryan J Barnett / Software Engineer / Platform SW
MS 137-157, 855 35th St NE, Cedar Rapids, IA, 52498-3161, US
rjbarnet at rockwellcollins.com
www.rockwellcollins.com

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Yann E. MORIN]

Hello All!

On 2013-09-14 21:03 +0200, Thomas Petazzoni spake thusly:
> Here is the second version of the BR2_EXTERNAL patch series.

I was wondering what the plan with BR2_EXTERNAL was, in the end?

From my recollection of the last DevDay, I understood we would go with
it, with a few changes:
  - add a generated, sub-makefile in $(O) with the definition of
    BR2_EXTERNAL, so that it is not needed to repeat it on the command
    line of future invocations of 'make'
  - enforce the same directory layout, package/ board/ and configs/

Right?

If so, Thomas, since your the original submitter: do you have pending
changes to that effect, or are you waiting for someone to kick in and
continue the work on this patch series?

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCHv2 0/4] Add a BR2_EXTERNAL mechanism

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sun, 24 Nov 2013 21:20:25 +0100, Yann E. MORIN wrote:

> On 2013-09-14 21:03 +0200, Thomas Petazzoni spake thusly:
> > Here is the second version of the BR2_EXTERNAL patch series.
> 
> I was wondering what the plan with BR2_EXTERNAL was, in the end?
> 
> From my recollection of the last DevDay, I understood we would go with
> it, with a few changes:
>   - add a generated, sub-makefile in $(O) with the definition of
>     BR2_EXTERNAL, so that it is not needed to repeat it on the command
>     line of future invocations of 'make'
>   - enforce the same directory layout, package/ board/ and configs/
> 
> Right?
> 
> If so, Thomas, since your the original submitter: do you have pending
> changes to that effect, or are you waiting for someone to kick in and
> continue the work on this patch series?

I don't have pending changes right now. But since Ryan is in Toulouse
this week (and we've been doing some Buildroot hacking together), I
think we'll try to spend one evening on the BR2_EXTERNAL stuff and come
up with an updated proposal that takes into account the ELCE
discussions.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com