[Buildroot] [PATCH 1/1] Makefile: Support merged defconfigs

[Buildroot] [PATCH 1/1] Makefile: Support merged defconfigs

[Sam Bobroff]

Within the Linux kernel, and several other packages, it's currently
possible to maintain defconfigs as diffs (fragments) against other
configs but this is not possible with buildroot itself.

This patch adds the capability, although using a slightly different
implementation. Files may be added in the normal config directories
($TOPDIR/configs or $BR2_EXTERNAL/configs) with the format
"xxx_defconfig.merge" that contain, one per line, the files to be
passed to merge_config.sh to create the matching xxx_defconfig file
(the first line should contain the base config followed by the
fragments).  The generated defconfig file is then handled as it would
normally be.

Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
---
For a discussion about this from the OP build project, see
https://github.com/open-power/op-build/issues/457

A couple of things about the implementation:

* merge_config.sh will only output to a file called ".config" so I've set it up
  to output directly to $TOPDIR/.config, since that's about to be overwritten
  anyway, but I don't feel this is particularly nice.
* It seems strange to pass --defconfig=.config when running conf but I couldn't
  quickly tell if there was something better to do. Using --oldconfig would
  pick up .config by default but I don't know if it would then process the file
  correctly so I went with what seemed safe.
* I realise that there are now four very similar rules that handle defconfigs.
  I could look at factoring them but it didn't seem worth the effort and
  complexity at this point.

 Makefile | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/Makefile b/Makefile
index 71735b5..eb80152 100644
--- a/Makefile
+++ b/Makefile
@@ -845,6 +845,16 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(BR2_EXTERNAL)/configs/$@ \
 		$< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
 
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(TOPDIR)/configs/%_defconfig.merge outputmakefile
+	cd $(TOPDIR)/configs && xargs -a $@.merge -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
+	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
+		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
+
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig.merge outputmakefile
+	cd $(BR2_EXTERNAL)/configs && xargs -a $@.merge -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
+	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
+		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
+
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< \
 		--savedefconfig=$(if $(DEFCONFIG),$(DEFCONFIG),$(CONFIG_DIR)/defconfig) \
-- 
2.1.0

[Buildroot] [PATCH 1/1] Makefile: Support merged defconfigs

[Samuel Mendoza-Jonas]

On Mon, May 09, 2016 at 03:22:32PM +1000, Sam Bobroff wrote:
> Within the Linux kernel, and several other packages, it's currently
> possible to maintain defconfigs as diffs (fragments) against other
> configs but this is not possible with buildroot itself.
> 
> This patch adds the capability, although using a slightly different
> implementation. Files may be added in the normal config directories
> ($TOPDIR/configs or $BR2_EXTERNAL/configs) with the format
> "xxx_defconfig.merge" that contain, one per line, the files to be
> passed to merge_config.sh to create the matching xxx_defconfig file
> (the first line should contain the base config followed by the
> fragments).  The generated defconfig file is then handled as it would
> normally be.
> 
> Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>

This neatly solves some problems for us over in op-build land. Tested
with several fragments and output is what I expect:

Tested-by: Samuel Mendoza-Jonas <sam@mendozajonas.com>

> ---
> For a discussion about this from the OP build project, see
> https://github.com/open-power/op-build/issues/457
> 
> A couple of things about the implementation:
> 
> * merge_config.sh will only output to a file called ".config" so I've set it up
>   to output directly to $TOPDIR/.config, since that's about to be overwritten
>   anyway, but I don't feel this is particularly nice.
> * It seems strange to pass --defconfig=.config when running conf but I couldn't
>   quickly tell if there was something better to do. Using --oldconfig would
>   pick up .config by default but I don't know if it would then process the file
>   correctly so I went with what seemed safe.
> * I realise that there are now four very similar rules that handle defconfigs.
>   I could look at factoring them but it didn't seem worth the effort and
>   complexity at this point.
> 
>  Makefile | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 71735b5..eb80152 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -845,6 +845,16 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(BR2_EXTERNAL)/configs/$@ \
>  		$< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
>  
> +%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(TOPDIR)/configs/%_defconfig.merge outputmakefile
> +	cd $(TOPDIR)/configs && xargs -a $@.merge -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
> +	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
> +		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
> +
> +%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig.merge outputmakefile
> +	cd $(BR2_EXTERNAL)/configs && xargs -a $@.merge -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
> +	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
> +		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
> +
>  savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  	@$(COMMON_CONFIG_ENV) $< \
>  		--savedefconfig=$(if $(DEFCONFIG),$(DEFCONFIG),$(CONFIG_DIR)/defconfig) \
> -- 
> 2.1.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH 1/1] Makefile: Support merged defconfigs

[Cyril Bur]

On Mon,  9 May 2016 15:22:32 +1000
Sam Bobroff <sam.bobroff@au1.ibm.com> wrote:

> Within the Linux kernel, and several other packages, it's currently
> possible to maintain defconfigs as diffs (fragments) against other
> configs but this is not possible with buildroot itself.
> 
> This patch adds the capability, although using a slightly different
> implementation. Files may be added in the normal config directories
> ($TOPDIR/configs or $BR2_EXTERNAL/configs) with the format
> "xxx_defconfig.merge" that contain, one per line, the files to be
> passed to merge_config.sh to create the matching xxx_defconfig file
> (the first line should contain the base config followed by the
> fragments).  The generated defconfig file is then handled as it would
> normally be.
> 
> Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
> ---
> For a discussion about this from the OP build project, see
> https://github.com/open-power/op-build/issues/457
> 

Hi Sam,

> A couple of things about the implementation:
> 
> * merge_config.sh will only output to a file called ".config" so I've set it up
>   to output directly to $TOPDIR/.config, since that's about to be overwritten
>   anyway, but I don't feel this is particularly nice.
> * It seems strange to pass --defconfig=.config when running conf but I couldn't
>   quickly tell if there was something better to do. Using --oldconfig would
>   pick up .config by default but I don't know if it would then process the file
>   correctly so I went with what seemed safe.

Looks like what you've done will work. I've come to the same conclusion as you
that --oldconfig might actually be a better fit, and by extension
--olddefconfig may be even better moving forward as the fragments age and new
symbols get added.

Then it occurred to me I've had this question before and I had to go look what I
did for linux in arch/powerpc/Makefile and turns out, olddefconfig.

> * I realise that there are now four very similar rules that handle defconfigs.
>   I could look at factoring them but it didn't seem worth the effort and
>   complexity at this point.
> 

I have no doubt there may be a refactored solution in there somewhere but I
don't believe it will be as easy to follow as what currently exists. Currently
it is easy to understand why there are four, what they are trying to achieve,
I'd bet that with any refactoring that clarity will be dulled.

>  Makefile | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 71735b5..eb80152 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -845,6 +845,16 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(BR2_EXTERNAL)/configs/$@ \
>  		$< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
>  
> +%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(TOPDIR)/configs/%_defconfig.merge outputmakefile
> +	cd $(TOPDIR)/configs && xargs -a $@.merge -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
> +	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
> +		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
> +
> +%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig.merge outputmakefile
> +	cd $(BR2_EXTERNAL)/configs && xargs -a $@.merge -x -P 1 $(TOPDIR)/support/kconfig/merge_config.sh -m -O $(TOPDIR)
> +	$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/.config \
> +		$< --defconfig=$(TOPDIR)/.config $(CONFIG_CONFIG_IN)
> +
>  savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  	@$(COMMON_CONFIG_ENV) $< \
>  		--savedefconfig=$(if $(DEFCONFIG),$(DEFCONFIG),$(CONFIG_DIR)/defconfig) \

[Buildroot] [PATCH 1/1] Makefile: Support merged defconfigs

[Thomas Petazzoni]

Hello,

On Mon,  9 May 2016 15:22:32 +1000, Sam Bobroff wrote:
> Within the Linux kernel, and several other packages, it's currently
> possible to maintain defconfigs as diffs (fragments) against other
> configs but this is not possible with buildroot itself.
> 
> This patch adds the capability, although using a slightly different
> implementation. Files may be added in the normal config directories
> ($TOPDIR/configs or $BR2_EXTERNAL/configs) with the format
> "xxx_defconfig.merge" that contain, one per line, the files to be
> passed to merge_config.sh to create the matching xxx_defconfig file
> (the first line should contain the base config followed by the
> fragments).  The generated defconfig file is then handled as it would
> normally be.
> 
> Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>

Warning: the below is definitely not a final opinion or decision. It is
merely the start of a discussion.

I do like the usage of fragments. Your Github link at
https://github.com/open-power/op-build/issues/457 even points to
Buildroot training materials that I wrote, which explains how to use
fragments.

However, I am not sure whether we want in Buildroot a mechanism to
build a defconfig from fragments. To me, Buildroot should remain a
simple tool that takes as input a complete configuration, and generates
a Linux system from it.

Whether this "complete configuration" comes from a full .config, a
minimal defconfig, a defconfig generated from fragments, a defconfig
generated by a separate tool, by a webpage, or anything else, is not
Buildroot's matter. Buildroot's job starts when the configuration is
ready.

Essentially, the part that's missing in your commit is the update to
the documentation. And when you'll start updating the manual to explain
how this feature is working, you'll probably see that it's a bit tricky
to explain.

So, I'm definitely not saying "no" to this, but my belief is that it's
on the edges of Buildroot features. On the one hand, it brings a
well-defined and correct way of storing fragments and generated a
defconfig from fragments, that everyone can use and rely on. On the
other hand, it adds one more slightly "tricky" feature to Buildroot,
increasing its learning curve.

For a customer project, Gustavo has written a "genconfig" shell script,
that takes as input various parameters that define the system to be
built (which HW platform, which software stack, whether it should be a
debug or release build) and produces the defconfig for Buildroot.

What do others think?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 1/1] Makefile: Support merged defconfigs

[Arnout Vandecappelle]

On 05/12/16 16:02, Thomas Petazzoni wrote:
> Hello,
>
> On Mon,  9 May 2016 15:22:32 +1000, Sam Bobroff wrote:
>> Within the Linux kernel, and several other packages, it's currently
>> possible to maintain defconfigs as diffs (fragments) against other
>> configs but this is not possible with buildroot itself.
>>
>> This patch adds the capability, although using a slightly different
>> implementation. Files may be added in the normal config directories
>> ($TOPDIR/configs or $BR2_EXTERNAL/configs) with the format
>> "xxx_defconfig.merge" that contain, one per line, the files to be
>> passed to merge_config.sh to create the matching xxx_defconfig file
>> (the first line should contain the base config followed by the
>> fragments).  The generated defconfig file is then handled as it would
>> normally be.
>>
>> Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
>
> Warning: the below is definitely not a final opinion or decision. It is
> merely the start of a discussion.
>
> I do like the usage of fragments. Your Github link at
> https://github.com/open-power/op-build/issues/457 even points to
> Buildroot training materials that I wrote, which explains how to use
> fragments.
>
> However, I am not sure whether we want in Buildroot a mechanism to
> build a defconfig from fragments. To me, Buildroot should remain a
> simple tool that takes as input a complete configuration, and generates
> a Linux system from it.
>
> Whether this "complete configuration" comes from a full .config, a
> minimal defconfig, a defconfig generated from fragments, a defconfig
> generated by a separate tool, by a webpage, or anything else, is not
> Buildroot's matter. Buildroot's job starts when the configuration is
> ready.
>
> Essentially, the part that's missing in your commit is the update to
> the documentation. And when you'll start updating the manual to explain
> how this feature is working, you'll probably see that it's a bit tricky
> to explain.
>
> So, I'm definitely not saying "no" to this, but my belief is that it's
> on the edges of Buildroot features. On the one hand, it brings a
> well-defined and correct way of storing fragments and generated a
> defconfig from fragments, that everyone can use and rely on. On the
> other hand, it adds one more slightly "tricky" feature to Buildroot,
> increasing its learning curve.
>
> For a customer project, Gustavo has written a "genconfig" shell script,
> that takes as input various parameters that define the system to be
> built (which HW platform, which software stack, whether it should be a
> debug or release build) and produces the defconfig for Buildroot.

  I tend to agree with Thomas's reasoning. However, I think it would be good if 
there was a clearer canonical way of using buildroot. Gustavoz' genconfig script 
could be part of that. I, for example, have a local.mk fragment that autodetects 
packages you have checked out in $(BR2_EXTERNAL)/src and adds OVERRIDE_SRCDIR 
for them. I remember there was this template br2-external github repo at some 
point that you could use as a basis to start your own br2-external project (but 
I can't find it back).

  I do think it is good if such supporting functionality is carried in the core 
buildroot repo. But it's better if it's a bit out of the way, not in the main 
Makefile, and probably called through some helper script.

  I have a similar feeling about the genimage template script BTW.

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 1/1] Makefile: Support merged defconfigs

[Thomas Petazzoni]

Hello,

On Thu, 12 May 2016 23:58:55 +0200, Arnout Vandecappelle wrote:

>   I tend to agree with Thomas's reasoning. However, I think it would be good if 
> there was a clearer canonical way of using buildroot. Gustavoz' genconfig script 
> could be part of that. I, for example, have a local.mk fragment that autodetects 
> packages you have checked out in $(BR2_EXTERNAL)/src and adds OVERRIDE_SRCDIR 
> for them. I remember there was this template br2-external github repo at some 
> point that you could use as a basis to start your own br2-external project (but 
> I can't find it back).

You're probably referring to
https://github.com/Openwide-Ingenierie/buildroot-submodule.

>   I do think it is good if such supporting functionality is carried in the core 
> buildroot repo. But it's better if it's a bit out of the way, not in the main 
> Makefile, and probably called through some helper script.
> 
>   I have a similar feeling about the genimage template script BTW.

Should we create a section in the manual with some kind of "best
practices" or "going further" ideas?

Regarding Gustavo's 'genconfig' script, I am not sure it makes a lot of
sense to have something like that in the tree. All what it does is very
project-specific, and down the road, it simply calls the
merge_config.sh script, which is already in our tree.

So, for the specific situation of creating a defconfig by combining
fragments, isn't the problem more a documentation problem than anything
else?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com