[Buildroot] Project configuration management

[Buildroot] Project configuration management

[Mateusz Słupny]

Hello all,

We are using buildroot for building a series of projects. What I feel is missing in buildroot is a way to store different configurations in scope of a single project. For example, we would like to prepare three types of builds, let's name them "Release", that is the basic build, "Develop", that is a Release build + dropbear + some other utilities, and "Extra", that contains all configuration options from Develop + some additional tools (gdb, valgrind, etc.). To achieve that, we have to maintain total of (number of projects) x (number of build types) different configuration files that are almost identical.

Can this goal be achieved without using multiple defconfig files that share approx 90% of their contents? What would you advise to avoid that?

First solution that comes to my mind is to allow nesting defconfig files with some include/source statement, but AFAICS that's not supported.

Please note that we are aware of "layered customizations" concept, and we're using it to some degree, but it doesn't solve all the issues, e.g. setting toolchain, selecting packages, and setting multiple paths to layers itself has to be done in defconfig files anyway.

Thanks and best regards,
-- 
Mateusz Slupny, lead developer, Vestiacom
E-mail: mateusz.slupny at vestiacom.com

[Buildroot] Project configuration management

[Thomas Petazzoni]

Hello Mateusz,

On Mon, 4 Apr 2016 15:55:44 +0200, Mateusz S?upny wrote:

> We are using buildroot for building a series of projects. What I feel
> is missing in buildroot is a way to store different configurations in
> scope of a single project. For example, we would like to prepare
> three types of builds, let's name them "Release", that is the basic
> build, "Develop", that is a Release build + dropbear + some other
> utilities, and "Extra", that contains all configuration options from
> Develop + some additional tools (gdb, valgrind, etc.). To achieve
> that, we have to maintain total of (number of projects) x (number of
> build types) different configuration files that are almost identical.
> 
> Can this goal be achieved without using multiple defconfig files that
> share approx 90% of their contents? What would you advise to avoid
> that?
> 
> First solution that comes to my mind is to allow nesting defconfig
> files with some include/source statement, but AFAICS that's not
> supported.
> 
> Please note that we are aware of "layered customizations" concept,
> and we're using it to some degree, but it doesn't solve all the
> issues, e.g. setting toolchain, selecting packages, and setting
> multiple paths to layers itself has to be done in defconfig files
> anyway.

The mechanism we typically advise in such situation is to use defconfig
fragments, and assemble them as needed to create the configuration you
feed into Buildroot. A shell script (or other) can help generating the
configuration fed into Buildroot from the fragments.

I'm Cc'ing Gustavo, who has been using this model for quite some time
for a large project that involves multiple HW platforms, I'm sure he
can give more insights on how to achieve that.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Project configuration management

[Grant Edwards]

On 2016-04-04, Mateusz S?upny <mateusz.slupny@vestiacom.com> wrote:

> We are using buildroot for building a series of projects. What I
> feel is missing in buildroot is a way to store different
> configurations in scope of a single project. For example, we would
> like to prepare three types of builds, let's name them "Release",
> that is the basic build, "Develop", that is a Release build +
> dropbear + some other utilities, and "Extra", that contains all
> configuration options from Develop + some additional tools (gdb,
> valgrind, etc.). To achieve that, we have to maintain total of
> (number of projects) x (number of build types) different
> configuration files that are almost identical.

To address this issue, I use a single default config file for the
project, and then a build shell script that makes modifications to the
.config file for each "build type":

# bash functions used to manipulate linux kernel-style .config
# files.

function SyncConfig()
{
    yes '' | $Make oldconfig >/dev/null
}

function UnsetValue()
{
    dosync=y
    test "$1" = '-n' && { dosync=n; shift; }
    Variable="$1"
    echo "UnsetValue $Variable"
    sed -i "s/^${Variable}=.*/# ${Variable} is not set/g" .config
    test $dosync = y && SyncConfig
    return 0
}

function SetValue()
{
    dosync=y
    test "$1" = '-n' && { dosync=n; shift; }
    Variable="$1"
    # default value is 'y'
    Value="${2-y}"
    echo "SetValue $Variable $Value"
    # if value isn't 'y', then put it in double quotes
    test "$Value" != y && Value="\"$Value\""
    # escape any slashes
    Value=${Value//\//\\\/}
    sed -i "s/^${Variable}=.*/${Variable}=${Value}/g" .config
    sed -i "s/^# ${Variable} is not set.*/${Variable}=${Value}/g" .config
    test $dosync = y && SyncConfig
    return 0
}


-- 
Grant Edwards               grant.b.edwards        Yow! It's a lot of fun
                                  at               being alive ... I wonder if
                              gmail.com            my bed is made?!?

[Buildroot] Project configuration management

[Arnout Vandecappelle]

On 04/04/16 20:44, Thomas Petazzoni wrote:
> Hello Mateusz,
>
> On Mon, 4 Apr 2016 15:55:44 +0200, Mateusz S?upny wrote:
>
>> We are using buildroot for building a series of projects. What I feel
>> is missing in buildroot is a way to store different configurations in
>> scope of a single project. For example, we would like to prepare
>> three types of builds, let's name them "Release", that is the basic
>> build, "Develop", that is a Release build + dropbear + some other
>> utilities, and "Extra", that contains all configuration options from
>> Develop + some additional tools (gdb, valgrind, etc.). To achieve
>> that, we have to maintain total of (number of projects) x (number of
>> build types) different configuration files that are almost identical.
>>
>> Can this goal be achieved without using multiple defconfig files that
>> share approx 90% of their contents? What would you advise to avoid
>> that?
>>
>> First solution that comes to my mind is to allow nesting defconfig
>> files with some include/source statement, but AFAICS that's not
>> supported.
>>
>> Please note that we are aware of "layered customizations" concept,
>> and we're using it to some degree, but it doesn't solve all the
>> issues, e.g. setting toolchain, selecting packages, and setting
>> multiple paths to layers itself has to be done in defconfig files
>> anyway.
>
> The mechanism we typically advise in such situation is to use defconfig
> fragments, and assemble them as needed to create the configuration you
> feed into Buildroot. A shell script (or other) can help generating the
> configuration fed into Buildroot from the fragments.

  And that shell script exists already: support/kconfig/merge_config.sh

  The main difficulty with this model is that there is no simple way to split 
configs. Fortunately, for buildroot this is typically not much of a problem, 
because your fragments will just add some packages.


  Regards,
  Arnout


>
> I'm Cc'ing Gustavo, who has been using this model for quite some time
> for a large project that involves multiple HW platforms, I'm sure he
> can give more insights on how to achieve that.
>
> Best regards,
>
> Thomas
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Project configuration management

[Mateusz Słupny]

We'll go with merge_config.sh script as it's already a part of buildroot and we should be able to integrate it easily.

Thank you very much for all the suggestions!

best regards,
-- 
Mateusz Slupny, lead developer, Vestiacom
E-mail: mateusz.slupny at vestiacom.com

> On 04 Apr 2016, at 22:45, Arnout Vandecappelle <arnout@mind.be> wrote:
> 
> On 04/04/16 20:44, Thomas Petazzoni wrote:
>> Hello Mateusz,
>> 
>> On Mon, 4 Apr 2016 15:55:44 +0200, Mateusz S?upny wrote:
>> 
>>> We are using buildroot for building a series of projects. What I feel
>>> is missing in buildroot is a way to store different configurations in
>>> scope of a single project. For example, we would like to prepare
>>> three types of builds, let's name them "Release", that is the basic
>>> build, "Develop", that is a Release build + dropbear + some other
>>> utilities, and "Extra", that contains all configuration options from
>>> Develop + some additional tools (gdb, valgrind, etc.). To achieve
>>> that, we have to maintain total of (number of projects) x (number of
>>> build types) different configuration files that are almost identical.
>>> 
>>> Can this goal be achieved without using multiple defconfig files that
>>> share approx 90% of their contents? What would you advise to avoid
>>> that?
>>> 
>>> First solution that comes to my mind is to allow nesting defconfig
>>> files with some include/source statement, but AFAICS that's not
>>> supported.
>>> 
>>> Please note that we are aware of "layered customizations" concept,
>>> and we're using it to some degree, but it doesn't solve all the
>>> issues, e.g. setting toolchain, selecting packages, and setting
>>> multiple paths to layers itself has to be done in defconfig files
>>> anyway.
>> 
>> The mechanism we typically advise in such situation is to use defconfig
>> fragments, and assemble them as needed to create the configuration you
>> feed into Buildroot. A shell script (or other) can help generating the
>> configuration fed into Buildroot from the fragments.
> 
> And that shell script exists already: support/kconfig/merge_config.sh
> 
> The main difficulty with this model is that there is no simple way to split configs. Fortunately, for buildroot this is typically not much of a problem, because your fragments will just add some packages.
> 
> 
> Regards,
> Arnout
> 
> 
>> 
>> I'm Cc'ing Gustavo, who has been using this model for quite some time
>> for a large project that involves multiple HW platforms, I'm sure he
>> can give more insights on how to achieve that.
>> 
>> Best regards,
>> 
>> Thomas
>> 
> 
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Project configuration management

[Thomas Petazzoni]

Hello,

On Mon, 4 Apr 2016 22:45:45 +0200, Arnout Vandecappelle wrote:

> > The mechanism we typically advise in such situation is to use defconfig
> > fragments, and assemble them as needed to create the configuration you
> > feed into Buildroot. A shell script (or other) can help generating the
> > configuration fed into Buildroot from the fragments.
> 
>   And that shell script exists already: support/kconfig/merge_config.sh

That's not exactly the sort of script I was thinking of. I was thinking
of a more high-level, project specific script, like maybe:

	genconfig -h <hwplatform> -s <softwarestack> -r <releasetype>

or anything like that, which would internally have the knowledge of
which fragments to use for which aspect (HW platform, software stack,
etc.). Of course, internally, this script can use merge_config.sh, but
having a more high-level script might help users to more easily
generate their Buildroot configuration.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Project configuration management

[Gustavo Zacarias]

On 06/04/16 11:52, Thomas Petazzoni wrote:

> That's not exactly the sort of script I was thinking of. I was thinking
> of a more high-level, project specific script, like maybe:
>
> 	genconfig -h <hwplatform> -s <softwarestack> -r <releasetype>
>
> or anything like that, which would internally have the knowledge of
> which fragments to use for which aspect (HW platform, software stack,
> etc.). Of course, internally, this script can use merge_config.sh, but
> having a more high-level script might help users to more easily
> generate their Buildroot configuration.
>
> Thomas

Hi.
Adding a bit on this since you CCed me, this customer has multiple 
platforms (boards/architectures), with multiple capabilities (different 
video in/out frontends) and localized features.
What i've used is pretty much what Thomas says, a script that takes 
several parameters as input:

genconfig -p platform -b build_type -c capabilities -l localization ...

Where platform would be different hardware targets, build_type is what 
kind of rootfs/image/firmware it's building (debug, release, etc), which 
capabilities (isdb-t, dvb-s2, etc), and which localization (target 
country, for language, menus, logos, etc).

The script then combines different defconfig fragments according to 
these parameters and validates the different combinations (for instance 
some platform X can't have isdb-t or so on), generating a suitable 
defconfig and setting up an out-of-tree build ready to go.

Of course the "chunkization" of the defconfig fragments requires some 
knowledge, but it essentially boils down to:

* Hardware platform
   + Toolchain setup (varies for debug & release)
   + Target filesystem setup (might be different for debug & release)
   + Kernel setup (ditto above)

* Apps
   + Open-source apps for the target (might vary for debug & release)
   + Propietary apps (varies according to capabilities and localization)

* Localization
   + Language packs, certificates, default settings, marketing stuff, etc

Regards.