[Buildroot] utils/genrandconfig discards all pre-defined BR2_PACKAGE_ packages

[Buildroot] utils/genrandconfig discards all pre-defined BR2_PACKAGE_ packages

[Alexey Brodkin via buildroot]

Hello,

While playing with Buildroot autobuilder "nodes" I noticed one strange
behavior: if I add "BR2_PACKAGE_xxx=yyy" in my architecture's config fragment
(I mean those in "support/config-fragments/autobuild") most likely it won't
appear in the resulting ".config". Well, unless it gets configured back by the
random configuration selector. Which to me looks a bit incorrect.

First, why was I doing that: I wanted to fix a headers version with
"BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_x_y=y". And to me it was
not a package, but a property of the toolchain, so could be kinda legitimate
use of the fragment (though you may argue for sure).

Then there's another observation to make that case a bit stronger if you will ;)
We do append "support/config-fragments/minimal.config" to any given fragment
with rationale of "disabling Buildroot options that are turned on by default,
in order to arrive at minimal build time". And some of its entries are BR2_PACKAGE_xx:
 - # BR2_PACKAGE_BUSYBOX is not set
 - BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y

And those 2 will disappear in the end as well, while I guess we'd like them to be applied.

From what I may see it happens because of this part in the root Makefile:
---------------------------->8------------------------
randpackageconfig ... : $(BUILD_DIR)/buildroot-config/conf outputmakefile
	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
 "utils/genrandconfig" 
---------------------------->8------------------------
and "make randpackageconfig" is being called from "utils/genrandconfig".

The first thing I though of was removal of that filtering together with
the first ".config" expansion in the "utils/genrandconfig" with "make olddefconfig"
like that assuming that "olddefconfig" is being called in "randpackageconfig"
target anyway:
---------------------------->8------------------------
diff --git a/Makefile b/Makefile
index f9cf0f4cb3..1ac33d9da3 100644
--- a/Makefile
+++ b/Makefile
@@ -1004,7 +1004,7 @@ randconfig allyesconfig alldefconfig allnoconfig: $(BUILD_DIR)/buildroot-config/
        @$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null

 randpackageconfig allyespackageconfig allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
-       @grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
+       @cat $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
        @$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
                KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
                $< --$(subst package,,$@) $(CONFIG_CONFIG_IN)
diff --git a/utils/genrandconfig b/utils/genrandconfig
index 622cfd4891..5234a0726f 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -392,9 +392,6 @@ def gen_config(args):
     with open(configfile, "w+") as configf:
         configf.writelines(configlines)

-    subprocess.check_call(["make", "O=%s" % args.outputdir, "-C", args.buildrootdir,
-                           "olddefconfig"])
-
     if not is_toolchain_usable(configfile, toolchainconfig):
         return 2
---------------------------->8------------------------

But I'm not sure if that approach is correct in solving observed problem.

Any thoughts on this one?

-Alexey
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] utils/genrandconfig discards all pre-defined BR2_PACKAGE_ packages

[Arnout Vandecappelle]

On 16/11/2021 17:31, Alexey Brodkin wrote:
> Hello,
> 
> While playing with Buildroot autobuilder "nodes" I noticed one strange
> behavior: if I add "BR2_PACKAGE_xxx=yyy" in my architecture's config fragment
> (I mean those in "support/config-fragments/autobuild") most likely it won't
> appear in the resulting ".config". Well, unless it gets configured back by the
> random configuration selector. Which to me looks a bit incorrect.
> 
> First, why was I doing that: I wanted to fix a headers version with
> "BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_x_y=y". And to me it was
> not a package, but a property of the toolchain, so could be kinda legitimate
> use of the fragment (though you may argue for sure).
> 
> Then there's another observation to make that case a bit stronger if you will ;)
> We do append "support/config-fragments/minimal.config" to any given fragment
> with rationale of "disabling Buildroot options that are turned on by default,
> in order to arrive at minimal build time". And some of its entries are BR2_PACKAGE_xx:
>   - # BR2_PACKAGE_BUSYBOX is not set
>   - BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y
> 
> And those 2 will disappear in the end as well, while I guess we'd like them to be applied.

  I completely agree that we want them to be applied!


>  From what I may see it happens because of this part in the root Makefile:
> ---------------------------->8------------------------
> randpackageconfig ... : $(BUILD_DIR)/buildroot-config/conf outputmakefile
> 	@grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
>   "utils/genrandconfig"
> ---------------------------->8------------------------
> and "make randpackageconfig" is being called from "utils/genrandconfig".
> 
> The first thing I though of was removal of that filtering together with
> the first ".config" expansion in the "utils/genrandconfig" with "make olddefconfig"

  Well, that kind of defeats the purpose of randpackageconfig (and especially 
allnopackageconfig). The idea of those targets is to take an existing config and 
do the rand/allno/allyes on only the package part.

  A better solution is to change genrandconfig itself.
- Create the partial config in a different file, e.g. .config.in.
- Run randconfig instead of randpackageconfig.
- Set KCONFIG_ALLCONFIG={builddir}.config.in in the environment.

  I think the KCONFIG_ALLCONFIG is the only real difference between randconfig 
and randpackageconfig.


> like that assuming that "olddefconfig" is being called in "randpackageconfig"
> target anyway:
> ---------------------------->8------------------------
> diff --git a/Makefile b/Makefile
> index f9cf0f4cb3..1ac33d9da3 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1004,7 +1004,7 @@ randconfig allyesconfig alldefconfig allnoconfig: $(BUILD_DIR)/buildroot-config/
>          @$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
> 
>   randpackageconfig allyespackageconfig allnopackageconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> -       @grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
> +       @cat $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
>          @$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
>                  KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
>                  $< --$(subst package,,$@) $(CONFIG_CONFIG_IN)
> diff --git a/utils/genrandconfig b/utils/genrandconfig
> index 622cfd4891..5234a0726f 100755
> --- a/utils/genrandconfig
> +++ b/utils/genrandconfig
> @@ -392,9 +392,6 @@ def gen_config(args):
>       with open(configfile, "w+") as configf:
>           configf.writelines(configlines)
> 
> -    subprocess.check_call(["make", "O=%s" % args.outputdir, "-C", args.buildrootdir,
> -                           "olddefconfig"])

  This change can be done independently, right?

  Regards,
  Arnout

> -
>       if not is_toolchain_usable(configfile, toolchainconfig):
>           return 2
> ---------------------------->8------------------------
> 
> But I'm not sure if that approach is correct in solving observed problem.
> 
> Any thoughts on this one?
> 
> -Alexey
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] utils/genrandconfig discards all pre-defined BR2_PACKAGE_ packages

[Arnout Vandecappelle]

On 22/11/2021 20:13, Arnout Vandecappelle wrote:
>
>
> On 16/11/2021 17:31, Alexey Brodkin wrote:
>> Hello,
>>
>> While playing with Buildroot autobuilder "nodes" I noticed one strange
>> behavior: if I add "BR2_PACKAGE_xxx=yyy" in my architecture's config fragment
>> (I mean those in "support/config-fragments/autobuild") most likely it won't
>> appear in the resulting ".config". Well, unless it gets configured back by the
>> random configuration selector. Which to me looks a bit incorrect.
>>
>> First, why was I doing that: I wanted to fix a headers version with
>> "BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_x_y=y". And to me it was
>> not a package, but a property of the toolchain, so could be kinda legitimate
>> use of the fragment (though you may argue for sure).
>>
>> Then there's another observation to make that case a bit stronger if you will ;)
>> We do append "support/config-fragments/minimal.config" to any given fragment
>> with rationale of "disabling Buildroot options that are turned on by default,
>> in order to arrive at minimal build time". And some of its entries are 
>> BR2_PACKAGE_xx:
>>   - # BR2_PACKAGE_BUSYBOX is not set
>>   - BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y
>>
>> And those 2 will disappear in the end as well, while I guess we'd like them 
>> to be applied.
>
>  I completely agree that we want them to be applied!
>
>
>>  From what I may see it happens because of this part in the root Makefile:
>> ---------------------------->8------------------------
>> randpackageconfig ... : $(BUILD_DIR)/buildroot-config/conf outputmakefile
>>     @grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
>>   "utils/genrandconfig"
>> ---------------------------->8------------------------
>> and "make randpackageconfig" is being called from "utils/genrandconfig".
>>
>> The first thing I though of was removal of that filtering together with
>> the first ".config" expansion in the "utils/genrandconfig" with "make 
>> olddefconfig"
>
>  Well, that kind of defeats the purpose of randpackageconfig (and especially 
> allnopackageconfig). The idea of those targets is to take an existing config 
> and do the rand/allno/allyes on only the package part.
>
>  A better solution is to change genrandconfig itself.
> - Create the partial config in a different file, e.g. .config.in.
> - Run randconfig instead of randpackageconfig.
> - Set KCONFIG_ALLCONFIG={builddir}.config.in in the environment.
>
>  I think the KCONFIG_ALLCONFIG is the only real difference between randconfig 
> and randpackageconfig.

  Scratch that, there's an important additional difference: randpackageconfig 
also fixes all the other non-package options. In particular, the different 
choices in arch, toolchain and build options menu will be fully randomized 
unless they're specified explicitly in the config fragment. This is often not 
the case, e.g. for the floating point options for ARM, but also e.g. "Toolchain 
origin" for the external toolchains. Therefore, the config fragments also have 
to be updated with explicit choices for all the options (except some that we 
maybe *do* want to be randomized, like some of the build options).


>
>
>> like that assuming that "olddefconfig" is being called in "randpackageconfig"
>> target anyway:
>> ---------------------------->8------------------------
>> diff --git a/Makefile b/Makefile
>> index f9cf0f4cb3..1ac33d9da3 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -1004,7 +1004,7 @@ randconfig allyesconfig alldefconfig allnoconfig: 
>> $(BUILD_DIR)/buildroot-config/
>>          @$(COMMON_CONFIG_ENV) $< --olddefconfig $(CONFIG_CONFIG_IN) >/dev/null
>>
>>   randpackageconfig allyespackageconfig allnopackageconfig: 
>> $(BUILD_DIR)/buildroot-config/conf outputmakefile
>> -       @grep -v BR2_PACKAGE_ $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
>> +       @cat $(BR2_CONFIG) > $(CONFIG_DIR)/.config.nopkg
>>          @$(COMMON_CONFIG_ENV) SKIP_LEGACY=y \
>>                  KCONFIG_ALLCONFIG=$(CONFIG_DIR)/.config.nopkg \
>>                  $< --$(subst package,,$@) $(CONFIG_CONFIG_IN)
>> diff --git a/utils/genrandconfig b/utils/genrandconfig
>> index 622cfd4891..5234a0726f 100755
>> --- a/utils/genrandconfig
>> +++ b/utils/genrandconfig
>> @@ -392,9 +392,6 @@ def gen_config(args):
>>       with open(configfile, "w+") as configf:
>>           configf.writelines(configlines)
>>
>> -    subprocess.check_call(["make", "O=%s" % args.outputdir, "-C", 
>> args.buildrootdir,
>> -                           "olddefconfig"])
>
>  This change can be done independently, right?

  And I see now that this is not something that can be removed: it's exactly 
this command that makes sure the config fragments are extended to their full 
(default) values before passing it into KCONFIG_ALLCONFIG.


  Regards,
  Arnout


>
>  Regards,
>  Arnout
>
>> -
>>       if not is_toolchain_usable(configfile, toolchainconfig):
>>           return 2
>> ---------------------------->8------------------------
>>
>> But I'm not sure if that approach is correct in solving observed problem.
>>
>> Any thoughts on this one?
>>
>> -Alexey
>>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot