[Buildroot] [PATCH 0/3 v3] core/pkg-kconfig: ensure we have necessary tools to run configurators

[Buildroot] [PATCH 0/3 v3] core/pkg-kconfig: ensure we have necessary tools to run configurators

[Yann E. MORIN]

Hello All!

Recent version of the linux kernel introduced major changes in the
kconfig infrastructure.

Starting with linux-4.16, the kconfig parser code is no longer shipped
with the linux source tree, and mut be generated from the lex/yacc
parser sources.

Similarly, starting with linux-4.18, the kconfig infra allows calling
back to the shell to run various commands, especially calling to the
compiler to test its features (and hide/show options accordingly).

So, we need to ensure the appropriate tools are available before we
attempt to run any of the configurators.

This three-fold series firs introduces the necessary chjanges in the
kconfig infrastructure, so that packages can specify dependencies to be
built and installed before the configurators are run.

Then, the linux package is fixed to take advantage of that forits
dependencies on host-flex and host-bison.

Finally, the linux package is also fixed to guarantee the toolchain is
available for the configurators.

Changes v2 -> v3:
  - drop append-assignment in patch 2

Changes v1 -> v2:
  - introduce the KCONFIG_DEPENDENCIES in the kconfig-package infra
    (Thomas)
  - change linux to take advantage of that  (Thomas)


Regards,
Yann E. MORIN.


The following changes since commit 81ea4a243b63d7bb1fec580910c553af4ae072c1

  package/lttng-tools: bump version to 2.10.5 (2018-08-01 14:28:30 +0200)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 5878b50c7b69951bc528d25c98a691a888ee5051

  linux: kconfig needs the toolchain (2018-08-03 22:02:18 +0200)


----------------------------------------------------------------
Yann E. MORIN (3):
      core/pkg-kconfig: allow dependencies before configurators
      linux: kconfig needs host-{flex,bison} to build the configurators
      linux: kconfig needs the toolchain

 docs/manual/adding-packages-kconfig.txt |  4 ++++
 linux/linux.mk                          | 11 ++++++++++-
 package/pkg-kconfig.mk                  |  9 +++++++++
 3 files changed, 23 insertions(+), 1 deletion(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/3 v3] core/pkg-kconfig: allow dependencies before configurators

[Yann E. MORIN]

Some users of kconfig need some packages to be built before their
kconfig infra be used.

For example, the linux kernel, starting with 4.16, needs flex and bison
to generate the parser code. Furthermore, starting with 4.18, it will
also need the cross-compiler before parsing the kconfig stuff, because
that calls the compiler to check its features.

Currently, this is broken, even the flex/bison ones, even though they
are listed, because there is no way to define dependencie that are
guaranteed before the (visual) configurators. For example:

    $ make distclean
    $ make menuconfig
      --> enable the linux kernel, choose a defconfig, save, exit
    $ make linux-menuconfig
    [...]
      HOSTCC  scripts/basic/fixdep
      HOSTCC  scripts/kconfig/conf.o
      YACC    scripts/kconfig/zconf.tab.c
    /bin/sh: bison: command not found
      LEX     scripts/kconfig/zconf.lex.c
    scripts/Makefile.lib:196: recipe for target 'scripts/kconfig/zconf.tab.c' failed
    make[3]: *** [scripts/kconfig/zconf.tab.c] Error 127
    make[3]: *** Waiting for unfinished jobs....
    /bin/sh: flex: command not found
    scripts/Makefile.lib:188: recipe for target 'scripts/kconfig/zconf.lex.c' failed
    make[3]: *** [scripts/kconfig/zconf.lex.c] Error 127
    Makefile:528: recipe for target 'rpc_defconfig' failed
    make[2]: *** [rpc_defconfig] Error 2
    linux/linux.mk:511: recipe for target '/home/ymorin/dev/buildroot/buildroot/output/build/linux-4.17.11/.config' failed
    make[1]: *** [/home/ymorin/dev/buildroot/buildroot/output/build/linux-4.17.11/.config] Error 2
    Makefile:79: recipe for target '_all' failed
    make: *** [_all] Error 2

So, we introduce a new type of dependencies for kconfig-based packages,
that are guaranteed to be built and installed before the (visual)
configurators are called.

Since those dependencies are phony targets and therefore always out of
date, a normal dependency would cause the .config target to be rebuilt
on each invocation of make. So we use an order-only pre-requisite, like
is done for the patch dependency.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 docs/manual/adding-packages-kconfig.txt | 4 ++++
 package/pkg-kconfig.mk                  | 9 +++++++++
 2 files changed, 13 insertions(+)

diff --git a/docs/manual/adding-packages-kconfig.txt b/docs/manual/adding-packages-kconfig.txt
index d4d8630a79..982c1229ea 100644
--- a/docs/manual/adding-packages-kconfig.txt
+++ b/docs/manual/adding-packages-kconfig.txt
@@ -81,3 +81,7 @@ be set to suit the needs of the package under consideration:
   be well suited for all packages that use the standard kconfig
   infrastructure as inherited from the Linux kernel; some packages use
   a derivative of kconfig that use a different location.
+
+* +FOO_KCONFIG_DEPENDENCIES+: the list of packages (most probably, host
+  packages) that need to be built before this package's kconfig is
+  interpreted. Seldom used. By default, empty.
diff --git a/package/pkg-kconfig.mk b/package/pkg-kconfig.mk
index 81bba5220c..d6c95b897e 100644
--- a/package/pkg-kconfig.mk
+++ b/package/pkg-kconfig.mk
@@ -41,6 +41,10 @@ endef
 
 define inner-kconfig-package
 
+# Register the kconfig dependencies as regular dependencies, so that
+# they are also accounted for in the generated graphs.
+$(2)_DEPENDENCIES += $$($(2)_KCONFIG_DEPENDENCIES)
+
 # Call the generic package infrastructure to generate the necessary
 # make targets.
 # Note: this must be done _before_ attempting to use $$($(2)_DIR) in a
@@ -121,6 +125,11 @@ $$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): $$($(2)_KCONFIG_FILE) $$($(2)_KCONFIG_F
 # it explicitly. It doesn't hurt to always have it though.
 $$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): | $(1)-patch
 
+# Some packages may need additional tools to be present by the time their
+# kconfig structure is parsed (e.g. the linux kernel may need to call to
+# the compiler to test its features).
+$$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): | $$($(2)_KCONFIG_DEPENDENCIES)
+
 # In order to get a usable, consistent configuration, some fixup may be needed.
 # The exact rules are specified by the package .mk file.
 define $(2)_FIXUP_DOT_CONFIG
-- 
2.14.1

[Buildroot] [PATCH 2/3 v3] linux: kconfig needs host-{flex, bison} to build the configurators

[Yann E. MORIN]

Take this opportunity to also drop the append-assignment as the first
assignment to LINUX_DEPENDENCIES.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

---
Changes v2 -> v3:
  - also fix the append-assignment
---
 linux/linux.mk | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/linux/linux.mk b/linux/linux.mk
index 7527b11673..60dad3a249 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -62,7 +62,11 @@ LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))
 LINUX_PATCH = $(filter ftp://% http://% https://%,$(LINUX_PATCHES))
 
 LINUX_INSTALL_IMAGES = YES
-LINUX_DEPENDENCIES += host-bison host-flex host-kmod
+LINUX_DEPENDENCIES = host-kmod
+
+# Starting with 4.16, the generated kconfig code is no longer
+# shipped with the kernel sources, so we need flex and bison.
+LINUX_KCONFIG_DEPENDENCIES = host-bison host-flex
 
 # host tools needed for kernel compression
 ifeq ($(BR2_LINUX_KERNEL_LZ4),y)
-- 
2.14.1

[Buildroot] [PATCH 3/3 v3] linux: kconfig needs the toolchain

[Yann E. MORIN]

Starting with linux-4.18, the kconfig from the kernel can call
to the compiler to test its capabilities; see:

    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/Kconfig.include

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---
Changes v1 -> v2:
  - take advantage of the kconfig-package infra  (Thomas)
---
 linux/linux.mk | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/linux/linux.mk b/linux/linux.mk
index 60dad3a249..48b2a16ce7 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -68,6 +68,11 @@ LINUX_DEPENDENCIES = host-kmod
 # shipped with the kernel sources, so we need flex and bison.
 LINUX_KCONFIG_DEPENDENCIES = host-bison host-flex
 
+# Starting with 4.18, the kconfig in the kernel calls the
+# cross-compiler to check its capabilities. So we need the
+# toolchain before we can call the configurators.
+LINUX_KCONFIG_DEPENDENCIES += toolchain
+
 # host tools needed for kernel compression
 ifeq ($(BR2_LINUX_KERNEL_LZ4),y)
 LINUX_DEPENDENCIES += host-lz4
-- 
2.14.1

[Buildroot] [PATCH 3/3 v3] linux: kconfig needs the toolchain

[Jan Kundrát]

On p?tek 3. srpna 2018 22:16:56 CEST, Yann E. MORIN wrote:
> Starting with linux-4.18, the kconfig from the kernel can call
> to the compiler to test its capabilities; see:
>
>     
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/Kconfig.include
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
>
> ---
> Changes v1 -> v2:
>   - take advantage of the kconfig-package infra  (Thomas)
> ---
>  linux/linux.mk | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/linux/linux.mk b/linux/linux.mk
> index 60dad3a249..48b2a16ce7 100644
> --- a/linux/linux.mk
> +++ b/linux/linux.mk
> @@ -68,6 +68,11 @@ LINUX_DEPENDENCIES = host-kmod
>  # shipped with the kernel sources, so we need flex and bison.
>  LINUX_KCONFIG_DEPENDENCIES = host-bison host-flex
>  
> +# Starting with 4.18, the kconfig in the kernel calls the
> +# cross-compiler to check its capabilities. So we need the
> +# toolchain before we can call the configurators.
> +LINUX_KCONFIG_DEPENDENCIES += toolchain
> +
>  # host tools needed for kernel compression
>  ifeq ($(BR2_LINUX_KERNEL_LZ4),y)
>  LINUX_DEPENDENCIES += host-lz4

Tested-by: Jan Kundr?t <jan.kundrat@cesnet.cz>
Reviewed-by: Jan Kundr?t <jan.kundrat@cesnet.cz>

[Buildroot] [PATCH 2/3 v3] linux: kconfig needs host-{flex, bison} to build the configurators

[Jan Kundrát]

On p?tek 3. srpna 2018 22:16:55 CEST, Yann E. MORIN wrote:
> Take this opportunity to also drop the append-assignment as the first
> assignment to LINUX_DEPENDENCIES.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>
> ---
> Changes v2 -> v3:
>   - also fix the append-assignment
> ---
>  linux/linux.mk | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/linux/linux.mk b/linux/linux.mk
> index 7527b11673..60dad3a249 100644
> --- a/linux/linux.mk
> +++ b/linux/linux.mk
> @@ -62,7 +62,11 @@ LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))
>  LINUX_PATCH = $(filter ftp://% http://% https://%,$(LINUX_PATCHES))
>  
>  LINUX_INSTALL_IMAGES = YES
> -LINUX_DEPENDENCIES += host-bison host-flex host-kmod
> +LINUX_DEPENDENCIES = host-kmod
> +
> +# Starting with 4.16, the generated kconfig code is no longer
> +# shipped with the kernel sources, so we need flex and bison.
> +LINUX_KCONFIG_DEPENDENCIES = host-bison host-flex
>  
>  # host tools needed for kernel compression
>  ifeq ($(BR2_LINUX_KERNEL_LZ4),y)

Tested-by: Jan Kundr?t <jan.kundrat@cesnet.cz>

[Buildroot] [PATCH 1/3 v3] core/pkg-kconfig: allow dependencies before configurators

[Jan Kundrát]

On p?tek 3. srpna 2018 22:16:54 CEST, Yann E. MORIN wrote:
> Some users of kconfig need some packages to be built before their
> kconfig infra be used.
>
> For example, the linux kernel, starting with 4.16, needs flex and bison
> to generate the parser code. Furthermore, starting with 4.18, it will
> also need the cross-compiler before parsing the kconfig stuff, because
> that calls the compiler to check its features.
>
> Currently, this is broken, even the flex/bison ones, even though they
> are listed, because there is no way to define dependencie that are
> guaranteed before the (visual) configurators. For example:
>
>     $ make distclean
>     $ make menuconfig
>       --> enable the linux kernel, choose a defconfig, save, exit
>     $ make linux-menuconfig
>     [...]
>       HOSTCC  scripts/basic/fixdep
>       HOSTCC  scripts/kconfig/conf.o
>       YACC    scripts/kconfig/zconf.tab.c
>     /bin/sh: bison: command not found
>       LEX     scripts/kconfig/zconf.lex.c
>     scripts/Makefile.lib:196: recipe for target 
> 'scripts/kconfig/zconf.tab.c' failed
>     make[3]: *** [scripts/kconfig/zconf.tab.c] Error 127
>     make[3]: *** Waiting for unfinished jobs....
>     /bin/sh: flex: command not found
>     scripts/Makefile.lib:188: recipe for target 
> 'scripts/kconfig/zconf.lex.c' failed
>     make[3]: *** [scripts/kconfig/zconf.lex.c] Error 127
>     Makefile:528: recipe for target 'rpc_defconfig' failed
>     make[2]: *** [rpc_defconfig] Error 2
>     linux/linux.mk:511: recipe for target 
> '/home/ymorin/dev/buildroot/buildroot/output/build/linux-4.17.11/.config' 
> failed
>     make[1]: *** 
> [/home/ymorin/dev/buildroot/buildroot/output/build/linux-4.17.11/.config] 
> Error 2
>     Makefile:79: recipe for target '_all' failed
>     make: *** [_all] Error 2
>
> So, we introduce a new type of dependencies for kconfig-based packages,
> that are guaranteed to be built and installed before the (visual)
> configurators are called.
>
> Since those dependencies are phony targets and therefore always out of
> date, a normal dependency would cause the .config target to be rebuilt
> on each invocation of make. So we use an order-only pre-requisite, like
> is done for the patch dependency.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  docs/manual/adding-packages-kconfig.txt | 4 ++++
>  package/pkg-kconfig.mk                  | 9 +++++++++
>  2 files changed, 13 insertions(+)
>
> diff --git a/docs/manual/adding-packages-kconfig.txt 
> b/docs/manual/adding-packages-kconfig.txt
> index d4d8630a79..982c1229ea 100644
> --- a/docs/manual/adding-packages-kconfig.txt
> +++ b/docs/manual/adding-packages-kconfig.txt
> @@ -81,3 +81,7 @@ be set to suit the needs of the package under 
> consideration:
>    be well suited for all packages that use the standard kconfig
>    infrastructure as inherited from the Linux kernel; some packages use
>    a derivative of kconfig that use a different location.
> +
> +* +FOO_KCONFIG_DEPENDENCIES+: the list of packages (most probably, host
> +  packages) that need to be built before this package's kconfig is
> +  interpreted. Seldom used. By default, empty.
> diff --git a/package/pkg-kconfig.mk b/package/pkg-kconfig.mk
> index 81bba5220c..d6c95b897e 100644
> --- a/package/pkg-kconfig.mk
> +++ b/package/pkg-kconfig.mk
> @@ -41,6 +41,10 @@ endef
>  
>  define inner-kconfig-package
>  
> +# Register the kconfig dependencies as regular dependencies, so that
> +# they are also accounted for in the generated graphs.
> +$(2)_DEPENDENCIES += $$($(2)_KCONFIG_DEPENDENCIES)
> +
>  # Call the generic package infrastructure to generate the necessary
>  # make targets.
>  # Note: this must be done _before_ attempting to use $$($(2)_DIR) in a
> @@ -121,6 +125,11 @@ $$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): 
> $$($(2)_KCONFIG_FILE) $$($(2)_KCONFIG_F
>  # it explicitly. It doesn't hurt to always have it though.
>  $$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): | $(1)-patch
>  
> +# Some packages may need additional tools to be present by the time their
> +# kconfig structure is parsed (e.g. the linux kernel may need to call to
> +# the compiler to test its features).
> +$$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): | $$($(2)_KCONFIG_DEPENDENCIES)
> +
>  # In order to get a usable, consistent configuration, some 
> fixup may be needed.
>  # The exact rules are specified by the package .mk file.
>  define $(2)_FIXUP_DOT_CONFIG

Tested-by: Jan Kundr?t <jan.kundrat@cesnet.cz>

[Buildroot] [PATCH 2/3 v3] linux: kconfig needs host-{flex, bison} to build the configurators

[Thomas Petazzoni]

Hello,

On Fri,  3 Aug 2018 22:16:55 +0200, Yann E. MORIN wrote:
> Take this opportunity to also drop the append-assignment as the first
> assignment to LINUX_DEPENDENCIES.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

I am a bit confused. In this series, you suggest to make
host-flex/host-bison dependencies of the configuration step of Linux.
But in another series
(http://patchwork.ozlabs.org/project/buildroot/list/?series=59260), you
suggest to make flex/bison hard requirements of Buildroot, so that we
don't have to build them.

Could you clarify in which direction you would like to go ? Also, if we
make bison/flex hard requirements of Buildroot, I'd like to have a
clear rule on whether we keep or not our host-bison/host-flex packages,
and if we keep them, when they should be used vs. when the system
bison/flex should be used.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 2/3 v3] linux: kconfig needs host-{flex, bison} to build the configurators

[Yann E. MORIN]

Thomas, All,

On 2018-08-14 16:21 +0200, Thomas Petazzoni spake thusly:
> On Fri,  3 Aug 2018 22:16:55 +0200, Yann E. MORIN wrote:
> > Take this opportunity to also drop the append-assignment as the first
> > assignment to LINUX_DEPENDENCIES.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
> > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> 
> I am a bit confused. In this series, you suggest to make
> host-flex/host-bison dependencies of the configuration step of Linux.
> But in another series
> (http://patchwork.ozlabs.org/project/buildroot/list/?series=59260), you
> suggest to make flex/bison hard requirements of Buildroot, so that we
> don't have to build them.
> 
> Could you clarify in which direction you would like to go ?

This very series is about solving a technical problem, that
kconfig-based packages may have dependencies required even before
running the configurators.

On the other hand, that other series is about changing our requirements.

If we decide to require flex+bison, so in this series, only patch 2 can
be dropped.

> Also, if we
> make bison/flex hard requirements of Buildroot, I'd like to have a
> clear rule on whether we keep or not our host-bison/host-flex packages,
> and if we keep them, when they should be used vs. when the system
> bison/flex should be used.

My position is that we will always want to have our host flex and bison
to build packages that install things in host/, in staging/ or in
target/.

For the linux kernel and other kconfig-based packages, we don't care
which flex/bison are used, because the resulting binaries are not
installed, unless those packages also generate code eventually installed
in host/, staging/ or target/

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/3 v3] linux: kconfig needs host-{flex, bison} to build the configurators

[Thomas Petazzoni]

Hello,

On Tue, 14 Aug 2018 17:27:43 +0200, Yann E. MORIN wrote:

> > Could you clarify in which direction you would like to go ?  
> 
> This very series is about solving a technical problem, that
> kconfig-based packages may have dependencies required even before
> running the configurators.
> 
> On the other hand, that other series is about changing our requirements.
> 
> If we decide to require flex+bison, so in this series, only patch 2 can
> be dropped.

OK.

> > Also, if we
> > make bison/flex hard requirements of Buildroot, I'd like to have a
> > clear rule on whether we keep or not our host-bison/host-flex packages,
> > and if we keep them, when they should be used vs. when the system
> > bison/flex should be used.  
> 
> My position is that we will always want to have our host flex and bison
> to build packages that install things in host/, in staging/ or in
> target/.
> 
> For the linux kernel and other kconfig-based packages, we don't care
> which flex/bison are used, because the resulting binaries are not
> installed, unless those packages also generate code eventually installed
> in host/, staging/ or target/

So the system-wide bison/flex would only be used for the kconfig
programs built by Buildroot itself (if we move to a newer kconfig
version that doesn't have the pre-generated files) or by
Linux/U-Boot/Busybox/Barebox. Any other use of flex/bison would use
host-flex and host-bison. That's your proposal ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 2/3 v3] linux: kconfig needs host-{flex, bison} to build the configurators

[Yann E. MORIN]

Thomas, All,

On 2018-08-14 21:53 +0200, Thomas Petazzoni spake thusly:
> On Tue, 14 Aug 2018 17:27:43 +0200, Yann E. MORIN wrote:
> > > Also, if we
> > > make bison/flex hard requirements of Buildroot, I'd like to have a
> > > clear rule on whether we keep or not our host-bison/host-flex packages,
> > > and if we keep them, when they should be used vs. when the system
> > > bison/flex should be used.  
> > 
> > My position is that we will always want to have our host flex and bison
> > to build packages that install things in host/, in staging/ or in
> > target/.
> > 
> > For the linux kernel and other kconfig-based packages, we don't care
> > which flex/bison are used, because the resulting binaries are not
> > installed, unless those packages also generate code eventually installed
> > in host/, staging/ or target/
> 
> So the system-wide bison/flex would only be used for the kconfig
> programs built by Buildroot itself (if we move to a newer kconfig
> version that doesn't have the pre-generated files) or by
> Linux/U-Boot/Busybox/Barebox. Any other use of flex/bison would use
> host-flex and host-bison. That's your proposal ?

Yes, that is my proposal.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/3 v3] linux: kconfig needs host-{flex, bison} to build the configurators

[Thomas Petazzoni]

Hello,

On Tue, 14 Aug 2018 23:03:49 +0200, Yann E. MORIN wrote:

> > So the system-wide bison/flex would only be used for the kconfig
> > programs built by Buildroot itself (if we move to a newer kconfig
> > version that doesn't have the pre-generated files) or by
> > Linux/U-Boot/Busybox/Barebox. Any other use of flex/bison would use
> > host-flex and host-bison. That's your proposal ?  
> 
> Yes, that is my proposal.

OK. Let's see what Arnout/Peter have to say about this. If they don't
express any opinion by Thursday, I'll go with your proposal which seems
reasonable to me.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 2/3 v3] linux: kconfig needs host-{flex, bison} to build the configurators

[Arnout Vandecappelle]

On 14-08-18 17:27, Yann E. MORIN wrote:
> Thomas, All,
> 
> On 2018-08-14 16:21 +0200, Thomas Petazzoni spake thusly:
>> On Fri,  3 Aug 2018 22:16:55 +0200, Yann E. MORIN wrote:
>>> Take this opportunity to also drop the append-assignment as the first
>>> assignment to LINUX_DEPENDENCIES.
>>>
>>> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>>> Cc: Arnout Vandecappelle <arnout@mind.be>
>>> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>>
>> I am a bit confused. In this series, you suggest to make
>> host-flex/host-bison dependencies of the configuration step of Linux.
>> But in another series
>> (http://patchwork.ozlabs.org/project/buildroot/list/?series=59260), you
>> suggest to make flex/bison hard requirements of Buildroot, so that we
>> don't have to build them.
>>
>> Could you clarify in which direction you would like to go ?
> 
> This very series is about solving a technical problem, that
> kconfig-based packages may have dependencies required even before
> running the configurators.
> 
> On the other hand, that other series is about changing our requirements.
> 
> If we decide to require flex+bison, so in this series, only patch 2 can
> be dropped.
> 
>> Also, if we
>> make bison/flex hard requirements of Buildroot, I'd like to have a
>> clear rule on whether we keep or not our host-bison/host-flex packages,
>> and if we keep them, when they should be used vs. when the system
>> bison/flex should be used.
> 
> My position is that we will always want to have our host flex and bison
> to build packages that install things in host/, in staging/ or in
> target/.

 My position is that we do indeed need our host flex and bison for everything
installed in target/ (which implies everything installed in staging/), but not
for things installed in host/. We definitely need it for BR2_REPRODUCIBLE.

 It would also be nice if dependencies.sh could do a version check and accept
the system flex/bison if it is the right version, like we do for cmake and tar.

 But both of these wishes make things more complicated, so I'm OK with just
keeping all current host-flex/bison dependencies.


> For the linux kernel and other kconfig-based packages, we don't care
> which flex/bison are used, because the resulting binaries are not
> installed, unless those packages also generate code eventually installed
> in host/, staging/ or target/

 Note that this means you'll still need host-flex/bison for the kernel, because
the kernel may build dtc and that gets installed in host...

 Regards,
 Arnout

> 
> Regards,
> Yann E. MORIN.
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 2/3 v3] linux: kconfig needs host-{flex, bison} to build the configurators

[Thomas Petazzoni]

Hello,

On Wed, 15 Aug 2018 00:50:42 +0200, Arnout Vandecappelle wrote:

>  My position is that we do indeed need our host flex and bison for everything
> installed in target/ (which implies everything installed in staging/), but not
> for things installed in host/. We definitely need it for BR2_REPRODUCIBLE.
> 
>  It would also be nice if dependencies.sh could do a version check and accept
> the system flex/bison if it is the right version, like we do for cmake and tar.
> 
>  But both of these wishes make things more complicated, so I'm OK with just
> keeping all current host-flex/bison dependencies.

In fact, I think adding flex and bison as mandatory dependencies of the
system is going to cause too much disruption. All the autobuilders
would start complaining, all the Travis/Gitlab CI setups would have to
be adjusted, and everyone building in containers/minimal environments
will be affected.

So instead, I'd like to detect if flex/bison are available on the
system. If they are available, then we use them for the Linux kernel
configuration process. If they are not available, we build them.

Instead of:

  LINUX_KCONFIG_DEPENDENCIES = host-bison host-flex

I'd prefer to see:

  LINUX_KCONFIG_DEPENDENCIES = $(BR2_FLEX_HOST_DEPENDENCY) $(BR2_BISON_HOST_DEPENDENCY)

And the usual logic in support/dependencies/ to check for bison/flex.
This will avoid disrupting all existing setups, and still provide the
speed-up that we don't need to build flex/bison when doing "make
linux-menuconfig".

> > For the linux kernel and other kconfig-based packages, we don't care
> > which flex/bison are used, because the resulting binaries are not
> > installed, unless those packages also generate code eventually installed
> > in host/, staging/ or target/  
> 
>  Note that this means you'll still need host-flex/bison for the kernel, because
> the kernel may build dtc and that gets installed in host...

Are you sure flex/bison are used when building the DTC copy in the
Linux kernel ? Indeed we only added host-bison/host-flex as
dependencies of linux in commit
1b9faedf32be26d9c983c573ccd98f57fc6f6569, when kconfig stopped shipping
its pre-generated files. How could DTC be built before
1b9faedf32be26d9c983c573ccd98f57fc6f6569 if it required bison/flex ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 2/3 v3] linux: kconfig needs host-{flex, bison} to build the configurators

[Yann E. MORIN]

Thomas, All,

On 2018-08-15 14:18 +0200, Thomas Petazzoni spake thusly:
> On Wed, 15 Aug 2018 00:50:42 +0200, Arnout Vandecappelle wrote:
> >  My position is that we do indeed need our host flex and bison for everything
> > installed in target/ (which implies everything installed in staging/), but not
> > for things installed in host/. We definitely need it for BR2_REPRODUCIBLE.
> > 
> >  It would also be nice if dependencies.sh could do a version check and accept
> > the system flex/bison if it is the right version, like we do for cmake and tar.
> > 
> >  But both of these wishes make things more complicated, so I'm OK with just
> > keeping all current host-flex/bison dependencies.
> 
> In fact, I think adding flex and bison as mandatory dependencies of the
> system is going to cause too much disruption. All the autobuilders
> would start complaining, all the Travis/Gitlab CI setups would have to
> be adjusted, and everyone building in containers/minimal environments
> will be affected.

Yes, they will be affected. If not today, then later, because at some
point we will also bump to require new versions of componentns. Oh,
wait, we already bumped to gcc-4.4 (even if that is mostly transparent).

So I don't think we should refrain from updating, or adding to, our
requirements just because some people will be affected...

> So instead, I'd like to detect if flex/bison are available on the
> system. If they are available, then we use them for the Linux kernel
> configuration process. If they are not available, we build them.
> 
> Instead of:
>   LINUX_KCONFIG_DEPENDENCIES = host-bison host-flex
> I'd prefer to see:
>   LINUX_KCONFIG_DEPENDENCIES = $(BR2_FLEX_HOST_DEPENDENCY) $(BR2_BISON_HOST_DEPENDENCY)

Yeah, why not. That is easy enough.

> And the usual logic in support/dependencies/ to check for bison/flex.
> This will avoid disrupting all existing setups, and still provide the
> speed-up that we don't need to build flex/bison when doing "make
> linux-menuconfig".

Except we will soon to have to build the toolchain to be able to run
linux-menuconfig anyway... So, the flex/bison build time will not be
noticeable, except for external toolchains.

> > > For the linux kernel and other kconfig-based packages, we don't care
> > > which flex/bison are used, because the resulting binaries are not
> > > installed, unless those packages also generate code eventually installed
> > > in host/, staging/ or target/  
> > 
> >  Note that this means you'll still need host-flex/bison for the kernel, because
> > the kernel may build dtc and that gets installed in host...

But they would not be a LINUX_KCONFIG_DEPENDENCIES, just a plain
LINUX_DEPENDENCIES.

Regards,
Yann E. MORIN.

> Are you sure flex/bison are used when building the DTC copy in the
> Linux kernel ? Indeed we only added host-bison/host-flex as
> dependencies of linux in commit
> 1b9faedf32be26d9c983c573ccd98f57fc6f6569, when kconfig stopped shipping
> its pre-generated files. How could DTC be built before
> 1b9faedf32be26d9c983c573ccd98f57fc6f6569 if it required bison/flex ?
> 
> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
> Embedded Linux and Kernel engineering
> https://bootlin.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/3 v3] linux: kconfig needs host-{flex, bison} to build the configurators

[Arnout Vandecappelle]

On 15/08/2018 14:18, Thomas Petazzoni wrote:
> Hello,
> 
> On Wed, 15 Aug 2018 00:50:42 +0200, Arnout Vandecappelle wrote:
[snip]
> I'd prefer to see:
> 
>   LINUX_KCONFIG_DEPENDENCIES = $(BR2_FLEX_HOST_DEPENDENCY) $(BR2_BISON_HOST_DEPENDENCY)
> 
> And the usual logic in support/dependencies/ to check for bison/flex.
> This will avoid disrupting all existing setups, and still provide the
> speed-up that we don't need to build flex/bison when doing "make
> linux-menuconfig".

 +1. It also makes it easy to add the feature "build flex is system version is
not OK" later.


>>> For the linux kernel and other kconfig-based packages, we don't care
>>> which flex/bison are used, because the resulting binaries are not
>>> installed, unless those packages also generate code eventually installed
>>> in host/, staging/ or target/  
>>
>>  Note that this means you'll still need host-flex/bison for the kernel, because
>> the kernel may build dtc and that gets installed in host...
> 
> Are you sure flex/bison are used when building the DTC copy in the
> Linux kernel ? Indeed we only added host-bison/host-flex as
> dependencies of linux in commit
> 1b9faedf32be26d9c983c573ccd98f57fc6f6569, when kconfig stopped shipping
> its pre-generated files. How could DTC be built before
> 1b9faedf32be26d9c983c573ccd98f57fc6f6569 if it required bison/flex ?

 The shipped files for DTC were removed in
e039139be8c25145b103ab365ff1bd4a540066a3 which is v4.17-rc1~106^2~8^2~2 (while
the kconfig stuff was removed in 29c833061c1d8c2d1d23a62e7061561eadd76cdb which
is v4.16-rc1~105^2~26). So we already had the dependencies when the shipped DTC
files were removed from the kernel. And as Yann said, they're not needed for
kconfig itself, only for building.

 Regards,
 Arnout

> 
> Best regards,
> 
> Thomas
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF