[Buildroot] [RFC next 1/2] linux: fix builds for kernels < 5.6 and host-gcc >= 10

[Buildroot] [RFC next 1/2] linux: fix builds for kernels < 5.6 and host-gcc >= 10

[Julien Olivain]

During a linux-backports update, it was found that kernel v3.10.108 was
failing to compile. See:
https://lists.buildroot.org/pipermail/buildroot/2022-August/649507.html

This issue was introduced by commit 9b41b54be07711c10ad13ce157be272ed1cf402e
"linux: fix build with host-gcc 10+", which removes all declarations of
"yylloc" symbols in the dtc parser. This symbol is generated by bison, if the
"%locations" directive is provided in the parser. See:
https://git.savannah.gnu.org/cgit/bison.git/tree/doc/bison.texi?h=v3.8.2#n5984

Kernel versions < 5.6 did not include this directive, so removing all
yylloc declararions in the parser also was failing for those version.

In the kernel, dtc was updated to v1.5.1-22-gc40aeb60b47a in:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=0cec114e36606412908a35695a5db944cec2e3db
This commit is included in kernel v5.6.

This dtc update include the dtc commit:
https://git.kernel.org/pub/scm/utils/dtc/dtc.git/commit/?id=7150286225476345bd6e7312331e3baf4d621c32
which adds the '%locations' directive.

This commit fixes the issue by programmatically adding the '%locations'
Bison directive, if it's not found in the parser file.

Fixes:
- https://bugs.busybox.net/show_bug.cgi?id=14971

Signed-off-by: Julien Olivain <ju.o@free.fr>
---
Tested by:
- Building a recent kernel:
  qemu_aarch64_virt_defconfig (Kernel v5.15.18)
- Building an old kernel:
  qemu_arm_vexpress_defconfig with kernel 3.10.108
---
 linux/linux.mk | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/linux/linux.mk b/linux/linux.mk
index 3d9ac37959..f3eb9c32cb 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -155,6 +155,7 @@ LINUX_MAKE_FLAGS = \
 	INSTALL_MOD_PATH=$(TARGET_DIR) \
 	CROSS_COMPILE="$(TARGET_CROSS)" \
 	WERROR=0 \
+	REGENERATE_PARSERS=1 \
 	DEPMOD=$(HOST_DIR)/sbin/depmod
 
 ifeq ($(BR2_REPRODUCIBLE),y)
@@ -286,6 +287,19 @@ define LINUX_DROP_YYLLOC
 endef
 LINUX_POST_PATCH_HOOKS += LINUX_DROP_YYLLOC
 
+# Kernel version < 5.6 breaks if host-gcc version is >= 10 and
+# 'yylloc' symbol is removed in previous hook, due to missing
+# '%locations' bison directive in dtc-parser.y.  See:
+# https://bugs.busybox.net/show_bug.cgi?id=14971
+define LINUX_ADD_DTC_LOCATIONS
+	$(Q)DTC_PARSER=$(@D)/scripts/dtc/dtc-parser.y; \
+	if test -e "$${DTC_PARSER}" \
+		&& ! grep -Eq '^%locations$$' "$${DTC_PARSER}" ; then \
+		$(SED) '/^%{$$/i %locations' "$${DTC_PARSER}"; \
+	fi
+endef
+LINUX_POST_PATCH_HOOKS += LINUX_ADD_DTC_LOCATIONS
+
 # Older linux kernels use deprecated perl constructs in timeconst.pl
 # that were removed for perl 5.22+ so it breaks on newer distributions
 # Try a dry-run patch to see if this applies, if it does go ahead
-- 
2.37.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [RFC next 2/2] support/testing/tests/core/test_kernel3_10: new test

[Julien Olivain]

This commit add a test to check that a relatively old 3.10 kernel are
building successfully in Buildroot. This test was written for
investigating the bug described in:
- https://bugs.busybox.net/show_bug.cgi?id=14971

Signed-off-by: Julien Olivain <ju.o@free.fr>
---
Tested with:

    python3 -m flake8 \
        support/testing/tests/core/test_kernel3_10.py
    [no-output]

    support/testing/run-tests \
        -d dl \
        -o output_folder tests.core.test_kernel3_10.TestOldKernel3_10
    ...
    OK
---
 DEVELOPERS                                    |  1 +
 support/testing/tests/core/test_kernel3_10.py | 43 +++++++++++++++++++
 2 files changed, 44 insertions(+)
 create mode 100644 support/testing/tests/core/test_kernel3_10.py

diff --git a/DEVELOPERS b/DEVELOPERS
index 4287fd9cf7..5cf63e919e 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -1680,6 +1680,7 @@ F:	package/python-gnupg/
 F:	package/python-pyalsa/
 F:	package/riscv-isa-sim/
 F:	package/zynaddsubfx/
+F:	support/testing/tests/core/test_kernel3_10.py
 F:	support/testing/tests/package/sample_python_distro.py
 F:	support/testing/tests/package/sample_python_gnupg.py
 F:	support/testing/tests/package/sample_python_pyalsa.py
diff --git a/support/testing/tests/core/test_kernel3_10.py b/support/testing/tests/core/test_kernel3_10.py
new file mode 100644
index 0000000000..5ff5ed1cad
--- /dev/null
+++ b/support/testing/tests/core/test_kernel3_10.py
@@ -0,0 +1,43 @@
+import os
+
+import infra.basetest
+
+
+# For testing the compilation of an old 3.10 kernel, we use a armv7
+# gcc5 bootlin toolchain, which also includes 3.10 kernel headers.
+class TestOldKernel3_10(infra.basetest.BRTest):
+    config = \
+        """
+        BR2_arm=y
+        BR2_cortex_a9=y
+        BR2_ARM_EABIHF=y
+        BR2_ARM_ENABLE_NEON=y
+        BR2_ARM_ENABLE_VFP=y
+        BR2_LINUX_KERNEL=y
+        BR2_LINUX_KERNEL_CUSTOM_VERSION=y
+        BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="3.10.108"
+        BR2_LINUX_KERNEL_DEFCONFIG="vexpress"
+        BR2_LINUX_KERNEL_DTS_SUPPORT=y
+        BR2_LINUX_KERNEL_INTREE_DTS_NAME="vexpress-v2p-ca9"
+        BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
+        BR2_TARGET_ROOTFS_CPIO=y
+        BR2_TARGET_ROOTFS_CPIO_GZIP=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
+        BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
+        BR2_TOOLCHAIN_EXTERNAL_CXX=y
+        BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
+        BR2_TOOLCHAIN_EXTERNAL_GCC_5=y
+        BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_10=y
+        BR2_TOOLCHAIN_EXTERNAL_URL="https://toolchains.bootlin.com/downloads/releases/toolchains/armv7-eabihf/tarballs/armv7-eabihf--glibc--stable-2017.05-toolchains-1-1.tar.bz2"
+        """
+
+    def test_run(self):
+        initrd = os.path.join(self.builddir, "images", "rootfs.cpio.gz")
+        kern = os.path.join(self.builddir, "images", "zImage")
+        dtb = os.path.join(self.builddir, "images", "vexpress-v2p-ca9.dtb")
+        self.emulator.boot(arch="armv7", kernel=kern,
+                           kernel_cmdline=["console=ttyAMA0"],
+                           options=["-M", "vexpress-a9", "-dtb", dtb, "-initrd", initrd])
+        self.emulator.login()
+        self.assertRunOk("true")
-- 
2.37.2

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC next 1/2] linux: fix builds for kernels < 5.6 and host-gcc >= 10

[Yann E. MORIN]

Julien, All,

On 2022-08-23 00:14 +0200, Julien Olivain spake thusly:
> During a linux-backports update, it was found that kernel v3.10.108 was
> failing to compile. See:
> https://lists.buildroot.org/pipermail/buildroot/2022-August/649507.html
> 
> This issue was introduced by commit 9b41b54be07711c10ad13ce157be272ed1cf402e
> "linux: fix build with host-gcc 10+", which removes all declarations of
> "yylloc" symbols in the dtc parser. This symbol is generated by bison, if the
> "%locations" directive is provided in the parser. See:
> https://git.savannah.gnu.org/cgit/bison.git/tree/doc/bison.texi?h=v3.8.2#n5984
> 
> Kernel versions < 5.6 did not include this directive, so removing all
> yylloc declararions in the parser also was failing for those version.
> 
> In the kernel, dtc was updated to v1.5.1-22-gc40aeb60b47a in:
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=0cec114e36606412908a35695a5db944cec2e3db
> This commit is included in kernel v5.6.
> 
> This dtc update include the dtc commit:
> https://git.kernel.org/pub/scm/utils/dtc/dtc.git/commit/?id=7150286225476345bd6e7312331e3baf4d621c32
> which adds the '%locations' directive.
> 
> This commit fixes the issue by programmatically adding the '%locations'
> Bison directive, if it's not found in the parser file.
> 
> Fixes:
> - https://bugs.busybox.net/show_bug.cgi?id=14971
> 
> Signed-off-by: Julien Olivain <ju.o@free.fr>

I eventually took some time to look into that issue, and I can't think
of a better solution than what you proposed.

Applied to master, thanks.

Regards,
Yann E. MORIN.

> ---
> Tested by:
> - Building a recent kernel:
>   qemu_aarch64_virt_defconfig (Kernel v5.15.18)
> - Building an old kernel:
>   qemu_arm_vexpress_defconfig with kernel 3.10.108
> ---
>  linux/linux.mk | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/linux/linux.mk b/linux/linux.mk
> index 3d9ac37959..f3eb9c32cb 100644
> --- a/linux/linux.mk
> +++ b/linux/linux.mk
> @@ -155,6 +155,7 @@ LINUX_MAKE_FLAGS = \
>  	INSTALL_MOD_PATH=$(TARGET_DIR) \
>  	CROSS_COMPILE="$(TARGET_CROSS)" \
>  	WERROR=0 \
> +	REGENERATE_PARSERS=1 \
>  	DEPMOD=$(HOST_DIR)/sbin/depmod
>  
>  ifeq ($(BR2_REPRODUCIBLE),y)
> @@ -286,6 +287,19 @@ define LINUX_DROP_YYLLOC
>  endef
>  LINUX_POST_PATCH_HOOKS += LINUX_DROP_YYLLOC
>  
> +# Kernel version < 5.6 breaks if host-gcc version is >= 10 and
> +# 'yylloc' symbol is removed in previous hook, due to missing
> +# '%locations' bison directive in dtc-parser.y.  See:
> +# https://bugs.busybox.net/show_bug.cgi?id=14971
> +define LINUX_ADD_DTC_LOCATIONS
> +	$(Q)DTC_PARSER=$(@D)/scripts/dtc/dtc-parser.y; \
> +	if test -e "$${DTC_PARSER}" \
> +		&& ! grep -Eq '^%locations$$' "$${DTC_PARSER}" ; then \
> +		$(SED) '/^%{$$/i %locations' "$${DTC_PARSER}"; \
> +	fi
> +endef
> +LINUX_POST_PATCH_HOOKS += LINUX_ADD_DTC_LOCATIONS
> +
>  # Older linux kernels use deprecated perl constructs in timeconst.pl
>  # that were removed for perl 5.22+ so it breaks on newer distributions
>  # Try a dry-run patch to see if this applies, if it does go ahead
> -- 
> 2.37.2
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC next 2/2] support/testing/tests/core/test_kernel3_10: new test

[Yann E. MORIN]

Julien, All,

On 2022-08-23 00:14 +0200, Julien Olivain spake thusly:
> This commit add a test to check that a relatively old 3.10 kernel are
> building successfully in Buildroot. This test was written for
> investigating the bug described in:
> - https://bugs.busybox.net/show_bug.cgi?id=14971
> 
> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
[--SNIP--]
> diff --git a/support/testing/tests/core/test_kernel3_10.py b/support/testing/tests/core/test_kernel3_10.py
> new file mode 100644
> index 0000000000..5ff5ed1cad
> --- /dev/null
> +++ b/support/testing/tests/core/test_kernel3_10.py
> @@ -0,0 +1,43 @@
> +import os
> +
> +import infra.basetest
> +
> +
> +# For testing the compilation of an old 3.10 kernel, we use a armv7
> +# gcc5 bootlin toolchain, which also includes 3.10 kernel headers.

I'm not sure we want such a test. If we wanted to test that we can build
an older kernel, then we should use a state-of-the-art toolchain, i.e.
at least the default settings in Buildroot, not an older toolchain.

However, that would validate that he kernel itself can be built with a
recent compiler, not that the kernel build infra can be built; indeed,
the infra uses the host compiler, not the target compiler, but we have
no way to test with a latest-ish gcc:

  * our reference image is still using an old system especially because
    we also want to ensure that we can still use Buildroot on such old
    systems;

  * we can't know what users have on their machine if they run the
    runtime test locally.

So, I am not sure this test is entirely meaningful.

Regards,
Yann E. MORIN.

> +class TestOldKernel3_10(infra.basetest.BRTest):
> +    config = \
> +        """
> +        BR2_arm=y
> +        BR2_cortex_a9=y
> +        BR2_ARM_EABIHF=y
> +        BR2_ARM_ENABLE_NEON=y
> +        BR2_ARM_ENABLE_VFP=y
> +        BR2_LINUX_KERNEL=y
> +        BR2_LINUX_KERNEL_CUSTOM_VERSION=y
> +        BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="3.10.108"
> +        BR2_LINUX_KERNEL_DEFCONFIG="vexpress"
> +        BR2_LINUX_KERNEL_DTS_SUPPORT=y
> +        BR2_LINUX_KERNEL_INTREE_DTS_NAME="vexpress-v2p-ca9"
> +        BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
> +        BR2_TARGET_ROOTFS_CPIO=y
> +        BR2_TARGET_ROOTFS_CPIO_GZIP=y
> +        BR2_TOOLCHAIN_EXTERNAL=y
> +        BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
> +        BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
> +        BR2_TOOLCHAIN_EXTERNAL_CXX=y
> +        BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
> +        BR2_TOOLCHAIN_EXTERNAL_GCC_5=y
> +        BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_10=y
> +        BR2_TOOLCHAIN_EXTERNAL_URL="https://toolchains.bootlin.com/downloads/releases/toolchains/armv7-eabihf/tarballs/armv7-eabihf--glibc--stable-2017.05-toolchains-1-1.tar.bz2"
> +        """
> +
> +    def test_run(self):
> +        initrd = os.path.join(self.builddir, "images", "rootfs.cpio.gz")
> +        kern = os.path.join(self.builddir, "images", "zImage")
> +        dtb = os.path.join(self.builddir, "images", "vexpress-v2p-ca9.dtb")
> +        self.emulator.boot(arch="armv7", kernel=kern,
> +                           kernel_cmdline=["console=ttyAMA0"],
> +                           options=["-M", "vexpress-a9", "-dtb", dtb, "-initrd", initrd])
> +        self.emulator.login()
> +        self.assertRunOk("true")
> -- 
> 2.37.2
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [RFC next 1/2] linux: fix builds for kernels < 5.6 and host-gcc >= 10

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Julien, All,
 > On 2022-08-23 00:14 +0200, Julien Olivain spake thusly:
 >> During a linux-backports update, it was found that kernel v3.10.108 was
 >> failing to compile. See:
 >> https://lists.buildroot.org/pipermail/buildroot/2022-August/649507.html
 >> 
 >> This issue was introduced by commit 9b41b54be07711c10ad13ce157be272ed1cf402e
 >> "linux: fix build with host-gcc 10+", which removes all declarations of
 >> "yylloc" symbols in the dtc parser. This symbol is generated by bison, if the
 >> "%locations" directive is provided in the parser. See:
 >> https://git.savannah.gnu.org/cgit/bison.git/tree/doc/bison.texi?h=v3.8.2#n5984
 >> 
 >> Kernel versions < 5.6 did not include this directive, so removing all
 >> yylloc declararions in the parser also was failing for those version.
 >> 
 >> In the kernel, dtc was updated to v1.5.1-22-gc40aeb60b47a in:
 >> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=0cec114e36606412908a35695a5db944cec2e3db
 >> This commit is included in kernel v5.6.
 >> 
 >> This dtc update include the dtc commit:
 >> https://git.kernel.org/pub/scm/utils/dtc/dtc.git/commit/?id=7150286225476345bd6e7312331e3baf4d621c32
 >> which adds the '%locations' directive.
 >> 
 >> This commit fixes the issue by programmatically adding the '%locations'
 >> Bison directive, if it's not found in the parser file.
 >> 
 >> Fixes:
 >> - https://bugs.busybox.net/show_bug.cgi?id=14971
 >> 
 >> Signed-off-by: Julien Olivain <ju.o@free.fr>

 > I eventually took some time to look into that issue, and I can't think
 > of a better solution than what you proposed.

 > Applied to master, thanks.

Committed to 2022.11.x and 2022.02.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot