[Buildroot] [PATCH 0/3 v2] package/kmod: add support for compressed modules (branch yem/host-kmod-z)

[Buildroot] [PATCH 0/3 v2] package/kmod: add support for compressed modules (branch yem/host-kmod-z)

[Yann E. MORIN]

Hello All!

This series adds conditional compression support to the host-kmod,
so that we correctly handle compressed kernel modules.


Regards,
Yann E. MORIN.


The following changes since commit de70d9b933c23b88b7b6f07561a62fd813274e9c

  package/ne10: drop arch-related comment (2020-05-03 10:07:05 +0200)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to d66d07fe82ae1c515eb9a010f1df2cbcc10f1a87

  package/lmod: add compression support for the host variant (2020-05-03 15:06:48 +0200)


----------------------------------------------------------------
Yann E. MORIN (3):
      package/kmod: explicitly disable compression when not enabled
      package/kmod: add option for host variant
      package/lmod: add compression support for the host variant

 linux/Config.in             |  1 +
 package/Config.in.host      |  1 +
 package/kmod/Config.in.host | 16 ++++++++++++++++
 package/kmod/kmod.mk        | 18 ++++++++++++++++++
 4 files changed, 36 insertions(+)
 create mode 100644 package/kmod/Config.in.host

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/3 v2] package/kmod: explicitly disable compression when not enabled

[Yann E. MORIN]

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
 package/kmod/kmod.mk | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/package/kmod/kmod.mk b/package/kmod/kmod.mk
index e2dfea5c7b..cdbfc5a469 100644
--- a/package/kmod/kmod.mk
+++ b/package/kmod/kmod.mk
@@ -35,11 +35,15 @@ endif
 ifeq ($(BR2_PACKAGE_ZLIB),y)
 KMOD_DEPENDENCIES += zlib
 KMOD_CONF_OPTS += --with-zlib
+else
+KMOD_CONF_OPTS += --without-zlib
 endif
 
 ifeq ($(BR2_PACKAGE_XZ),y)
 KMOD_DEPENDENCIES += xz
 KMOD_CONF_OPTS += --with-xz
+else
+KMOD_CONF_OPTS += --without-xz
 endif
 
 ifeq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),y)
-- 
2.20.1

[Buildroot] [PATCH 2/3 v2] package/kmod: add option for host variant

[Yann E. MORIN]

Curently, host-kmod has no option to enable it, because only
the kernel depends on it, and this is unconditional (because
we can't know if modules will be enabled in the kernel config).

But we're soon to add options to enable various features of
kmod, so we'll need a place where to show those features.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Cc: Lucian Buga <lucianbuga@gmail.com>
---
 linux/Config.in             | 1 +
 package/Config.in.host      | 1 +
 package/kmod/Config.in.host | 6 ++++++
 3 files changed, 8 insertions(+)
 create mode 100644 package/kmod/Config.in.host

diff --git a/linux/Config.in b/linux/Config.in
index d7373ab25a..231d5a5962 100644
--- a/linux/Config.in
+++ b/linux/Config.in
@@ -2,6 +2,7 @@ menu "Kernel"
 
 config BR2_LINUX_KERNEL
 	bool "Linux Kernel"
+	select BR2_PACKAGE_HOST_KMOD # Unconditional, even if modules not enabled
 	select BR2_PACKAGE_HOST_IMAGEMAGICK if BR2_LINUX_KERNEL_CUSTOM_LOGO_PATH != ""
 	help
 	  Enable this option if you want to build a Linux kernel for
diff --git a/package/Config.in.host b/package/Config.in.host
index f1246c708f..140ffa7198 100644
--- a/package/Config.in.host
+++ b/package/Config.in.host
@@ -38,6 +38,7 @@ menu "Host utilities"
 	source "package/imx-usb-loader/Config.in.host"
 	source "package/jq/Config.in.host"
 	source "package/jsmin/Config.in.host"
+	source "package/kmod/Config.in.host"
 	source "package/libp11/Config.in.host"
 	source "package/lld/Config.in.host"
 	source "package/lpc3250loader/Config.in.host"
diff --git a/package/kmod/Config.in.host b/package/kmod/Config.in.host
new file mode 100644
index 0000000000..ea99a790f6
--- /dev/null
+++ b/package/kmod/Config.in.host
@@ -0,0 +1,6 @@
+config BR2_PACKAGE_HOST_KMOD
+	bool "host kmod"
+	help
+	  kmod is a set of tools to handle common tasks with Linux
+	  kernel modules like insert, remove, list, check properties,
+	  resolve dependencies and aliases.
-- 
2.20.1

[Buildroot] [PATCH 3/3 v2] package/lmod: add compression support for the host variant

[Yann E. MORIN]

The kernel may install compressed modules. At the end of the buid, we
then run depmod, to generate modules.dep and a few assorted files, so
that loadign modules works properly on the target (loading by alias,
loading dependencies...)

However, depmod needs support for compressed modules, or it would
generate empty modules.dep et al.

Systematically adding support for gz and xz, and the required deps, was
deemed too much of a burden [0], so we add too options to enable either
or both compression.

[0] http://lists.busybox.net/pipermail/buildroot/2018-April/218410.html

Yegor Yefremov <yegorslists@googlemail.com>
Cc: Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Lucian Buga <lucianbuga@gmail.com>

---
Changes v1 -> v2:
  - make compression support optional  (Thomas)
---
 package/kmod/Config.in.host | 10 ++++++++++
 package/kmod/kmod.mk        | 14 ++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/package/kmod/Config.in.host b/package/kmod/Config.in.host
index ea99a790f6..810fac3156 100644
--- a/package/kmod/Config.in.host
+++ b/package/kmod/Config.in.host
@@ -4,3 +4,13 @@ config BR2_PACKAGE_HOST_KMOD
 	  kmod is a set of tools to handle common tasks with Linux
 	  kernel modules like insert, remove, list, check properties,
 	  resolve dependencies and aliases.
+
+if BR2_PACKAGE_HOST_KMOD
+
+config BR2_PACKAGE_HOST_KMOD_GZ
+	bool "support gzip-compresed modules"
+
+config BR2_PACKAGE_HOST_KMOD_XZ
+	bool "support xz-compresed modules"
+
+endif
diff --git a/package/kmod/kmod.mk b/package/kmod/kmod.mk
index cdbfc5a469..979c3e87e6 100644
--- a/package/kmod/kmod.mk
+++ b/package/kmod/kmod.mk
@@ -75,6 +75,20 @@ else
 KMOD_CONF_OPTS += --disable-tools
 endif
 
+ifeq ($(BR2_PACKAGE_HOST_KMOD_GZ),y)
+HOST_KMOD_DEPENDENCIES += host-zlib
+HOST_KMOD_CONF_OPTS += --with-zlib
+else
+HOST_KMOD_CONF_OPTS += --without-zlib
+endif
+
+ifeq ($(BR2_PACKAGE_HOST_KMOD_XZ),y)
+HOST_KMOD_DEPENDENCIES += host-xz
+HOST_KMOD_CONF_OPTS += --with-xz
+else
+HOST_KMOD_CONF_OPTS += --without-xz
+endif
+
 # We only install depmod, since that's the only tool used for the
 # host.
 define HOST_KMOD_INSTALL_TOOLS
-- 
2.20.1

[Buildroot] [PATCH 1/3 v2] package/kmod: explicitly disable compression when not enabled

[Matthew Weber]

Yann,

On Sun, May 3, 2020 at 8:12 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Yegor Yefremov <yegorslists@googlemail.com>

Tested support for a compressed and signed modules (found [1] was
needed to allow modinfo to display signed module details)

BR2_aarch64=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_CUSTOM_VERSION=y
BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.16.7"
BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/aarch64-virt/linux.config"
BR2_LINUX_KERNEL_NEEDS_HOST_OPENSSL=y
BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y
BR2_PACKAGE_XZ=y
BR2_PACKAGE_LIBZIP=y
BR2_PACKAGE_OPENSSL=y      # modinfo requires so it can display signature info
BR2_PACKAGE_KMOD=y
BR2_PACKAGE_KMOD_TOOLS=y
BR2_TARGET_ROOTFS_EXT2=y
# BR2_TARGET_ROOTFS_TAR is not set
BR2_PACKAGE_HOST_KMOD_GZ=y
BR2_PACKAGE_HOST_KMOD_XZ=y

Linux Kernel options enabled
       CONFIG_MODULES=y
       CONFIG_MODULE_UNLOAD=y
       CONFIG_MODVERSIONS=y
       CONFIG_MODULE_SRCVERSION_ALL=y
       CONFIG_MODULE_SIG=y
       CONFIG_MODULE_SIG_FORCE=y
       CONFIG_MODULE_SIG_SHA256=y
       CONFIG_MODULE_COMPRESS=y
       CONFIG_MODULE_COMPRESS_XZ=y

Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>

[1] http://patchwork.ozlabs.org/project/buildroot/patch/20200514182806.56592-1-matthew.weber at rockwellcollins.com/

[Buildroot] [PATCH 2/3 v2] package/kmod: add option for host variant

[Matthew Weber]

Yann,


On Sun, May 3, 2020 at 8:14 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Curently, host-kmod has no option to enable it, because only
> the kernel depends on it, and this is unconditional (because
> we can't know if modules will be enabled in the kernel config).
>
> But we're soon to add options to enable various features of
> kmod, so we'll need a place where to show those features.
>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Yegor Yefremov <yegorslists@googlemail.com>
> Cc: Lucian Buga <lucianbuga@gmail.com>

Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>

[Buildroot] [PATCH 3/3 v2] package/lmod: add compression support for the host variant

[Matthew Weber]

Yann,



On Sun, May 3, 2020 at 8:14 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> The kernel may install compressed modules. At the end of the buid, we
> then run depmod, to generate modules.dep and a few assorted files, so
> that loadign modules works properly on the target (loading by alias,
> loading dependencies...)
>
> However, depmod needs support for compressed modules, or it would
> generate empty modules.dep et al.
>
> Systematically adding support for gz and xz, and the required deps, was
> deemed too much of a burden [0], so we add too options to enable either
> or both compression.
>
> [0] http://lists.busybox.net/pipermail/buildroot/2018-April/218410.html
>
> Yegor Yefremov <yegorslists@googlemail.com>
> Cc: Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Lucian Buga <lucianbuga@gmail.com>

Tested-by: Matt Weber <matthew.weber@rockwellcollins.com>

[Buildroot] [PATCH 3/3 v2] package/lmod: add compression support for the host variant

[Matthew Weber]

Yann,

On Sun, May 3, 2020 at 8:14 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> The kernel may install compressed modules. At the end of the buid, we
buid -> build
> then run depmod, to generate modules.dep and a few assorted files, so
> that loadign modules works properly on the target (loading by alias,
loading -> loading

> loading dependencies...)
>
> However, depmod needs support for compressed modules, or it would
> generate empty modules.dep et al.
>
> Systematically adding support for gz and xz, and the required deps, was
> deemed too much of a burden [0], so we add too options to enable either
> or both compression.
>
> [0] http://lists.busybox.net/pipermail/buildroot/2018-April/218410.html
>
> Yegor Yefremov <yegorslists@googlemail.com>
> Cc: Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Lucian Buga <lucianbuga@gmail.com>
>
> ---
> Changes v1 -> v2:
>   - make compression support optional  (Thomas)
> ---
>  package/kmod/Config.in.host | 10 ++++++++++
>  package/kmod/kmod.mk        | 14 ++++++++++++++
>  2 files changed, 24 insertions(+)
>
> diff --git a/package/kmod/Config.in.host b/package/kmod/Config.in.host
> index ea99a790f6..810fac3156 100644
> --- a/package/kmod/Config.in.host
> +++ b/package/kmod/Config.in.host
> @@ -4,3 +4,13 @@ config BR2_PACKAGE_HOST_KMOD
>           kmod is a set of tools to handle common tasks with Linux
>           kernel modules like insert, remove, list, check properties,
>           resolve dependencies and aliases.
> +
> +if BR2_PACKAGE_HOST_KMOD
> +
> +config BR2_PACKAGE_HOST_KMOD_GZ
> +       bool "support gzip-compresed modules"

gzip-compresed ->  gzip-compressed

> +
> +config BR2_PACKAGE_HOST_KMOD_XZ
> +       bool "support xz-compresed modules"

xz-compresed -> xz-compressed

-Matt

[Buildroot] [PATCH 1/3 v2] package/kmod: explicitly disable compression when not enabled

[Matthew Weber]

Kris,


On Sun, May 3, 2020 at 8:12 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Yegor Yefremov <yegorslists@googlemail.com>
> ---
>  package/kmod/kmod.mk | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/package/kmod/kmod.mk b/package/kmod/kmod.mk
> index e2dfea5c7b..cdbfc5a469 100644
> --- a/package/kmod/kmod.mk
> +++ b/package/kmod/kmod.mk
> @@ -35,11 +35,15 @@ endif
>  ifeq ($(BR2_PACKAGE_ZLIB),y)
>  KMOD_DEPENDENCIES += zlib
>  KMOD_CONF_OPTS += --with-zlib
> +else
> +KMOD_CONF_OPTS += --without-zlib
>  endif
>
>  ifeq ($(BR2_PACKAGE_XZ),y)
>  KMOD_DEPENDENCIES += xz
>  KMOD_CONF_OPTS += --with-xz
> +else
> +KMOD_CONF_OPTS += --without-xz
>  endif
>
>  ifeq ($(BR2_PACKAGE_PYTHON)$(BR2_PACKAGE_PYTHON3),y)
> --
> 2.20.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Reflecting this for your review

[Buildroot] [PATCH 1/3 v2] package/kmod: explicitly disable compression when not enabled

[Kris Bahnsen]

Yann,


> On Sun, May 3, 2020 at 8:12 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
>> Cc: Yegor Yefremov <yegorslists@googlemail.com>


Tested support for compressed kernel modules. Did not test signed modules.

BR2_arm=y
BR2_cortex_a9=y
BR2_ARM_ENABLE_NEON=y
BR2_ARM_ENABLE_VFP=y
BR2_ARM_FPU_VFPV3=y
BR2_CCACHE=y
BR2_TOOLCHAIN_BUILDROOT_GLIBC=y
BR2_KERNEL_HEADERS_4_14=y
BR2_GCC_VERSION_8_X=y
BR2_TOOLCHAIN_BUILDROOT_CXX=y
BR2_GCC_ENABLE_LTO=y
BR2_GCC_ENABLE_OPENMP=y
BR2_TARGET_GENERIC_HOSTNAME="tsa38x"
BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_MDEV=y
BR2_TARGET_TZ_INFO=y
BR2_ROOTFS_OVERLAY="$(BR2_EXTERNAL_TECHNOLOGIC_PATH)/board/tsa38x/rootfs_overlay"
BR2_ROOTFS_POST_BUILD_SCRIPT="$(BR2_EXTERNAL_TECHNOLOGIC_PATH)/board/tsa38x/post-build.sh"
BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_CUSTOM_TARBALL=y
BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION="$(call 
github,embeddedarm,linux-a38x,9007c40f288d3eb01689c66ac0fad166d63c1958)/linux-9007c40f288d3eb01689c66ac0fad166d63c1958.tar.gz"
BR2_LINUX_KERNEL_DEFCONFIG="tsa38x"
BR2_LINUX_KERNEL_DTS_SUPPORT=y
BR2_LINUX_KERNEL_INTREE_DTS_NAME="armada-385-ts7840"
BR2_LINUX_KERNEL_INSTALL_TARGET=y
BR2_PACKAGE_LINUX_TOOLS_GPIO=y
BR2_PACKAGE_LINUX_TOOLS_IIO=y
BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y
BR2_PACKAGE_BUSYBOX_WATCHDOG=y
BR2_PACKAGE_BUSYBOX_WATCHDOG_PERIOD="30"
BR2_PACKAGE_BZIP2=y
BR2_PACKAGE_XZ=y
BR2_PACKAGE_LINUX_FIRMWARE=y
BR2_PACKAGE_LINUX_FIRMWARE_IMX_SDMA=y
BR2_PACKAGE_LIBZIP=y
BR2_PACKAGE_OPENSSL=y
BR2_PACKAGE_LIBOPENSSL_BIN=y
BR2_PACKAGE_EXPAT=y
BR2_PACKAGE_LIBEVENT=y
BR2_PACKAGE_LIBGLIB2=y
# BR2_PACKAGE_IFUPDOWN_SCRIPTS is not set
BR2_PACKAGE_KMOD=y
BR2_PACKAGE_KMOD_TOOLS=y
BR2_TARGET_ROOTFS_TAR_XZ=y
BR2_PACKAGE_HOST_KMOD_XZ=y
BR2_PACKAGE_HOST_UBOOT_TOOLS=y
BR2_PACKAGE_HOST_UTIL_LINUX=y


Relevant Linux Kernel options

CONFIG_MODULES_USE_ELF_REL=y
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
CONFIG_STRICT_KERNEL_RWX=y
CONFIG_ARCH_HAS_STRICT_MODULE_RWX=y
CONFIG_STRICT_MODULE_RWX=y
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SIG is not set
CONFIG_MODULE_COMPRESS=y
CONFIG_MODULE_COMPRESS_XZ=y

Tested-by: Kris Bahnsen <kris@embeddedarm.com>

-- 
Best Regards,
________________________________________________________________
Kris Bahnsen, Technologic Systems     | voice: (480) 837-5200
16525 East Laser Drive                | fax: (480) 837-5300
Fountain Hills, AZ 85268              | web: www.embeddedARM.com

[Buildroot] [PATCH 0/3 v2] package/kmod: add support for compressed modules (branch yem/host-kmod-z)

[Thomas Petazzoni]

On Sun,  3 May 2020 15:09:58 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Yann E. MORIN (3):
>       package/kmod: explicitly disable compression when not enabled
>       package/kmod: add option for host variant
>       package/lmod: add compression support for the host variant

Thanks, I've applied after taking into account the review comments from
Matt Weber.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com